laserninja commented on issue #16033: URL: https://github.com/apache/iceberg/issues/16033#issuecomment-4954483234
Yes, the lz4-java route is what PR #16054 takes, for exactly the reasons you mention. The Puffin spec requires an LZ4 single compression frame with content size present, and aircompressor v2 (what we're on today) only exposes block-level LZ4, not the frame format. Getting frame support via airlift/aircompressor#357 would mean bumping aircompressor v2 → v3, which is a much larger, riskier change to pull in just for footer compression. lz4-java's LZ4FrameOutputStream/LZ4FrameInputStream give us spec-compliant frame compression directly, with a round-trip test in TestPuffinFormat verifying it. The dependency was already declared in the version catalog and is now wired into iceberg-core. (Also updated the LICENSE attribution to point at the maintained fork, github.com/yawkat/lz4-java, per your other comment.) Happy to revisit the aircompressor path as a follow-up once v3 lands, but I'd suggest keeping this fix on lz4-java so we can unblock the runtime UnsupportedOperationException without a major dependency bump. Would be good to hear if anyone feels strongly about the aircompressor direction though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
