yihua commented on PR #975: URL: https://github.com/apache/hudi-rs/pull/975#issuecomment-6718188302
Pushed another revision in the latest commit. What it fixes: - The metadata HFile codec now comes from the write option `hoodie.hfile.compression.algorithm`, the same key Java uses (and that apache/hudi#20149 carries into the metadata write config), parsed the way Java parses it: `gz` when unset or empty, otherwise a Java codec name in any case. - `gz` and `none` are supported. Other codecs such as `snappy` or `zstd` are rejected before anything is written, since the hudi-rs and released Java HFile readers only decode NONE and GZ. A table without a metadata table writes no HFile and, as in Java, does not check the value. - The option is not stored in hoodie.properties: each HFile records its codec in the trailer, so files written with different codecs read and compact together. - The codec is resolved once per write and used by every metadata HFile write (bootstrap, log blocks, compaction) through one helper. - Checked the checksum-chunk boundary case that apache/hudi#20149 fixes on the Java side: the hudi-rs writer already computes it correctly, and a test now pins a block ending exactly on a chunk boundary. All 11 Spark parity scenarios pass on this commit. LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
