yihua opened a new pull request, #933:
URL: https://github.com/apache/hudi-rs/pull/933
## Description
closes #932
The record-key bloom filter and key range in the base file footer are now
written only when `hoodie.parquet.bloom.filter.enabled` is true or
`hoodie.index.type` is a bloom index; when neither holds, the footer carries
nothing extra and no key is hashed. Sizing and format are unchanged when the
filter is on. hudi-rs locates keys through the record index and never reads
these footers, while hashing every key at Java's default sizing (30 MurmurHash
passes per key) was one of the largest costs of a bulk load.
One interop note for review: a Java writer that later upserts into such a
table with `hoodie.index.type=BLOOM` fails on files without the footer (Java
behaves the same when the filter is turned off), so a table meant for Java's
bloom index should be created with `hoodie.parquet.bloom.filter.enabled=true`.
The Spark parity scenario for Java index lookups now creates its tables that
way.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`test_base_file_footer_omits_bloom_filter_by_default` fails on the previous
default and passes now; the footer content test and the config resolution test
opt in and still check Java's layout and sizing. Spark parity scenario I
(SIMPLE, BLOOM and GLOBAL_BLOOM upserts from Spark on rs-written tables) passes
with the opt-in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
