cshuo opened a new issue, #20669: URL: https://github.com/apache/hudi/issues/20669
### Problem Native Parquet and ORC log writers currently build Hudi record-key bloom filters, but the current native-log read paths do not use them. MDT bloom-filter indexing also excludes log files. This adds per-record CPU work and increases file size without a corresponding read benefit. ### Proposed change Disable Hudi record-key bloom generation for native Parquet/ORC data and delete logs. Preserve base-file behavior, HFile bloom filters used for key lookups, and physical column statistics. ### Local benchmark Avro native Parquet writer, ZSTD level 1, default BF settings. Median write time includes writer creation, record writes, and close. | Records/file | Payload/record | BF on | BF off | Time reduction | File size reduction | |---:|---:|---:|---:|---:|---:| | 10K | 64 B | 24.31 ms | 18.10 ms | 25.5% | 38.1% | | 100K | 64 B | 139.74 ms | 89.35 ms | 36.1% | 10.8% | | 10K | 1 KB | 69.27 ms | 63.35 ms | 8.5% | 5.1% | | 100K | 1 KB | 573.86 ms | 523.09 ms | 8.8% | 1.1% | Both reductions are relative to BF-enabled writes. Apple M3, Java 11; two JVMs with warmup and alternating BF on/off runs, 16 measurements per configuration. These are local file-writer measurements, not end-to-end throughput. ORC was not benchmarked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
