cshuo opened a new issue, #20669:
URL: https://github.com/apache/hudi/issues/20669

   ### Problem
   
   Native Parquet and ORC log writers currently build Hudi record-key bloom 
filters, but the current native-log read paths do not use them. MDT 
bloom-filter indexing also excludes log files.
   
   This adds per-record CPU work and increases file size without a 
corresponding read benefit.
   
   ### Proposed change
   
   Disable Hudi record-key bloom generation for native Parquet/ORC data and 
delete logs. Preserve base-file behavior, HFile bloom filters used for key 
lookups, and physical column statistics.
   
   ### Local benchmark
   
   Avro native Parquet writer, ZSTD level 1, default BF settings. Median write 
time includes writer creation, record writes, and close.
   
   | Records/file | Payload/record | BF on | BF off | Time reduction | File 
size reduction |
   |---:|---:|---:|---:|---:|---:|
   | 10K | 64 B | 24.31 ms | 18.10 ms | 25.5% | 38.1% |
   | 100K | 64 B | 139.74 ms | 89.35 ms | 36.1% | 10.8% |
   | 10K | 1 KB | 69.27 ms | 63.35 ms | 8.5% | 5.1% |
   | 100K | 1 KB | 573.86 ms | 523.09 ms | 8.8% | 1.1% |
   
   Both reductions are relative to BF-enabled writes.
   
   Apple M3, Java 11; two JVMs with warmup and alternating BF on/off runs, 16 
measurements per configuration. These are local file-writer measurements, not 
end-to-end throughput. ORC was not benchmarked.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to