voonhous opened a new issue, #20627: URL: https://github.com/apache/hudi/issues/20627
Part of #20625. Line references are against master `db90a0da`. ### Problem `HudiTrinoReaderContext` extends `HoodieReaderContext` directly and inherits `getRecordSizeEstimator()` = `HoodieRecordSizeEstimator` and `getRecordSerializer()` = `DefaultSerializer` (Kryo). The estimator walks the whole record object graph, shared Avro schema included, on every 100th log record. The estimate therefore counts the schema per record, and the 1 GiB merge buffer spills far too early. Measured locally with `HudiMergeBufferBenchmark` (300k log records, JDK 25): estimate 13,304 B/record, 235,852 of 300,000 records spilled, put about 11 to 14 us/record. ### Direction Override both in `HudiTrinoReaderContext`. `HoodieAvroReaderContext` already does this with `AvroRecordSizeEstimator` and `BufferedRecordSerializer` + `AvroRecordSerializer`. Preferably avoid the object-graph walk entirely, e.g. compute the size from the schema and value sizes. ### Exit criteria - `HudiMergeBufferBenchmark`: 0 records spilled at the default budget, put <= 1 us/record, estimate <= 2 KB/record. - At most one estimator walk per file group, pinned by a unit test that counts estimator calls on a multi-thousand-record buffer. - `HudiMorReadBenchmark` scan with `-Dbench.s3.latency.enabled=0`: `_rt` CPU <= 25% of master's. - Spill path still correct: a test with a tiny `hoodie.memory.merge.max.size` forces spilling and asserts the merged rows equal the in-memory case (`COMMIT_TIME_ORDERING` and `EVENT_TIME_ORDERING`, with deletes). - Existing hudi-trino tests green. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
