voonhous opened a new issue, #20630: URL: https://github.com/apache/hudi/issues/20630
Part of #20625. Line references are against master `db90a0da`. ### Problem On the merge path every base row is read as a Trino page, converted to an Avro `GenericData.Record` (`HudiAvroSerializer.serialize`: boxed numerics, decoded strings, decimal and timestamp objects), merged, then converted back to a page (`buildRecordInPage`). This includes the large majority of rows that have no log update. The base-file-only path passes blocks through untouched. ### Direction Design first. Options include an engine-native reader context over Trino blocks (like Spark's `InternalRow` context), or passing base rows with no log entry straight through as pages and converting only the rows that merge. ### Exit criteria - Short design note on this issue before implementation. - `HudiMorReadBenchmark` scan with `-Dbench.s3.latency.enabled=0`, measured on top of the size estimator fix: `_rt`/`_ro` CPU ratio <= 3x. - Allocation profile of scan `_rt` shows no per-row `GenericData.Record` for base rows without a log entry. - Benchmark asserts plus the merge-semantics suites (`TestHudiMor*`, custom-merger tests) green, including projection and column-pruning cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
