voonhous opened a new issue, #20630:
URL: https://github.com/apache/hudi/issues/20630

   Part of #20625. Line references are against master `db90a0da`.
   
   ### Problem
   On the merge path every base row is read as a Trino page, converted to an 
Avro `GenericData.Record` (`HudiAvroSerializer.serialize`: boxed numerics, 
decoded strings, decimal and timestamp objects), merged, then converted back to 
a page (`buildRecordInPage`). This includes the large majority of rows that 
have no log update. The base-file-only path passes blocks through untouched.
   
   ### Direction
   Design first. Options include an engine-native reader context over Trino 
blocks (like Spark's `InternalRow` context), or passing base rows with no log 
entry straight through as pages and converting only the rows that merge.
   
   ### Exit criteria
   - Short design note on this issue before implementation.
   - `HudiMorReadBenchmark` scan with `-Dbench.s3.latency.enabled=0`, measured 
on top of the size estimator fix: `_rt`/`_ro` CPU ratio <= 3x.
   - Allocation profile of scan `_rt` shows no per-row `GenericData.Record` for 
base rows without a log entry.
   - Benchmark asserts plus the merge-semantics suites (`TestHudiMor*`, 
custom-merger tests) green, including projection and column-pruning cases.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to