voonhous opened a new issue, #20635:
URL: https://github.com/apache/hudi/issues/20635

   Part of #20625. Line references are against master `db90a0da`.
   
   ### Problem
   `HudiPageSource.getNextSourcePage` loops `while (recordIterator.hasNext())` 
with no `pageBuilder.isFull()` check (`HudiPageSource.java:116-126`), so the 
entire merged file group is built into a single page before anything flows 
downstream. There is no pipelining, a large transient heap per split, and 
`LIMIT` cannot stop early.
   
   ### Direction
   Return a page when the page builder is full and continue on the next call.
   
   ### Exit criteria
   - Merge-path pages are bounded by the page builder's size limit; covered by 
a test on a file group larger than one page.
   - `HudiMorReadBenchmark` asserts green; scan `_rt` wall and CPU not worse 
than before.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to