voonhous opened a new issue, #20635: URL: https://github.com/apache/hudi/issues/20635
Part of #20625. Line references are against master `db90a0da`. ### Problem `HudiPageSource.getNextSourcePage` loops `while (recordIterator.hasNext())` with no `pageBuilder.isFull()` check (`HudiPageSource.java:116-126`), so the entire merged file group is built into a single page before anything flows downstream. There is no pipelining, a large transient heap per split, and `LIMIT` cannot stop early. ### Direction Return a page when the page builder is full and continue on the next call. ### Exit criteria - Merge-path pages are bounded by the page builder's size limit; covered by a test on a file group larger than one page. - `HudiMorReadBenchmark` asserts green; scan `_rt` wall and CPU not worse than before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
