linliu-code opened a new pull request, #655: URL: https://github.com/apache/hudi-rs/pull/655
**Stacked on #639–#654** — their commits appear here until they merge. **Review only the last commit.** ## Why The reader landed **eager-only** in #654: it returns a file group as one batch, so peak memory tracks the base file. This reads the base file a row group at a time instead. ## How The merge loop is synchronous and has to block on the base stream — only legal off the async worker threads. So it runs on a blocking-pool thread and hands batches back over a channel; the caller gets an ordinary `BoxStream` and never sees the blocking. `BlockingBatchReader` adapts the async base stream back into the iterator the merge loop wants, and its doc states the contract it depends on. **The channel holds one batch, deliberately.** It lets the producer decode row group N+1 while the consumer works on N, while bounding the extra memory to a single row group. A deeper channel would multiply peak memory by its depth — which is the thing streaming exists to avoid. ## Measured On the equivalent path in `onehouseinc/hudi-rs-internal#113`, against a generated 600k-row MOR table (181 MB, one base file, ten log files): | path | median wall | peak RSS | |---|---:|---:| | eager | 1316 ms | **1114 MB** | | streaming | 1223 ms | **770 MB** | Same row count either way. ## The alternative that was measured and rejected An **async-native** variant — awaiting the base file from inside the merge loop rather than blocking on it — was built and benchmarked head-to-head. Within **2%** on both wall time and peak memory, less than the run-to-run spread within either. It would have cost a `dyn`-safe async trait method and changes across five files, so it was not taken. Its real advantage is a thread-per-read scaling argument that a single-file-group benchmark cannot show. ## Tests - New: asserts the streaming read returns what the eager one does. Verified non-vacuous by making the streaming branch fail and confirming the test catches it. - Fixed: an existing test drained a streaming source from an async context — the exact misuse the adapter's contract forbids. It now drains on a blocking thread, like every real caller. 1103 lib tests green; build warning-free. Clippy findings in `reader_v2` go from 22 to 20 (an orphaned doc block left over from the port is removed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
