linliu-code opened a new pull request, #670: URL: https://github.com/apache/hudi-rs/pull/670
**Stacked on #639–#669** — review only the last two commits. ## The problem Reading a merge-on-read file group pulls **every log file into memory whole** and decodes **every block in it**, then applies five gates and discards most of the results. Peak memory tracks the largest log file, and the decode cost is paid for blocks nothing will use. ## The change **Commit 1 — a windowed reader.** `StorageReader` gains a second mode: `new_streaming` fetches nothing up front and serves reads out of a 16 MB window, refilling as the cursor moves. `new` still reads the whole file and is untouched; both modes look identical through `Read`/`Seek`. `LogBlockFetcher` comes with it — an object-store handle and a path, no bytes — so a block can read its own range later. **Commit 2 — a lazy block scan.** The scan walks headers out of the window, recording where each block's content sits without reading it. The gates run on headers, as they already did (Pass 1 never looked at content). Each surviving block then reads its own range. So what is held is **one window during the sweep, one block's content after it** — instead of the whole file plus every decoded block. ## Why the ranged reads are spawned onto a shared runtime `Read` is synchronous, `object_store` is not, and both obvious bridges fail: - the sync read can be reached from **inside** another runtime, where `block_on` panics with *"Cannot start a runtime from within a runtime"* - a runtime built per read takes hyper's connection dispatcher down with it when dropped, failing every later request against the same cached store So reads are spawned onto a process-lifetime runtime and waited on over a channel. ## Correctness The property this rests on: **sweep-then-inflate must equal the eager read.** Three tests assert it directly, one per block type — same block count, same headers, same decoded row counts — for Avro, Parquet and Delete blocks. Beyond that, the whole v2 path now runs this way, so the existing suite is the real evidence: **58 harness cases and 19 gold fixtures still match their Spark snapshots**, reading real merge-on-read tables end to end. Three more tests on the reader itself: streaming matches eager across window refills on a file larger than two windows; seeking backwards refills and seeking past the end reads nothing; a fetcher reads only its own range. ## Two details worth review **Corruption is still detected during the sweep**, not deferred with the content. A corrupt block cannot be trusted to say where the next one starts, so that check cannot wait. **The recorded content range starts at the block's content-length field, not after it** — decoding reads that field itself, so the same bytes reach the same decoder either way. Getting this wrong produced `Invalid log block version: 228`, which is how the equivalence tests earned their keep. Full workspace green: 1200 lib + 79 table-read + 39 datafusion + 21 + 12. No new clippy findings in the changed files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
