goutamadwant opened a new pull request, #701:
URL: https://github.com/apache/hudi-rs/pull/701
## Description
Add `Table::get_file_slices_stream` so engines can begin scheduling reads
before every partition and file slice has been planned.
For storage-backed tables, the new path:
- discovers partition paths from the object store's recursive stream;
- deduplicates and prunes partitions as paths arrive;
- bounds per-partition listings with `hoodie.plan.listing.parallelism`;
- applies footer pruning and timeline exclusion per partition;
- emits slices without a table-wide file-group or file-slice collection.
Snapshot and incremental query semantics match `get_file_slices`.
Incremental planning still materializes the touched file-group set.
Metadata-table-backed planning still materializes files-partition records
because the metadata reader does not expose a row stream. Both paths use the
new public stream API without changing the eager API.
The recursive discovery pass intentionally performs one delimiter listing
for each emitted partition. Object stores do not guarantee partition-contiguous
object order; reusing discovery entries would require retaining incomplete file
groups table-wide. The stream instead retains one deduplication entry per
emitted partition and bounds file groups and slices by listing concurrency.
Global stream order is intentionally unspecified.
On a synthetic local table with 2,003 slices, the eager API returned only
after all 2,003 slices were materialized. Post-change runs yielded the first
slice in 244–262 ms and drained all 2,003 slices in 2.1–4.2 s under concurrent
system load. These measurements demonstrate time-to-first-slice behavior, not a
stable total-throughput improvement.
Closes #590.
Related: #597 optimizes the existing eager recursive listing path. This
change adds a separate object-store-backed stream and does not modify
`get_leaf_dirs`.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [x] Manual tests
- [x] Details are described below
Automated coverage verifies snapshot, incremental, metadata-table, and
empty-table parity; first-slice-before-complete-discovery behavior; deferred
listing-error propagation; recursive relative paths; and non-empty table-prefix
boundaries.
Manual verification used the synthetic 2,003-slice table described above and
confirmed that the eager and streamed APIs both returned all 2,003 slices.
Commands run:
- `cargo fmt --all -- --check`
- `cargo clippy -p hudi-core --all-targets --no-default-features -- -D
warnings`
- `cargo clippy -p hudi-core --all-targets --all-features -- -D warnings`
- `cargo test -p hudi-core --no-default-features`
- `cargo test -p hudi-core --lib`
- `cargo check -p hudi -p hudi-cpp -p hudi-python --no-default-features`
- `cargo doc -p hudi-core --no-default-features --no-deps`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
