CTTY opened a new pull request, #3710:
URL: https://github.com/apache/iceberg-rust/pull/3710

   Stacked on #3670. Only the last commit, `perf(scan): plan manifest entries 
in chunks instead of one task each`, belongs to this PR; I will rebase once 
#3670 merges.
   
   ## Which issue does this PR close?
   
   - None. Follow-up to #3670.
   
   ## What changes are included in this PR?
   
   #3670 parallelises manifest parsing, which helps tables with many manifests 
but not tables whose cost is in the number of entries. After parsing, 
`plan_files` spawned one cpu task per live entry and passed every entry through 
two 16-slot channels (manifest stage to entry stage, entry stage to caller). 
That costs about 8 µs of CPU per entry, as much as decoding the entry itself.
   
   - The manifest stage sends live entries in chunks of up to 256 
(`ENTRY_CHUNK_SIZE`).
   - The entry stage spawns one task per chunk, prunes and builds each entry in 
it, and sends the chunk's planned files as one batch. The returned stream 
flattens the batches, so its type is unchanged.
   - Delete manifests are chunked the same way.
   - `with_manifest_entry_concurrency_limit` now bounds chunks in flight rather 
than single entries; its docs say so.
   
   A large manifest is still split into many chunks, so it spreads over workers.
   
   Chunk size, measured on 4 workers: 64 brings back some per-chunk overhead 
(+6% on 600 × 100); 4096 leaves too few chunks to balance one large manifest 
(+16% on 1 × 60,000); 256 and 1024 perform the same. 256 is chosen because it 
buffers fewer planned files (at most 16 × 256 in the output channel).
   
   No public API change (`make check-public-api` is clean).
   
   ### Measurements
   
   `plan_files`, 4-worker runtime, local files, cold cache, median of 7. Shape 
is manifests × entries per manifest; each entry has bounds and counts for 4 
columns.
   
   | shape | #3670 | this PR | CPU #3670 → this PR |
   |-------|-------|---------|------|
   | 600 × 1 | 70 ms | 72 ms | 250 → 250 ms |
   | 600 × 100 | 283 ms | 172 ms | 1150 → 660 ms |
   | 6 × 10,000 | 248 ms | 126 ms | 910 → 440 ms |
   | 1 × 60,000 | 519 ms | 383 ms | 920 → 460 ms |
   
   What remains in the 1 × 60,000 case is mostly the single-threaded decode of 
that one manifest.
   
   ## Are these changes tested?
   
   Existing scan tests cover pruning, deletes, error propagation (corrupt, 
missing and cancelled manifests from #3670) and the delete/data ordering that 
avoids deadlock. New test:
   
   - `scan::tests::test_plan_files_spans_entry_chunks`: a manifest with `2 * 
ENTRY_CHUNK_SIZE + 1` entries plans every entry exactly once. It fails if the 
partial last chunk is not flushed.
   
   `cargo test -p iceberg --lib`, `cargo clippy --all-targets --all-features -D 
warnings`, `cargo fmt --check` and `make check-public-api` are clean.
   
   ## AI Disclosure
   
   Drafted with AI assistance (Kiro), then reviewed, tested and benchmarked by 
me.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to