Dandandan opened a new pull request, #2334:
URL: https://github.com/apache/datafusion-ballista/pull/2334
## Summary
`RangeShuffleReaderExec` merges all of its input streams to completion even
when the `SortPreservingMergeExec` directly above it only wants the first few
rows. Its `StreamingMerge` was built without a fetch:
```rust
StreamingMergeBuilder::new()
.with_streams(sub_streams)
.with_batch_size(config.batch_size())
.with_reservation(reservation)
.build()? // no .with_fetch(...)
```
So a `fetch=20` top-N over a wide fan-in merges the reader's entire input and
the consumer then discards almost all of it.
## Change
Both operators merge on the same ordering, so the consumer's first `n` rows
can
only come from the reader's first `n`. `RangeShuffleReaderExec` now carries
an
optional `fetch`, implements `fetch()` / `with_fetch()`, and passes the
limit to
`StreamingMergeBuilder`. The adapter pushes the limit down when it plants the
reader beneath such a merge.
DataFusion's own limit pushdown cannot do this — the reader is planted at
adapt
time, after the optimizer chain has already run.
## Scope
This reduces merge **work**. It does **not** lower the merge's peak memory:
an
N-way `StreamingMerge` needs one batch from every source before it can emit a
row, so its floor is `fan_in × batch_size` and is paid up front, before the
fetch can take effect.
That distinction matters for #2321, where the reported failure is a *first*
reservation (`0.0 B already allocated`). This change is worth having on its
own
terms, but it is not a fix for that OOM.
## Testing
- `cargo test -p ballista-core --lib` — 280 pass
- `cargo test -p ballista-scheduler --lib` — 351 pass
- New `fetch_roundtrips_through_with_fetch` checks the limit is visible on
the
operator and survives `with_new_children`, which the clone-style rebuilds
would otherwise drop silently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
