coderfender opened a new issue, #26383:
URL: https://github.com/apache/datafusion/issues/26383
### Is your feature request related to a problem or challenge?
`CrossJoinExec` buffers the **entire left input** in memory with no spill
path, so a large left side can exhaust the memory pool even when the right side
(and the useful working set) is tiny.
In `datafusion/physical-plan/src/joins/cross_join.rs`, `load_left_input`
folds all left batches into a `Vec`, growing a `MemoryReservation` per batch:
```rust
async fn load_left_input(..., reservation: MemoryReservation, ...) {
let (batches, _metrics, reservation) = stream
.fold((Vec::new(), metrics, reservation), |(mut batches, metrics,
reservation), batch| async {
let batch_size = batch.get_array_memory_size();
reservation.try_grow(batch_size)?; // grow-or-error; no spill
batches.push(batch);
...
}).await?;
}
```
When the pool refuses `try_grow`, execution fails with `Resources exhausted:
Additional allocation failed for CrossJoinExec ... can spill: false`. Unlike
`SortExec` (and the recent sort-merge-join spill work), `CrossJoinExec` has no
`SpillManager` fallback.
**Reproduction** (observed via datafusion-comet, which delegates Spark
`CartesianProductExec` to `CrossJoinExec`): a 1,000,000-row left range crossed
with a 1-row right range under a ~4 MiB pool fails at its fair share, while
engines that spill the buffered side (e.g. Spark's `UnsafeCartesianRDD`)
complete and return the result. DataFusion 55.2.0's `load_left_input` confirms
unconditional accumulation.
### Describe the solution you'd like
Add a spill path to `CrossJoinExec` so the buffered left input can spill to
disk when the reservation cannot grow, then re-read it from disk during probing
— mirroring the approach `SortExec` and the sort-merge-join operator already
use (`SpillManager` + on-disk runs). This would let cross joins over a large
left input complete under memory pressure instead of erroring.
### Describe alternatives you've considered
- Streaming the left and buffering/spilling the right instead (closer to
Spark's model), though that changes which side is materialized.
- Keeping the current behavior but documenting the unbounded-left-buffer
limitation.
### Additional context
Found while adding native cross-join support in apache/datafusion-comet
(apache/datafusion-comet#6938); Comet currently gates that path as opt-in
specifically because of this non-spillable buffering. Happy to work on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
