comphead commented on issue #2652: URL: https://github.com/apache/datafusion-ballista/issues/2652#issuecomment-6419988602
> Drafted with Claude Code at my request, after comparing the proposal with Spark master and StarRocks `main`. Please push back where it's wrong. ## Comparison with Spark and StarRocks Both systems already use this model, which supports the overall direction. - **Spark** gained a pipelined shuffle in 2026, used first by Real-Time Mode: SPARK-58185, SPARK-58263, SPARK-58398, SPARK-58508. It places stage groups together, admits a group only if it fits in free slots, and allows one task failure at most (`maxTaskFailures = 1`). A fetch failure aborts the whole group, and the caller reruns it. - **StarRocks** places every fragment up front by default (`AllAtOnceExecutionSchedule`). It pushes data over brpc with bounded buffers on both sides, and any failure cancels the query. Both confirm several of the proposal's choices: - **No request cap on reads.** Spark's streaming reader connects to every writer and bounds memory with a byte quota per writer instead. - **No retry of a single stage.** Spark's scheduler comments say resubmitting one stage alone "would deadlock the group". - **No AQE on pipelined edges.** ## Gaps worth addressing 1. **Memory is overcommitted under the vcore rule.** Each task gets a `FairSpillPool` of `per_vcore × vcores_consumed` ([executor_process.rs#L233-L257](https://github.com/apache/datafusion-ballista/blob/b89a4e4654b0bd5259856600d0e45703326b9d3f/ballista/executor/src/executor_process.rs#L233-L257)). In pipelined mode, every stage's tasks on an executor are alive at the same time. Their pools then add up to more than `memory_pool_size`. Reserving the largest stage's vcores is fine for CPU, because blocked tasks are parked futures. Memory needs either the sum, or task pools sized from the job's reservation on that executor. Spark counts the sum of all stages' partitions. 2. **Consumed partitions vs. not-yet-registered ones.** Entries are dropped once a partition is consumed, and a request that finds no entry waits for the producer. So a late or duplicate request would hang until the timeout instead of failing fast. StarRocks has the push-side version of this problem. `DataStreamMgr::transmit_chunk` returns OK and drops data when the receiver is gone, with a TODO noting it can't tell teardown from error. Keeping tombstones until job cleanup would avoid it. 3. **Early finish inside non-final stages.** StarRocks short-circuits a consumer that finishes early, so its producers unblock, and then cancels the remaining work on `LIMIT_REACH`. The proposal should say that a dropped Flight stream counts as a normal finish for the producer, not an error. 4. **Bound queues in bytes.** Both systems size exchange buffers in bytes. Spark's reader quota is 32 MB in total. StarRocks uses `exchg_node_buffer_size_bytes` on the receiver and `sink_buffer_mem_limit_per_driver` on the sender. Limits counted in batches don't bound memory when rows are wide. 5. **Open question 3.** Neither system falls back to another execution model. StarRocks queues with a pending timeout and a cap on queued queries, and Spark's barrier mode retries on an interval. I'd suggest a bounded wait with an explicit timeout error rather than a silent fallback to blocking. 6. **Open question 4.** Eager start matches both systems. StarRocks deploys top-down only because it pushes data, and Ballista's bounded wait for registration covers that race. StarRocks' opt-in `PhasedExecutionSchedule` (build side first) exists to save memory, which argues for making start policy pluggable. 7. **Follow-ups.** For whole-query retry: StarRocks retries only before the first result byte reaches the client, and never for Arrow Flight SQL clients. For #1375: StarRocks holds probe-side scans with a short explicit wait (`global_runtime_filter_wait_timeout`, 20 ms) rather than relying on backpressure. It also turns on same-node pass-through by default (`enable_exchange_pass_through`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
