andygrove opened a new issue, #2811:
URL: https://github.com/apache/datafusion-ballista/issues/2811
**Describe the bug**
`stage_losing_input_after_partial_success_is_recovered`
(`chaos-testing/tests/ha.rs`, Scenario K) fails intermittently in CI. When the
map-side executor is killed while a final-aggregate task is streaming its
shuffle input, the task sometimes fails with a raw transport error instead of a
fetch failure:
```
query must recover: Execution error: Job ... failed: Job failed due to stage
2 failed:
Task failed due to runtime execution error:
DataFusionError(Shared(External(ArrowError(IpcError(
"Arrow error: Ipc error: code: 'Unknown error', message: \"h2 protocol
error: error reading a body
from connection\", source: hyper::Error(Body, Error { kind: Io(Custom {
kind: BrokenPipe,
error: \"stream closed because of a broken pipe\" }) })")))))
```
The scheduler treats this as an ordinary task failure and fails the job, so
the stage is not re-run as the scenario expects.
**To Reproduce**
Flaky, so it needs a few runs of the chaos test:
```
cargo test -p ballista-chaos --test ha
stage_losing_input_after_partial_success_is_recovered
```
Observed in CI:
- `main` push run 42113028346 (2026-11-05), `test linux ballista`
- PR #2806 run 42417144891 (2026-11-07), `macos test` (the PR does not touch
this code path)
**Expected behavior**
Losing the executor that holds the shuffle input should be reported as a
fetch-partition failure so the scheduler re-runs the producing stage, whichever
way the connection breaks (clean `Unavailable`, h2 reset, or mid-body broken
pipe).
**Additional context**
#2202 made wrapped `FetchFailed` errors classify as fetch-partition errors,
but a body-read failure on the Flight `do_get` stream arrives as
`ArrowError::IpcError` wrapping a tonic/hyper error and appears not to be
matched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
