goingforstudying-ctrl opened a new pull request, #2119: URL: https://github.com/apache/datafusion-ballista/pull/2119
Hit this while running queries with joins against flaky object storage: a transient IO error on the build side of a hash join kills the whole job instead of getting retried. Turned out the retry classifier in `ballista/core/src/error.rs` does a shallow `matches!(*e, DataFusionError::IoError(_))` on the outermost variant. But errors coming off a join's shared build side get wrapped in `DataFusionError::Shared` (an `Arc` for sharing across consumers), so the `IoError` inside is never seen and the task falls through to the catch-all `retryable: false` arm. AQE off works fine because the error arrives unwrapped, which is how I narrowed it down to the wrapping. DataFusion already has `find_root()` for exactly this, so the fix is to classify on the root error instead of the wrapper. One-line change plus unit tests covering bare, `Shared`-wrapped, `Context`-wrapped-`Shared`, and non-IO cases. Closes #2028. Tested with `cargo test -p ballista-core --lib error::` (5 new tests pass) and `cargo clippy -p ballista-core --lib` is clean. I couldn't run the full cluster chaos harness from #2026 locally, but the unit tests cover the classifier paths directly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
