andygrove opened a new pull request, #2239: URL: https://github.com/apache/datafusion-ballista/pull/2239
# Which issue does this PR close? Backport of #2119 to `branch-54`. The issue it fixes is #2028. # Rationale for this change A transient IO error on the build side of a hash join kills the whole job instead of being retried. The retry classifier in `ballista/core/src/error.rs` does a shallow `matches!(*e, DataFusionError::IoError(_))` on the outermost variant. Errors coming off a join's shared build side get wrapped in `DataFusionError::Shared` (an `Arc`, for sharing across consumers), so the `IoError` inside is never seen and the task falls through to the catch-all `retryable: false` arm. With AQE off the error arrives unwrapped and retry works, which is how the wrapping was narrowed down as the cause. This turns recoverable object-store flakiness into job failures, so it is worth having on the release branch. # What changes are included in this PR? A clean cherry-pick of b1627080, unmodified. Classifies on `find_root()` instead of the outermost variant, so the retryability decision is based on the root error regardless of wrapping. One-line change plus unit tests covering bare, `Shared`-wrapped, `Context`-wrapped-`Shared`, and non-IO cases. # Are there any user-facing changes? No API changes. Tasks that fail with an IO error wrapped in `DataFusionError::Shared` are now retried rather than failing the job. --- Verified locally on the `branch-54` base: `cargo fmt --all -- --check` is clean, and `cargo check --workspace --all-targets --locked` completes with no warnings on a combined stack of the six backports being proposed together. Test execution is left to CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
