HyukjinKwon opened a new pull request, #57096:
URL: https://github.com/apache/spark/pull/57096
> [!NOTE]
> Draft/verification PR from a personal fork. The `[TEMP][DO-NOT-MERGE]`
fork-only workflow commit (`.github/workflows/zz_fork_aqe_verify.yml`) will be
dropped before this is finalized for review.
### What changes were proposed in this pull request?
Test-only change to deflake `AdaptiveQueryExecSuite`'s `SPARK-47148: AQE
should avoid to submit shuffle job on cancellation`.
The test uses a `slow_udf` in scalar subqueries on the `df`/`df3` shuffle
stages to delay their submission, so that the `df2` coalesce stage (no
subquery) submits its shuffle job and fails with "coalesce test error" first,
while the two delayed stages are still sleeping and thus get cancelled before
submission. The test then asserts the failure chain contains "coalesce test
error" and that exactly the middle stage was materialized.
With a 3000ms sleep this ordering is timing-dependent: under a loaded CI
runner the coalesce stage's map task can be slow enough that a delayed stage is
submitted, or the coalesce stage is itself cancelled, before "coalesce test
error" is thrown — making the error disappear from the failure chain and
failing the assertion `errMsgList.exists(_.contains("coalesce test error"))`.
Increase the sleep to 15000ms so the coalesce stage reliably wins the race.
The per-test timeout is 20 minutes, so this adds no meaningful cost.
### Why are the changes needed?
The test is flaky on the `build_java21` lane (observed on branch-4.0),
failing with `...contains("coalesce test error") was false`.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
Ran the SPARK-47148 AQE cancellation test repeatedly on GitHub Actions on a
fork.
**CI results**
- Failing (before): https://github.com/apache/spark/actions/runs/28778790780
- Passing (after): <!-- PASS_LINK -->
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
