HyukjinKwon opened a new pull request, #57097:
URL: https://github.com/apache/spark/pull/57097
### What changes were proposed in this pull request?
**[DO-NOT-MERGE draft — flaky-test deflake, verification in progress]**
This PR fixes two independent flaky failures observed in scheduled CI on
`master`:
1. **`FileDataSourceV2FallBackSuite` "Fallback Parquet V2 to V1"** (seen in
`build_maven_java21_macos26`, `sql#core - other tests`).
The test asserted `commands.length == 1` on a `QueryExecutionListener`.
An extra eager `collect` callback on the analyzed `UnresolvedDataSource` can
race onto the listener bus, making the count `2` and failing the test
non-deterministically. Fixed by filtering the recorded events to the write
`"command"` entry and asserting on that, instead of requiring the bus to
contain exactly one event.
2. **`build_sparkr_window` silent 2-hour timeout** (windows-2025). The "Run
SparkR tests" step occasionally hangs while the R process waits on the SparkR
backend socket handshake; with no step-level timeout this silently consumes the
full 120-minute job timeout and surfaces as a *cancellation* rather than a
fast, retryable failure. Added `timeout-minutes: 60` to the step so a stalled
handshake fails fast (successful runs finish it well under 60 min).
Two further failures on the same macOS-26 Maven workflow were investigated
but are **not** fixed here because there is no safe in-repo fix (documented
below).
### Why are the changes needed?
These flakes intermittently red scheduled builds without indicating a real
product regression.
### Does this PR introduce _any_ user-facing change?
No. Test-only + CI-workflow-only changes.
### How was this patch tested?
- `FileDataSourceV2FallBackSuite` passes locally (`Tests: succeeded 5,
failed 0`).
- Fork CI verification runs linked below.
### Investigated-but-not-fixed (documented for reviewers)
- **`DynamicPartitionPruningHiveScanSuiteAEOff` "broadcast a single key in a
HashedRelation"** (`sql#hive - slow tests`): fails with an embedded-Derby
internal `ClassCastException` (`ReferencedColumnsDescriptorImpl` →
`ExecRowBuilder`) in the DataNucleus metastore layer, preceded by XA
transaction rollback errors. This is Derby-internal state corruption under the
slow macOS-26 JDK21 run, with no Spark product line to fix; mitigation would be
metastore isolation, out of scope for a deflake.
### CI verification
- master fixes (macOS-26 Maven): _pending_
- SparkR windows watchdog: _pending_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
