HyukjinKwon opened a new pull request, #56791:
URL: https://github.com/apache/spark/pull/56791
### What changes were proposed in this pull request?
Backport [SPARK-57652] (master commit
`850a41e0623834c9b7eb5457692f08b61b8d770a`)
to `branch-4.2`. It replaces the fixed `Thread.sleep(5000)` in the
`"snapshotStartBatchId with transformWithState"` test of
`StateDataSourceTransformWithStateSuite` with a deterministic
`waitForStateSnapshot`
helper that blocks (via `eventually`) until the RocksDB snapshot `.zip` for
the
needed state-store version has actually been uploaded — called immediately
after
that version is committed, while it is still the current version.
### Why are the changes needed?
`branch-4.2` does not have the deflake fix, so the test is still flaky
there. It
fails on the scheduled **Maven (branch-4.2, Scala 2.13, JDK 21)** `sql#core -
extended tests` job (run 28189497187), in
`StateDataSourceTransformWithStateSuiteWithRowChecksum`:
```
- snapshotStartBatchId with transformWithState (with changelog
checkpointing) (encoding = unsaferow) *** FAILED ***
org.apache.spark.SparkException: [CANNOT_LOAD_STATE_STORE.UNCATEGORIZED]
An error occurred during loading state.
Caused by: java.io.FileNotFoundException: .../state/0/1/2.zip does not
exist
```
The maintenance thread only ever snapshots the **current** state-store
version, so
the version-2 snapshot the `snapshotStartBatchId=1` reader needs can only be
created while version 2 is current. A `Thread.sleep` at the end of the
stream is
racy: once later batches advance the current version, maintenance never goes
back
to snapshot version 2, so `2.zip` may never appear. The backported helper
waits
for the snapshot while version 2 is still current, deterministically forcing
it to
exist.
### Does this PR introduce _any_ user-facing change?
No, test only.
### How was this patch tested?
Backported test, run on the branch-4.2 Maven JDK21 `sql#core - extended
tests`
job (see CI evidence). Clean cherry-pick of the master commit.
### CI evidence
- Before (red): https://github.com/apache/spark/actions/runs/28189497187
(branch-4.2 Maven JDK21, `sql#core - extended tests` —
`CANNOT_LOAD_STATE_STORE` / `2.zip does not exist`)
- After (green):
https://github.com/HyukjinKwon/spark/actions/runs/28205027037 (fork, branch-4.2
Maven JDK21, matrix narrowed to `sql#core - extended tests`)
### Was this patch authored or co-authored using generative AI tooling?
Yes, drafted with assistance from Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
