siying opened a new pull request, #48783:
URL: https://github.com/apache/spark/pull/48783
### What changes were proposed in this pull request?
In stream-stream join, only call getLatestCheckpointInfo() when the format
version indicates that checkpoint version is supported.
### Why are the changes needed?
We are hitting flaky test, failing with:
org.apache.spark.sql.streaming.StreamingQueryException: Job aborted due to
stage failure: Task 0 in stage 10.0 failed 1 times, most recent failure: Lost
task 0.0 in stage 10.0 (TID 12) (1b41214d-1151-4535-aa1b-60e66b974850 executor
driver): java.lang.AssertionError: assertion failed: two state stores in a
stream-stream joiner don't return the same batch version
at scala.Predef$.assert(Predef.scala:223)
at
org.apache.spark.sql.execution.streaming.state.SymmetricHashJoinStateManager.getLatestCheckpointInfo(SymmetricHashJoinStateManager.scala)
This is because when not supported, what is returned might not pass the
check and fail the assertion.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Run the flaky test multiple times and make sure it's not failing anymore.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
