HyukjinKwon opened a new pull request, #57095:
URL: https://github.com/apache/spark/pull/57095
> [!NOTE]
> Draft/verification PR from a personal fork. The `[TEMP][DO-NOT-MERGE]`
fork-only workflow commit (`.github/workflows/zz_fork_kafka_verify.yml`) will
be dropped before this is finalized for review.
### What changes were proposed in this pull request?
Test-only change to deflake `KafkaSourceStressForDontFailOnDataLossSuite`
(the `stress test for failOnDataLoss=false` test).
The test rapidly creates, deletes and recreates topics (with
`kafka.metadata.max.age.ms=1`) while a streaming query with
`failOnDataLoss=false` runs, and asserts the query never fails. An offset fetch
can race one of the test's own concurrent topic deletions and hit
`UnknownTopicOrPartitionException`. That is transient and expected here
(`failOnDataLoss=false`): the offset reader retries until the metadata refresh
drops the deleted topic from the pattern subscription. But a bounded retry
budget can be exhausted during a burst of churn under CI load. A previous
deflake (SPARK-57947) raised the budget to `10 × 500ms = 5s`, which still
flaked.
Widen the reader's retry budget so it comfortably outlasts the test's own
20s stress loop:
```
- .option("fetchOffset.numRetries", "10")
- .option("fetchOffset.retryIntervalMs", "500")
+ .option("fetchOffset.numRetries", "60")
+ .option("fetchOffset.retryIntervalMs", "1000")
```
When the loop ends, `query.stop()` interrupts any in-flight retry cleanly
(the retry loop guards on `Thread.currentThread().isInterrupted` and
`Thread.sleep` throws `InterruptedException`), so the larger budget does not
slow teardown.
### Why are the changes needed?
The test is flaky on the `build_maven` lane (observed on branch-4.1),
failing with `UnknownTopicOrPartitionException` even though
`failOnDataLoss=false`.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
Ran `KafkaSourceStressForDontFailOnDataLossSuite` repeatedly on GitHub
Actions on a fork.
**CI results**
- Failing (before): https://github.com/apache/spark/actions/runs/28871397866
- Passing (after): <!-- PASS_LINK -->
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
