HyukjinKwon opened a new pull request, #57022:
URL: https://github.com/apache/spark/pull/57022
> **[DO-NOT-MERGE]** — opened from a personal fork by an automated
multi-agent CI-greening session. It is here to validate the deflake on GitHub
Actions before it is marked ready for review. The `[DO-NOT-MERGE]` prefix and
draft status will be removed once the fork validation is green.
### What changes were proposed in this pull request?
Test-only change to deflake `KafkaSourceStressForDontFailOnDataLossSuite`
(the `stress test for failOnDataLoss=false` test).
The test rapidly creates, deletes and recreates topics (with
`kafka.metadata.max.age.ms=1`) while a streaming query with
`failOnDataLoss=false` runs, and asserts the query never fails. This PR widens
the offset-fetch retry budget the test configures:
```
- .option("fetchOffset.retryIntervalMs", "3000")
+ .option("fetchOffset.numRetries", "10")
+ .option("fetchOffset.retryIntervalMs", "500")
```
(from the default `numRetries=3` and the test's previous
`retryIntervalMs=3000`, i.e. ~3s of retrying, to `10 × 500ms` = up to ~5s).
### Why are the changes needed?
The test is flaky — it fails roughly **50%** of runs on the
`build_branch42_maven_java21` lane and occasionally on master, always with:
```
org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] ...
terminated with exception:
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: Failed to
fetch metadata for
partition failOnDataLoss-<n>-0 because metadata for topic
`failOnDataLoss-<n>` could not be found
```
Root cause: an offset fetch races one of the test's own concurrent topic
deletions. `KafkaOffsetReader{Admin,Consumer}.withRetries` retries transient
`NonFatal` failures, but only `fetchOffset.numRetries` times (default 3).
During a burst of the test's deliberate topic churn, 3 retries can be exhausted
before the topic's metadata becomes available again, so the transient
`UnknownTopicOrPartitionException` is rethrown and fails the query — even
though `failOnDataLoss=false`, which is exactly the condition this stress test
means to tolerate.
Giving the reader more retries at a shorter interval lets the test ride out
its self-induced churn (a deleted topic is typically recreated within a second)
without exhausting the budget and without materially slowing the query. This
does not change the production `failOnDataLoss` semantics; it only tunes the
test's own reader options.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
Because the failure is intermittent, the `kafka` Maven lane was run
repeatedly on a fork via GitHub Actions.
**Before (apache/spark, FAILING):**
- `Build / Maven (branch-4.2, JDK 21)`:
https://github.com/apache/spark/actions/runs/28749011147 —
`KafkaSourceStressForDontFailOnDataLossSuite: stress test for
failOnDataLoss=false *** FAILED ***` with the
`UnknownTopicOrPartitionException` above.
**After (HyukjinKwon fork, PASSING — Maven JDK21, matrix narrowed to the
`kafka-0-10*` modules):**
- _(validation runs in progress; links added once green)_
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
