The GitHub Actions job "Nightly (beta)" on flink.git/release-1.20 has failed.
Run started by GitHub user github-actions[bot] (triggered by 
github-actions[bot]).

Head commit for run:
cf37da108a06ba5c99ebd7d5778279e36aaed7e5 / Kim Gräsman <[email protected]>
[FLINK-36146][connector-base] Fix SingleThreadFetcherManager race condition

This issue triggered when running a highly parallel batch job with
FileSource pointed at a large S3 bucket. The visible symptom was:

    java.util.NoSuchElementException
        at 
java.base/java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:3471)
        at 
org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager.getRunningFetcher(SingleThreadFetcherManager.java:94)
        at 
org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager.addSplits(SingleThreadFetcherManager.java:82)
        at 
org.apache.flink.connector.base.source.reader.SourceReaderBase.addSplits(SourceReaderBase.java:242)
        at 
org.apache.flink.streaming.api.operators.SourceOperator.handleOperatorEvent(SourceOperator.java:428)
        ...

While the exact sequence of events is unclear, there's an obvious
time-of-check/time-of-use bug in getRunningFetcher where fetchers may be
cleared between the time it's checked for empty and the time we try to
take the first element.

Take the fetcher from a single iterator instead. ConcurrentHashMap's
iterator reads the next entry ahead, so next() cannot throw once
hasNext() returned true, even if the fetcher is removed concurrently.

Add a test.

Co-authored-by: Sergey Nuyanzin <[email protected]>
Co-authored-by: Martijn Visser <[email protected]>
Generated-by: Claude Code (Claude Opus 5.5)

Report URL: https://github.com/apache/flink/actions/runs/37560829122

With regards,
GitHub Actions via GitBox

Reply via email to