zhang-arvin opened a new pull request, #11896:
URL: https://github.com/apache/seatunnel/pull/11896
## Summary
Fixes #11889 - MongoDB CDC non-exactly-once snapshot extraction stuck issue
## Root Cause
In non-exactly-once mode, the consumer (`pollSplitRecordsIfNotExactlyOnce`)
stops reading from the queue after receiving the HIGH watermark event. However,
the producer (`MongodbScanFetchTask`) continues to synchronously execute the
backfill task which writes backfill events to the same queue. Since the
consumer has stopped consuming, the queue fills up and the producer blocks
forever, causing a deadlock.
### Deadlock chain:
1. Thread 306 (debezium-snapshot-reader) blocks on `queue.enqueue()` - queue
full
2. Thread 284 (Source Data Fetcher) blocks on `elementsQueue.offer()` -
downstream queue full
3. Thread 261 (BlockingWorker) `poll()` returns null, enters sleep loop
4. All downstream sink threads starved
### Root cause in code:
```
IncrementalSourceScanFetcher.pollSplitRecordsIfNotExactlyOnce():
reads HIGH watermark -> sets hasNextElement = false -> stops consuming
MongodbScanFetchTask.execute():
sends HIGH watermark
synchronously executes backfill (MongodbStreamFetchTask) -> writes to same
queue
QUEUE FULL -> DEADLOCK
```
## Fix
Skip the backfill step when `isExactlyOnce` is false, as backfill is only
needed for the exactly-once merge logic (merging overlapping snapshot and
change records). In non-exactly-once mode, an END watermark is sent directly
after the HIGH watermark to allow the task to complete normally.
## Verification
- The fix eliminates the queue deadlock by ensuring the producer does not
write unconsumed backfill events in non-exactly-once mode
- The task completes normally by sending END watermark directly after HIGH
watermark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
