Sonal Maheshwari created FLINK-41352:
----------------------------------------
Summary: [mongodb] Support change events larger than 16MB with
$changeStreamSplitLargeEvent
Key: FLINK-41352
URL: https://issues.apache.org/jira/browse/FLINK-41352
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Affects Versions: cdc-3.7.0
Reporter: Sonal Maheshwari
When a change event is larger than 16MB, e.g. an update whose looked-up full
document ({{fullDocument: updateLookup}}) plus update description exceed the
limit, MongoDB fails the getMore with BSONObjectTooLarge (10334).
{{MongoDBStreamFetchTask}} treats 10334 as an invalid change stream and reopens
the cursor from the same resume token, which hits the same event again. The
stream never moves past it and the lag keeps growing.
MongoDB 6.0.9+ provides the {{$changeStreamSplitLargeEvent}} stage, which
splits such events into fragments of at most 16MB. Proposal:
- Add an opt-in option {{scan.split-large-event.enabled}} (default false) that
appends the stage to the change stream pipeline.
- Reassemble the fragments in the stream fetch task into one change event that
carries the resume token of the last fragment, so checkpoints never store a
mid-event token.
- Handle the edge cases: cursor reopen mid-event, heartbeats while
reassembling, resuming from a fragment token whose event is no longer split,
unsupported MongoDB versions.
Related: FLINK-38782 reports the same 10334 error.
I have a working implementation with unit tests and Testcontainers tests on
MongoDB 8.0 (Flink 1.20 and 2.2) and would like to contribute it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
