退订
At 2026-11-04 20:26:00, "Sonal Maheshwari (Jira) via dev" <[email protected]> wrote: >Sonal Maheshwari created FLINK-41352: >---------------------------------------- > > Summary: [mongodb] Support change events larger than 16MB with > $changeStreamSplitLargeEvent > Key: FLINK-41352 > URL: https://issues.apache.org/jira/browse/FLINK-41352 > Project: Flink > Issue Type: Improvement > Components: Flink CDC > Affects Versions: cdc-3.7.0 > Reporter: Sonal Maheshwari > > >When a change event is larger than 16MB, e.g. an update whose looked-up full >document ({{fullDocument: updateLookup}}) plus update description exceed the >limit, MongoDB fails the getMore with BSONObjectTooLarge (10334). > >{{MongoDBStreamFetchTask}} treats 10334 as an invalid change stream and >reopens the cursor from the same resume token, which hits the same event >again. The stream never moves past it and the lag keeps growing. > >MongoDB 6.0.9+ provides the {{$changeStreamSplitLargeEvent}} stage, which >splits such events into fragments of at most 16MB. Proposal: > >- Add an opt-in option {{scan.split-large-event.enabled}} (default false) that >appends the stage to the change stream pipeline. >- Reassemble the fragments in the stream fetch task into one change event that >carries the resume token of the last fragment, so checkpoints never store a >mid-event token. >- Handle the edge cases: cursor reopen mid-event, heartbeats while >reassembling, resuming from a fragment token whose event is no longer split, >unsupported MongoDB versions. > >Related: FLINK-38782 reports the same 10334 error. > >I have a working implementation with unit tests and Testcontainers tests on >MongoDB 8.0 (Flink 1.20 and 2.2) and would like to contribute it. > > > >-- >This message was sent by Atlassian Jira >(v10.3.26#10030026)
