davidzollo opened a new issue, #11051: URL: https://github.com/apache/seatunnel/issues/11051
## Background SeaTunnel's `MongoDB-CDC` source currently lacks a reliable first-class startup mode for "start from now and consume incremental changes only". There is an existing user report showing that attempts to run in an incremental-only style still fall back to snapshot behavior, which makes repeat job submission unsafe for users who only want new changes. This issue is intended to track the engineering work needed to add a clear, contributor-friendly implementation target for that startup mode. ## Scope Add a `MongoDB-CDC` startup mode that: - skips the initial snapshot - starts from the latest available change-stream position - continues consuming incremental changes only This issue is about **latest-offset / incremental-only startup semantics**. It does **not** cover dynamic newly-added collection capture or new metadata fields. ## Suggested implementation approach ### 1. Make the startup mode explicit in connector-owned options - Expose the mode through `MongodbIncrementalSourceOptions` rather than relying on users to combine loosely related settings. - Keep configuration validation aligned with actual runtime support. - Update stale option descriptions so docs, validation, and behavior match. ### 2. Route startup behavior through one clear source path - When latest-offset mode is selected, the connector must skip snapshot/copy logic entirely. - The source should initialize from the latest change-stream position that matches the intended semantics of "new changes only". - Snapshot-related options should either be rejected or documented as ignored in this mode. ### 3. Preserve checkpoint / restore correctness - Startup-mode selection must flow through the same checkpoint / restore path as the existing MongoDB source. - Restart should resume from checkpoint state rather than accidentally re-entering a snapshot path. ### 4. Tests and validation Suggested test layers: - option parsing / validation tests - startup-path tests proving snapshot is skipped - restart / restore tests proving the mode remains incremental-only after recovery - doc examples in both English and Chinese ## Related history - User report: #9511 ## Suggested acceptance criteria - `MongoDB-CDC` exposes an explicit latest-offset / incremental-only startup mode. - Snapshot is skipped when this mode is selected. - Restart / restore does not silently fall back to snapshot behavior. - Option docs and runtime validation are consistent. - English and Chinese docs are updated. - Add focused tests for parsing, startup-path selection, and recovery behavior. ## Non-goals - Dynamic newly-added collection discovery. - New metadata keys. - General CDC transform enhancements. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
