li3zhi4 opened a new issue, #11739: URL: https://github.com/apache/seatunnel/issues/11739
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues) and found no existing issue tracking this audit: - No issue reports that other CDC connectors (Oracle / SQL Server / PostgreSQL / MongoDB) should support `stop.mode` bounded reads. - The closest related work is [PR #11618](https://github.com/apache/seatunnel/pull/11618) (MySQL CDC `stop.mode = "specific"` bounded read) and its linked issue [#11617](https://github.com/apache/seatunnel/issues/11617). - [#11364](https://github.com/apache/seatunnel/issues/11364) (STIP-30 unified CDC lag/progress observability) is unrelated — it is about metrics, not bounded termination. ### Motivation / Background [PR #11618](https://github.com/apache/seatunnel/pull/11618) implemented MySQL CDC `stop.mode = "specific"` (bounded read): the incremental split can now reach a genuine terminal state, a bounded MySQL binlog streaming source checks the current offset against the configured stop offset and dispatches an END watermark, and the shared reader drains its queue before signaling completion. As part of that PR, the "never stop" sentinel concept was generalized into an explicit `Offset.isNeverStop()` contract implemented by every CDC connector that shares the `IncrementalSourceStreamFetcher` reader (MySQL, PostgreSQL, SQL Server, Oracle, MongoDB) so unbounded connectors can never be accidentally marked finished. However, the option rules of the other `FetchTask`-based CDC connectors (**Oracle, SQL Server, PostgreSQL**) still only expose `stop.mode = "never"` today — they cannot yet use `stop.mode = "specific"` / bounded reads even though the shared reader-side infrastructure (queue drain, `isBoundedReadFinished()`, `isNeverStop()`) is now in place. ### Goal of this audit Track a systematic audit of extending `stop.mode` (bounded read) support to the other `FetchTask`-based CDC connectors, and to keep the findings visible for anyone extending stop-mode support there later: - Oracle: `OracleRedoLogFetchTask` — assess whether a bounded redo-log streaming source (similar to `BoundedMySqlStreamingChangeEventSource`) can check the stop offset in the `finally` block and dispatch an END watermark. - SQL Server: `SqlServerRedoLogFetchTask` — same assessment against `LsnOffset` stop offsets. - PostgreSQL: `PostgresWalFetchTask` — same assessment against `LsnOffset` stop offsets (note: PostgreSQL redo/consumption semantics differ from MySQL binlog, needs extra care). - MongoDB: `MongoDBFetchTask` — assess `ChangeStreamOffset` based bounded read. - Option rules: add `stop.mode = "specific"` (and possibly `"latest"`) to each connector's option rule set, mirroring the MySQL implementation. - E2E: add a bounded-read e2e test per connector (following `MysqlCDCStopModeSpecificIT`). Each connector should be checked for: 1. whether its fetch task can route a bounded split into a bounded streaming source (analogous to `MySqlBinlogFetchTask.execute()`), 2. whether `checkStopOffset`-style logic (dispatch END watermark + stop the wrapped context) is feasible, 3. whether the connector's offset `compareTo`/`isAtOrAfter` semantics are correct for stop-offset comparison (the MySQL PR fixed a mixed-GTID comparison fallback; other connectors may have similar pitfalls), 4. e2e coverage asserting the job terminates (FINISHED) at the configured stop offset and only pre-offset rows are synced. ### Expected outcome A per-connector feasibility matrix (bounded-read supportable / needs redesign / blocked), plus follow-up implementation PRs for the connectors where it is straightforward. ### Related - PR: https://github.com/apache/seatunnel/pull/11618 (MySQL CDC `stop.mode = "specific"` bounded read; generalized `isNeverStop()` contract) - Issue: https://github.com/apache/seatunnel/issues/11617 - Proposed by maintainers during review of #11618: audit the other `FetchTask`-based CDC connectors and track it in an issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
