cjw0810 opened a new issue, #12223: URL: https://github.com/apache/seatunnel/issues/12223
### Search before asking - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description ## Background The current SeaTunnel Oracle CDC connector mainly relies on Oracle LogMiner for change data capture. OpenLogReplicator (OLR) is an open-source Oracle redo log parser that reads and parses Oracle redo and archive logs directly. The Debezium Oracle Connector already provides an OpenLogReplicator ingestion adapter through: ```properties database.connection.adapter=olr ``` and connects to the OpenLogReplicator network endpoint using properties such as: ```properties openlogreplicator.source openlogreplicator.host openlogreplicator.port ``` SeaTunnel Oracle CDC currently does not expose OpenLogReplicator as an available connection adapter. This feature proposes integrating the existing Debezium OpenLogReplicator adapter into SeaTunnel Oracle CDC, while keeping the existing LogMiner implementation unchanged. --- ## Motivation OpenLogReplicator directly parses Oracle redo logs outside the Oracle database and streams change events to downstream clients. Supporting OLR in SeaTunnel would provide an additional Oracle CDC ingestion option for environments where redo logs are accessible to OpenLogReplicator. Compared with relying only on LogMiner, this provides users with another CDC architecture that can: - Reduce dependency on database-side LogMiner processing - Reduce Oracle-side CDC processing overhead in suitable deployment environments - Provide another option for high-throughput Oracle CDC workloads - Reuse the OpenLogReplicator integration already available in Debezium Oracle Connector - Keep compatibility with SeaTunnel's existing snapshot and incremental CDC framework The goal is not to replace LogMiner, but to provide an additional Oracle CDC adapter that users can choose based on their deployment environment. --- ## Proposed Configuration The proposed configuration follows the existing Debezium Oracle Connector adapter model. Example: ```hocon source { Oracle-CDC { database.connection.adapter = "olr" openlogreplicator.host = "192.168.190.249" openlogreplicator.port = 5000 openlogreplicator.source = "ORACLE" } } ``` The main new configuration options are: ```text database.connection.adapter openlogreplicator.host openlogreplicator.port openlogreplicator.source ``` When: ```text database.connection.adapter = "olr" ``` SeaTunnel Oracle CDC uses OpenLogReplicator as the incremental redo-log ingestion adapter. The existing LogMiner behavior remains unchanged when OLR is not configured. --- ## Proposed Implementation The implementation adds OpenLogReplicator support to the Oracle CDC connector and integrates it with SeaTunnel's existing CDC source framework. The main implementation includes: - OpenLogReplicator network client integration - OpenLogReplicator protocol request and response handling - Incremental redo change event streaming - INSERT event support - UPDATE_BEFORE / UPDATE_AFTER event support - DELETE event support - `latest` startup mode - `initial` snapshot followed by incremental streaming - SCN-based startup position - Checkpoint recovery - Snapshot-to-incremental transition - Compatibility with SeaTunnel snapshot split processing The implementation reuses the Debezium OpenLogReplicator integration where possible instead of implementing a new Oracle redo parser inside SeaTunnel. --- ## Initial Snapshot and Incremental Streaming One important part of the implementation is correctly handling change events that occur while an initial snapshot is still running. The expected workflow is: ```text Initial Snapshot | | concurrent INSERT / UPDATE / DELETE | v Snapshot completed | v Continue incremental streaming from snapshot watermark SCN | v No CDC events should be lost ``` During testing, an issue was found in the transition from snapshot mode to incremental streaming. When OpenLogReplicator was already running, the incremental reader sent a `CONTINUE` request. The request must carry: ```text c_scn = requested startup SCN c_idx = checkpoint index ``` instead of using the SCN field that is only used by the `START` request. Without this handling, OpenLogReplicator could receive an invalid SCN value instead of the expected incremental startup SCN. For example, before the fix, OpenLogReplicator could receive: ```text client requested scn: 18446744073709551615 ``` instead of the actual snapshot watermark SCN. This caused UPDATE / DELETE events generated during the initial snapshot phase to be skipped when switching to incremental streaming. After correcting the `CONTINUE` request, OpenLogReplicator receives the expected SCN and checkpoint index, and the CDC events generated during the snapshot phase are correctly replayed after the snapshot completes. --- ## Supported Startup Modes The current implementation has been validated with: ### latest Start directly from the current Oracle redo position and continuously capture: - INSERT - UPDATE - DELETE ### initial Perform a full snapshot first, and then automatically switch to incremental redo streaming. The implementation also handles DML operations occurring during the snapshot window. For example: ```text Snapshot is reading existing rows At the same time: UPDATE low-key row DELETE low-key row UPDATE high-key row DELETE high-key row Snapshot completes Incremental streaming starts from the corresponding watermark SCN The final target state remains consistent with Oracle ``` --- ## Validation The current implementation has been tested with the following environment: ```text SeaTunnel: 2.3.12 Oracle Database: Oracle 19c OpenLogReplicator: 1.9.0 Sink: Console / Apache Doris ``` The following scenarios have been validated: - OLR network connection - `latest` mode startup - `latest` mode INSERT - `latest` mode UPDATE - `latest` mode DELETE - `initial` snapshot - Initial snapshot followed by incremental streaming - INSERT executed after snapshot startup - UPDATE executed while snapshot is still running - DELETE executed while snapshot is still running - UPDATE on rows already read by snapshot - UPDATE on rows not yet read by snapshot - DELETE on rows already read by snapshot - DELETE on rows not yet read by snapshot - Snapshot watermark SCN transition - OLR `CONTINUE` request using SCN and checkpoint index - Console Sink validation - Doris Sink validation - Final Oracle / Doris data consistency verification For the tested scenarios, the final Oracle and Doris data are consistent. --- ## Compatibility This feature is intended to be backward compatible. Existing Oracle CDC configurations that do not enable the OLR adapter continue to use the existing behavior. OpenLogReplicator is enabled only when explicitly configured, for example: ```hocon database.connection.adapter = "olr" ``` Therefore, existing LogMiner users should not be affected by this change. --- ## Initial Scope The initial implementation focuses on Oracle environments where OpenLogReplicator can directly access the required Oracle redo and archive log files. The following deployment scenarios are not part of the initial scope: - RAC + ASM specific redo access - Oracle Downstream Mining - Direct ASM redo access from a remote OpenLogReplicator deployment - Automatic RAC/ASM redo transport management These scenarios involve additional deployment and storage considerations and can be discussed separately. The first goal of this feature is to provide a stable OpenLogReplicator ingestion adapter for the standard Oracle CDC workflow. --- ## References - Debezium Oracle Connector OpenLogReplicator ingestion adapter - OpenLogReplicator project ### Usage Scenario OpenLogReplicator can be used as an alternative Oracle CDC adapter when users want to capture changes by directly parsing Oracle redo logs instead of relying only on the existing LogMiner-based path. A typical deployment is: Oracle Database | | redo logs v OpenLogReplicator | | streaming protocol v SeaTunnel Oracle CDC Source | v Doris / Kafka / other sinks ### Related issues None. ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
