cjw0810 commented on issue #12223:
URL: https://github.com/apache/seatunnel/issues/12223#issuecomment-5599262332
Thanks for the detailed feedback. I agree that defining a smaller and
explicit first slice would make the implementation easier to review and
maintain.
I already have a working implementation, and its current design is close to
the boundaries you suggested. For the first PR, I would propose the following
scope.
### 1. Dependency / licensing
The implementation is based on the OpenLogReplicator integration approach
already available in the Debezium Oracle connector.
I agree that OLR-specific dependencies should remain isolated and should not
affect users who continue to use LogMiner.
Before submitting the PR, I will review the dependency layout carefully and
determine whether the existing Debezium OLR implementation can be reused
directly or whether only the required OLR-specific implementation should live
inside the Oracle CDC connector.
The goal is that users who do not enable OLR should not pull in unnecessary
OLR-specific dependencies or experience any behavior change.
### 2. Source contract
Yes, my intention is to keep OLR as an adapter behind the existing
`Oracle-CDC` source rather than introducing a separate SeaTunnel source type.
Conceptually:
```text
Oracle-CDC
├── LogMiner
└── OpenLogReplicator
```
The existing snapshot split, checkpoint, and source interfaces should remain
unchanged.
The adapter can be selected explicitly through configuration such as:
```hocon
database.connection.adapter = "olr"
openlogreplicator.host = "192.168.190.249"
openlogreplicator.port = 5000
openlogreplicator.source = "ORACLE"
```
Existing LogMiner configurations should continue to work unchanged.
### 3. Snapshot / incremental consistency and recovery
This is one of the main areas I have already tested.
The intended flow is:
```text
snapshot low/high watermark
↓
initial snapshot
↓
snapshot completes
↓
incremental reader starts from the snapshot watermark SCN
↓
OLR CONTINUE(c_scn, c_idx)
```
During testing, I found an issue in the snapshot-to-incremental transition.
When OpenLogReplicator was already running, SeaTunnel sent a `CONTINUE`
request, but the requested SCN was previously written to the field used by
`START` instead of the `CONTINUE` fields.
As a result, OpenLogReplicator could receive:
```text
client requested scn: 18446744073709551615
```
instead of the expected snapshot watermark SCN.
This caused UPDATE / DELETE events generated while the initial snapshot was
running to be skipped after switching to incremental streaming.
The current implementation fixes this by sending:
```text
c_scn = requested startup SCN
c_idx = checkpoint index
```
for the `CONTINUE` request.
After this fix, I verified UPDATE and DELETE operations executed while the
initial snapshot was still running, including rows that had already been read
by the snapshot and rows that had not yet been read.
The final source and sink states remained consistent.
For the first PR, I agree that the recovery contract should also explicitly
cover:
- restart from a SeaTunnel checkpoint
- reconnect after the OLR network connection is interrupted
- behavior when the OLR process is temporarily unavailable
- continuing from the persisted SCN / checkpoint index without duplicate or
lost events
These recovery scenarios should be covered by deterministic tests before the
first PR is considered complete.
### 4. Event mapping
The current event mapping is:
```text
OLR INSERT
-> SeaTunnel INSERT
OLR UPDATE
-> SeaTunnel UPDATE_BEFORE
-> SeaTunnel UPDATE_AFTER
OLR DELETE
-> SeaTunnel DELETE
```
These mappings have already been validated with both Console Sink and Doris
Sink.
For the first PR, I suggest keeping schema evolution outside the initial
scope.
The initial scope can focus on stable DML CDC behavior first:
```text
INSERT
UPDATE
DELETE
```
For malformed or unsupported OLR records, I agree that the expected failure
behavior should be clearly defined and covered by tests rather than silently
ignoring such records.
### 5. Test boundary
I agree that requiring a real Oracle 19c + OpenLogReplicator environment in
CI would make the first PR difficult to review and maintain.
A protocol-level fake OLR endpoint sounds like a good approach.
For example:
```text
Fake OLR endpoint
↓
OlrNetworkClient
↓
SeaTunnel Oracle CDC
```
It could provide deterministic regression tests for:
- INFO / START / CONTINUE protocol handling
- propagation of SCN and checkpoint index
- INSERT / UPDATE / DELETE event mapping
- snapshot-to-incremental transition
- checkpoint recovery
- disconnect / reconnect behavior
- invalid or malformed protocol messages
This would allow the core adapter behavior to be tested in CI without
requiring Oracle 19c or a real OpenLogReplicator deployment.
The real environment can remain as environment-specific E2E coverage.
### Current validation
I already have a working implementation and have tested it with:
```text
SeaTunnel: 2.3.12
Oracle Database: 19c
OpenLogReplicator: 1.9.0
Sink: Console / Apache Doris
```
The following scenarios have been validated in the real environment:
```text
latest mode:
- INSERT
- UPDATE
- DELETE
initial mode:
- initial snapshot
- snapshot -> incremental transition
- INSERT during/after snapshot
- UPDATE during snapshot
- DELETE during snapshot
- low-key rows already read by snapshot
- high-key rows not yet read by snapshot
sink:
- Console Sink
- Doris Sink
- final Oracle / Doris consistency verification
```
For the first PR, I am happy to keep the scope focused on the adapter
contract, snapshot/incremental consistency, event mapping, recovery behavior,
and protocol-level regression tests.
RAC/ASM-specific deployment support and schema evolution can remain outside
the first PR scope and be discussed separately.
Thanks again for the detailed suggestions, and thanks for offering to help
with the adapter contract and protocol-level regression tests. That would be
very helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
