DanielLeens commented on issue #11364:
URL: https://github.com/apache/seatunnel/issues/11364#issuecomment-4980846359
Thanks for continuing to refine this. After mapping the proposal more
directly onto the current engine task, checkpoint, and REST boundaries, I think
a few implementation details are worth making explicit before reworking #11395.
### 1. Separate reader and enumerator collection paths
Reader reports are produced on workers, while enumerator state is owned by
the coordinator side. The design should not assume that both can be sampled
through the same task loop.
A clearer v1 path would be:
- periodically batch reader reports from workers
- collect enumerator reports locally on the coordinator
- aggregate both by source vertex
This also gives the coordinator one clear place to calculate missing/stale
report counts.
### 2. Treat lifecycle events as refresh signals, not as the source of truth
Split-assigned, split-completed, and phase-transition events are useful for
requesting an early asynchronous refresh. However, events can be duplicated,
delayed, or lost during failure and restore.
Exact assigned/completed/running counts should therefore always be rebuilt
from the latest reader and enumerator state. Event delivery should not be
required for correctness.
### 3. Carry restore identity from the engine
The restore path should explicitly provide at least:
- restored checkpoint ID
- restored split identities and positions
- execution attempt or restore epoch
This context should reach the reader before restored splits are installed.
Connectors should not infer restore from `addSplits()`.
The report ordering key should also include the execution attempt:
```text
sourceVertexId + taskIndex + executionAttemptId + reportSequence
```
A task can be recreated after pipeline recovery while retaining the same
logical identity, and its local report sequence may restart from zero.
Comparing only task identity and sequence can allow a report from an older
attempt to suppress a new report.
### 4. Make the REST cardinality match multi-source jobs
A job may contain multiple CDC source vertices, possibly using different
connector types. A singular top-level `connectorType` cannot represent that
case. I suggest keeping the field additive in job detail, but making the source
cardinality explicit:
```json
{
"cdcProgress": {
"schemaVersion": 1,
"sources": [
{
"pipelineId": 1,
"vertexId": "source-1",
"connectorType": "MySQL-CDC",
"summary": {},
"tasks": []
}
]
}
}
```
Native positions should remain in REST only. GTIDs, LSNs, SCNs, split IDs,
and table paths should not become metric labels. Metrics should expose only
bounded, low-cardinality values such as report freshness, missing report count,
lifecycle, and no-progress duration.
### 5. Tie checkpoint positions to the real checkpoint lifecycle
```text
snapshotState(checkpointId)
-> retain a candidate position for that checkpoint
notifyCheckpointComplete(checkpointId)
-> promote that candidate to lastCompletedCheckpointPosition
notifyCheckpointAborted(checkpointId)
-> remove that candidate
```
Until this lifecycle is implemented, `lastCompletedCheckpointPosition`
should be reported as unsupported. Neither the startup position nor the latest
consumed position proves that a checkpoint completed.
### 6. Define connector capabilities per fact, not with one overall support
level
The capability matrix would be more useful if it separately described
support for:
- snapshot split progress
- current consumed position
- completed checkpoint position
- restored position
- source event time
- source head position
- exact offset lag
- time-based event delay
In particular, event delay should not be called source lag unless the
connector can also observe a comparable source-head position. MySQL binlog
position, PostgreSQL LSN, Oracle SCN, and TiDB resolved timestamp are different
position families and should remain connector-native rather than being forced
into one numeric lag contract.
These clarifications should keep the first implementation narrow while
avoiding contracts that become difficult to correct after public API exposure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
