goutamadwant commented on PR #13239:
URL: https://github.com/apache/seatunnel/pull/13239#issuecomment-7002158576
@nzw921rx thanks, that's all fair. Taking the four points in order, then the
two examples you asked for.
**1. Completion.** Agreed. A reader is `COMPLETED` only after it has the
no-more-splits signal and has drained everything it holds; finishing one file
while another waits is still `READING`. An enumerator is `COMPLETED` when
discovery and assignment are done, and that says nothing about readers, the
in-flight count carries that. `SNAPSHOT` / `CATCH_UP` / `INCREMENTAL` only
refine `READING` on the reader side.
**2. Counts.** I'll rename `runningSplitCount` to `inFlightSplitCount`,
defined as assigned minus completed, since the enumerator only knows it hasn't
heard back. Each count maps to something the enumerator already owns: `pending`
is the set behind `currentUnassignedSplitSize()`, `assigned` is the assigned
set keyed by split id, `completed` is completion reports received,
`unsplitUnits` is discovered tables/files not yet split. On `addSplitsBack()`
the split goes back to pending and leaves in-flight; because everything is
keyed by split id, reassigning it doesn't bump `assigned`. Where a connector
re-creates the split under a new id, as the CDC incremental assigner does, the
old id drops out and the new one comes in. You're right about incremental-only
CDC too: it does assign splits, so `pending`, `assigned` and `inFlight` should
be exact there, with only `completed` as `UNSUPPORTED`. My first draft had all
of them unsupported, which was wrong. And the file example should not coun
t `filePaths`; it should use the same sets as `currentUnassignedSplitSize()`.
**3. Position.** `activeSplitId` is the split that produced the position,
and the two always change together in one snapshot. Only successfully emitted
records advance the position and `lastPositionChangeAt`; heartbeats and
filtered events don't. That way an idle source shows an unchanged position with
an old timestamp, which is the thing an operator actually wants to see.
`lastSourceEventAt` is that record's source time, null if it has none. The
current tracker keeps the previous value in that case; I'll fix that. Zeta
exposes the coordinates as supplied, and checkpoint state stays the only
recovery position.
**4. Read path.** The record path updates local state and publishes an
immutable snapshot through one volatile field, at most once per interval or on
a split/phase change; the sampler reads the field and never locks the reader or
walks the enumerator's collections. One thing I'm adding: publish on completion
and on idle as well, otherwise a record consumed right after the last periodic
publish gets reported as the final `EXACT` position for the rest of the job.
`lastPositionChangeAt` moves only when the position moves.
### File source, one reader with two splits
Enumerator side, built on the assignment path and published through a
volatile field so the getter never reads the live sets:
```java
private volatile SourceEnumeratorMetrics metrics =
SourceEnumeratorMetrics.unknown();
private void publishMetrics() { // end of assignSplit() and
addSplitsBack()
int assigned = assignedSplit.size();
int pending = allSplit.size() - assigned; // ==
currentUnassignedSplitSize()
metrics = new SourceEnumeratorMetrics(
pending == 0 ? PHASE_COMPLETED : PHASE_ASSIGNING,
MetricValue.exact(pending), MetricValue.exact(assigned),
MetricValue.exact(assigned), // inFlight: files never
report completion
MetricValue.unsupported(), // completed
MetricValue.exact(0), // files are the units
Collections.emptyMap());
}
```
Reader given `a.csv` then `b.csv`:
| Event | phase | activeSplitId | position `FILE` | lastPositionChangeAt |
| --- | --- | --- | --- | --- |
| `addSplits([a, b])`, nothing read | `UNKNOWN` | null | `UNAVAILABLE` | 0 |
| first row of `a` emitted | `READING` | `a.csv` | `EXACT {path=a.csv,
rows=1}` | t1 |
| `a` done, `b` opened, no row yet | `READING` | `b.csv` | `EXACT
{path=a.csv, rows=1200}` | unchanged |
| rows of `b` emitted | `READING` | `b.csv` | `EXACT {path=b.csv, rows=n}` |
advances |
| no-more-splits, deque empty | `COMPLETED` | `b.csv` | `EXACT {path=b.csv,
rows=800}`, flushed on completion | unchanged |
Row 3 is the consequence of point 3: the position names the last emitted
record, so for a moment after a split switch the split id and the position's
path differ. The alternative is to clear the position to `UNAVAILABLE` on the
switch. I lean towards keeping the last record, but happy either way.
### MySQL CDC, incremental split returned and reassigned
`initial` mode, parallelism 2, one table. Enumerator:
| Event | phase | pending | assigned | inFlight | completed | unsplit |
| --- | --- | --- | --- | --- | --- | --- |
| 4 snapshot chunks, 2 assigned | `ASSIGNING` | 2 | 2 | 2 | 0 | 0 |
| all 4 complete | `COMPLETED` | 0 | 4 | 0 | 4 | 0 |
| snapshot phase retired, incremental split assigned to reader 1 |
`COMPLETED` | 0 | 1 | 1 | `UNSUPPORTED` | 0 |
| reader 1 fails, `addSplitsBack([incremental])` | `ASSIGNING` | 1 | 0 | 0 |
`UNSUPPORTED` | 0 |
| reader 2 asks, split re-created and assigned | `COMPLETED` | 0 | 1 | 1 |
`UNSUPPORTED` | 0 |
Two things this makes explicit that my first draft got wrong: snapshot
counts are retired together with the snapshot phase, the doc said "retained"
but the code never did that, so the doc will follow the code; and `completed`
turns `UNSUPPORTED` rather than 0 once only change-stream splits exist.
Reader 2 after the reassignment reports `INCREMENTAL` with the re-created
split id and a `BEST_EFFORT {file, pos}` position from the restored startup
offset, which becomes `EXACT {file, pos, gtids}` on the first emitted change
record. It skips `CATCH_UP` because the restored split's startup offset already
equals the maximum snapshot high watermark; a split that still has watermarks
to pass would go through `CATCH_UP` first.
So the changes from the first draft are: `running` → `inFlight`, the
one-count-per-split-id rule with `addSplitsBack()` spelled out, exact counts
for incremental-only CDC, position only on emitted records with
`lastSourceEventAt` per record, publish on completion and idle, and enumerator
examples that publish a snapshot. Phases, shapes and the five types stay as
they were.
If this looks right to you I'll rework #13239 to these types and bring
#13240 and #11512 along.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]