goutamadwant commented on PR #13239:
URL: https://github.com/apache/seatunnel/pull/13239#issuecomment-7002158576

   @nzw921rx thanks, that's all fair. Taking the four points in order, then the 
two examples you asked for.
   
   **1. Completion.** Agreed. A reader is `COMPLETED` only after it has the 
no-more-splits signal and has drained everything it holds; finishing one file 
while another waits is still `READING`. An enumerator is `COMPLETED` when 
discovery and assignment are done, and that says nothing about readers, the 
in-flight count carries that. `SNAPSHOT` / `CATCH_UP` / `INCREMENTAL` only 
refine `READING` on the reader side.
   
   **2. Counts.** I'll rename `runningSplitCount` to `inFlightSplitCount`, 
defined as assigned minus completed, since the enumerator only knows it hasn't 
heard back. Each count maps to something the enumerator already owns: `pending` 
is the set behind `currentUnassignedSplitSize()`, `assigned` is the assigned 
set keyed by split id, `completed` is completion reports received, 
`unsplitUnits` is discovered tables/files not yet split. On `addSplitsBack()` 
the split goes back to pending and leaves in-flight; because everything is 
keyed by split id, reassigning it doesn't bump `assigned`. Where a connector 
re-creates the split under a new id, as the CDC incremental assigner does, the 
old id drops out and the new one comes in. You're right about incremental-only 
CDC too: it does assign splits, so `pending`, `assigned` and `inFlight` should 
be exact there, with only `completed` as `UNSUPPORTED`. My first draft had all 
of them unsupported, which was wrong. And the file example should not coun
 t `filePaths`; it should use the same sets as `currentUnassignedSplitSize()`.
   
   **3. Position.** `activeSplitId` is the split that produced the position, 
and the two always change together in one snapshot. Only successfully emitted 
records advance the position and `lastPositionChangeAt`; heartbeats and 
filtered events don't. That way an idle source shows an unchanged position with 
an old timestamp, which is the thing an operator actually wants to see. 
`lastSourceEventAt` is that record's source time, null if it has none. The 
current tracker keeps the previous value in that case; I'll fix that. Zeta 
exposes the coordinates as supplied, and checkpoint state stays the only 
recovery position.
   
   **4. Read path.** The record path updates local state and publishes an 
immutable snapshot through one volatile field, at most once per interval or on 
a split/phase change; the sampler reads the field and never locks the reader or 
walks the enumerator's collections. One thing I'm adding: publish on completion 
and on idle as well, otherwise a record consumed right after the last periodic 
publish gets reported as the final `EXACT` position for the rest of the job. 
`lastPositionChangeAt` moves only when the position moves.
   
   ### File source, one reader with two splits
   
   Enumerator side, built on the assignment path and published through a 
volatile field so the getter never reads the live sets:
   
   ```java
   private volatile SourceEnumeratorMetrics metrics = 
SourceEnumeratorMetrics.unknown();
   
   private void publishMetrics() {               // end of assignSplit() and 
addSplitsBack()
       int assigned = assignedSplit.size();
       int pending = allSplit.size() - assigned; // == 
currentUnassignedSplitSize()
       metrics = new SourceEnumeratorMetrics(
               pending == 0 ? PHASE_COMPLETED : PHASE_ASSIGNING,
               MetricValue.exact(pending), MetricValue.exact(assigned),
               MetricValue.exact(assigned),      // inFlight: files never 
report completion
               MetricValue.unsupported(),         // completed
               MetricValue.exact(0),              // files are the units
               Collections.emptyMap());
   }
   ```
   
   Reader given `a.csv` then `b.csv`:
   
   | Event | phase | activeSplitId | position `FILE` | lastPositionChangeAt |
   | --- | --- | --- | --- | --- |
   | `addSplits([a, b])`, nothing read | `UNKNOWN` | null | `UNAVAILABLE` | 0 |
   | first row of `a` emitted | `READING` | `a.csv` | `EXACT {path=a.csv, 
rows=1}` | t1 |
   | `a` done, `b` opened, no row yet | `READING` | `b.csv` | `EXACT 
{path=a.csv, rows=1200}` | unchanged |
   | rows of `b` emitted | `READING` | `b.csv` | `EXACT {path=b.csv, rows=n}` | 
advances |
   | no-more-splits, deque empty | `COMPLETED` | `b.csv` | `EXACT {path=b.csv, 
rows=800}`, flushed on completion | unchanged |
   
   Row 3 is the consequence of point 3: the position names the last emitted 
record, so for a moment after a split switch the split id and the position's 
path differ. The alternative is to clear the position to `UNAVAILABLE` on the 
switch. I lean towards keeping the last record, but happy either way.
   
   ### MySQL CDC, incremental split returned and reassigned
   
   `initial` mode, parallelism 2, one table. Enumerator:
   
   | Event | phase | pending | assigned | inFlight | completed | unsplit |
   | --- | --- | --- | --- | --- | --- | --- |
   | 4 snapshot chunks, 2 assigned | `ASSIGNING` | 2 | 2 | 2 | 0 | 0 |
   | all 4 complete | `COMPLETED` | 0 | 4 | 0 | 4 | 0 |
   | snapshot phase retired, incremental split assigned to reader 1 | 
`COMPLETED` | 0 | 1 | 1 | `UNSUPPORTED` | 0 |
   | reader 1 fails, `addSplitsBack([incremental])` | `ASSIGNING` | 1 | 0 | 0 | 
`UNSUPPORTED` | 0 |
   | reader 2 asks, split re-created and assigned | `COMPLETED` | 0 | 1 | 1 | 
`UNSUPPORTED` | 0 |
   
   Two things this makes explicit that my first draft got wrong: snapshot 
counts are retired together with the snapshot phase, the doc said "retained" 
but the code never did that, so the doc will follow the code; and `completed` 
turns `UNSUPPORTED` rather than 0 once only change-stream splits exist.
   
   Reader 2 after the reassignment reports `INCREMENTAL` with the re-created 
split id and a `BEST_EFFORT {file, pos}` position from the restored startup 
offset, which becomes `EXACT {file, pos, gtids}` on the first emitted change 
record. It skips `CATCH_UP` because the restored split's startup offset already 
equals the maximum snapshot high watermark; a split that still has watermarks 
to pass would go through `CATCH_UP` first.
   
   So the changes from the first draft are: `running` → `inFlight`, the 
one-count-per-split-id rule with `addSplitsBack()` spelled out, exact counts 
for incremental-only CDC, position only on emitted records with 
`lastSourceEventAt` per record, publish on completion and idle, and enumerator 
examples that publish a snapshot. Phases, shapes and the five types stay as 
they were.
   
   If this looks right to you I'll rework #13239 to these types and bring 
#13240 and #11512 along.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to