goutamadwant commented on PR #13239:
URL: https://github.com/apache/seatunnel/pull/13239#issuecomment-6842826394

   @nzw921rx here is the proposal you asked for, using `SourceReaderMetrics` 
and `SourceEnumeratorMetrics`, with a batch example and a CDC example. No 
implementation changes until this is settled; I have converted the PR to draft.
   
   ### Summary
   
   A Source-level progress contract with two report types and no new 
interfaces. A reader or
   enumerator that can describe its progress returns an immutable snapshot from 
one default method;
   everything else keeps returning `null`. Batch and CDC sources use the same 
two shapes; what differs
   is the values they put in them. CDC is the first implementation; one batch 
source is used below to
   show the shape is not CDC-specific.
   
   Public surface (all `@Experimental`, package 
`org.apache.seatunnel.api.source`):
   
   | Type | Kind | Responsibility |
   | --- | --- | --- |
   | `SourceReader#getReaderMetrics()` | default method, returns `null` | 
Reader-owned facts, pulled by the engine |
   | `SourceSplitEnumerator#getEnumeratorMetrics()` | default method, returns 
`null` | Enumerator-owned facts, pulled by the engine |
   | `SourceReaderMetrics` | final immutable class | Phase, active split, 
consumed position, timestamps, bounded details |
   | `SourceEnumeratorMetrics` | final immutable class | Phase, split counts, 
bounded details |
   | `MetricValue<T>` + `MetricAccuracy` | final class + enum | A value and how 
much it can be trusted |
   | `SourcePosition` | final immutable class | Connector-native coordinates 
under a named family |
   
   Five types and two default methods. Compared with #13239 as pushed: no 
`SupportCdcProgress`
   marker, no `CdcProgressProvider`, no empty `CdcProgressReport`, no per-split 
detail list, no
   CDC-specific enums in the common contract.
   
   ### 1. What each component maintains and exposes
   
   ### Reader
   
   ```java
   public final class SourceReaderMetrics {
       String phase;                               // see §2
       String activeSplitId;                       // nullable before the first 
split
       MetricValue<SourcePosition> position;       // latest consumed position
       long lastPositionChangeAt;                  // epoch millis, 0 if never
       Long lastSourceEventAt;                     // source event time when 
known, else null
       Map<String, String> details;                // connector-specific, 
bounded, opaque to the engine
   }
   ```
   
   The reader reports what it has observed, not what has been checkpointed. The 
engine adds task
   identity, attempt, sequence and observation time when it collects.
   
   ### Enumerator
   
   ```java
   public final class SourceEnumeratorMetrics {
       String phase;                               // see §2
       MetricValue<Integer> pendingSplitCount;     // created, not yet assigned
       MetricValue<Integer> assignedSplitCount;    // handed to a reader 
(running + completed)
       MetricValue<Integer> runningSplitCount;     // assigned, completion not 
yet known
       MetricValue<Integer> completedSplitCount;   // assigned and reported 
complete
       MetricValue<Integer> unsplitUnitCount;      // discovered 
tables/files/partitions not yet split
       Map<String, String> details;
   }
   ```
   
   The enumerator reports only what it knows. Many batch enumerators never 
learn that a split
   finished; they report `completedSplitCount` and `runningSplitCount` as 
`UNSUPPORTED` instead of
   guessing.
   
   Both classes validate in the constructor: present counts are non-negative; 
`assigned ==
   completed + running` is enforced only when all three are `EXACT`; `details` 
is copied and capped
   (proposed: 32 entries, 256 characters per key and value); `phase` and 
`position` are non-null.
   
   ### 2. Common semantics versus implementation-specific
   
   | Common (fixed by the contract) | Implementation-specific (carried as 
values) |
   | --- | --- |
   | The two shapes above and their field meanings | Which `phase` strings a 
source uses beyond the common ones |
   | `MetricAccuracy` and the rule that `EXACT` is only claimed when runtime 
state proves it | The `SourcePosition` family and its keys (`MYSQL_BINLOG`, 
`POSTGRES_LSN`, `FILE`, ...) |
   | Common phases: `UNKNOWN` (no split yet), `READING`, `COMPLETED` (no more 
splits and no more elements) for readers; `DISCOVERING`, `ASSIGNING`, 
`COMPLETED`, `NOT_APPLICABLE` for enumerators | Finer reader phases: CDC uses 
`SNAPSHOT`, `CATCH_UP`, `INCREMENTAL` in place of `READING` |
   | Non-blocking, thread-safe, immutable, truthful provider guarantees (§3, 
§4) | Anything in `details` |
   
   Phases are strings with the common values as constants on the two classes. A 
consumer that does
   not know a connector-specific phase treats it as `READING`. An enum would be 
safer for consumers
   but would put CDC-specific values into a Source-level type or force a second 
enum per connector
   family; the string keeps the common type small. If the committers prefer an 
enum for the common
   phases plus a free-form `detailPhase`, that is a mechanical change.
   
   Position families are strings too, for the same reason. The contract fixes 
only that a position
   carries a family, a schema version and string coordinates, that payloads 
never contain
   credentials or connection details, and that consumers compare positions 
within one family only.
   
   ### 3. Unsupported or temporarily unavailable information
   
   Every count and the position are a `MetricValue<T>`:
   
   | Accuracy | Payload | Meaning |
   | --- | --- | --- |
   | `EXACT` | non-null | Current runtime state proves the value |
   | `BEST_EFFORT` | non-null | Useful for diagnosis, precision not guaranteed |
   | `UNSUPPORTED` | null | This source cannot provide the value |
   | `UNAVAILABLE` | null | Supported, but not available for this observation |
   
   A missing value is never replaced by zero or a guess. Nullable scalars 
(`activeSplitId`,
   `lastSourceEventAt`) are plain nullables because "unknown" is their only 
non-value. Returning
   `null` from the default method means "no report yet"; the engine does not 
record anything.
   
   ### 4. How the engine collects without blocking processing
   
   Unchanged from #11512, with names updated:
   
   - Both methods return a snapshot the component already maintains. They never 
perform source,
     network, checkpoint or other blocking I/O and never take a lock that 
record processing or split
     assignment holds. Implementations publish a new immutable instance on 
change (volatile field);
     the engine reads the field.
   - On each worker, `TaskExecutionService` samples `getReaderMetrics()` on the 
existing
     `print-execution-info-interval` cadence, wraps non-null results in an 
envelope (task location,
     source vertex, execution attempt, attempt-local sequence, observed-at) and 
sends one batch per
     worker to the coordinator. A throwing implementation is isolated per task 
and logged by class
     name only.
   - The coordinator asks each worker that hosts a split enumerator for 
`getEnumeratorMetrics()` on
     the same cadence, at most one request per worker in flight. Enumerators 
returning `null` add
     nothing to the batch.
   - The coordinator keeps the latest report per owner and task; lower attempts 
and non-increasing
     sequences are ignored; reports are removed with their pipeline.
   - Wire format: one fixed schema per report type, enums and phases by name, 
bounded entry and
     field counts. Because the shapes are fixed, the engine never deserializes 
connector classes.
   
   Nothing is exposed through REST, metrics or the CLI in this phase.
   
   ### 5. Batch example: Local file source
   
   Enumerator (`FileSourceSplitEnumerator` knows the file list and the assigned 
set; it is not told
   when a reader finishes a file):
   
   ```java
   @Override
   public SourceEnumeratorMetrics getEnumeratorMetrics() {
       int total = filePaths.size();
       int assigned = assignedSplit.size();
       return new SourceEnumeratorMetrics(
               assigned == total ? SourceEnumeratorMetrics.PHASE_COMPLETED
                                 : SourceEnumeratorMetrics.PHASE_ASSIGNING,
               MetricValue.exact(total - assigned),      // pending
               MetricValue.exact(assigned),              // assigned
               MetricValue.unsupported(),                // running: enumerator 
is not told
               MetricValue.unsupported(),                // completed: 
enumerator is not told
               MetricValue.exact(0),                     // unsplit units: 
files are the splits
               Collections.emptyMap());
   }
   ```
   
   Reader (one file at a time; rows emitted since the file was opened are 
exact):
   
   ```java
   @Override
   public SourceReaderMetrics getReaderMetrics() {
       return latest; // volatile; rebuilt when a file is opened, every N rows, 
and on completion
   }
   
   private void refreshMetrics(String path, long rowsRead, boolean done) {
       Map<String, String> coords = new LinkedHashMap<>();
       coords.put("path", path);
       coords.put("rows", Long.toString(rowsRead));
       latest = new SourceReaderMetrics(
               done ? SourceReaderMetrics.PHASE_COMPLETED : 
SourceReaderMetrics.PHASE_READING,
               path,
               MetricValue.exact(new SourcePosition("FILE", 1, coords)),
               System.currentTimeMillis(),
               null,
               Collections.emptyMap());
   }
   ```
   
   What this proves: the shapes hold without any CDC concept; a batch 
enumerator can be honest about
   not knowing completion; a batch reader's position is a split-local offset 
rather than a stream
   offset, and `SourcePosition` carries both.
   
   ### 6. CDC example: `connector-cdc-base`
   
   Enumerator (`SnapshotSplitAssigner` already keeps an immutable count 
snapshot; the incremental
   assigner has no snapshot phase):
   
   | Startup mode | Enumerator phase | pending / assigned / running / completed 
/ unsplit |
   | --- | --- | --- |
   | `initial`, `snapshot-only` | `DISCOVERING` → `ASSIGNING` → `COMPLETED` | 
all `EXACT` while counts are consistent, `BEST_EFFORT` otherwise; `unsplit` = 
tables not yet chunked |
   | `earliest`, `latest`, `specific`, `timestamp`, `committed-offset` | 
`NOT_APPLICABLE` | all `UNSUPPORTED` |
   
   Reader (`CdcReaderProgressTracker` derives the phase from the split it is 
processing):
   
   | Startup mode | Reader phase sequence | position before → after the first 
record |
   | --- | --- | --- |
   | `initial` | `UNKNOWN` → `SNAPSHOT` → `CATCH_UP` → `INCREMENTAL`; idle 
readers → `COMPLETED` | `UNAVAILABLE` → `BEST_EFFORT` (min snapshot watermark) 
→ `EXACT` |
   | `snapshot-only` | `UNKNOWN` → `SNAPSHOT` → `COMPLETED` | `UNAVAILABLE` 
throughout |
   | incremental-only modes | `UNKNOWN` → `INCREMENTAL` | `BEST_EFFORT` 
(configured start) → `EXACT` |
   
   `SNAPSHOT`, `CATCH_UP` and `INCREMENTAL` are CDC's refinement of `READING`. 
Position families:
   `MYSQL_BINLOG` (file, pos, gtids), others use the plugin name until a family 
is defined. The
   `details` map stays empty in v1.
   
   ### 7. Scope and compatibility
   
   - Default methods: no existing connector or engine changes behaviour. A 
connector that does not
     override them is simply not observed.
   - CDC (`connector-cdc-base`, MySQL family) is the only implementation in 
this STIP. The file
     example above is a design proof; implementing batch connectors is separate 
work, if wanted.
   - The contract stays `@Experimental` until at least one more source family 
(batch or a second CDC
     dialect such as PostgreSQL) has adopted it without changing the shapes.
   - Naming: the types carry structured progress, not numeric counters, so they 
do not go through
     `MetricsContext`; the doc will state that distinction. If 
`SourceReaderProgress` /
     `SourceEnumeratorProgress` reads clearer alongside `MetricsContext`, the 
rename is cheap now and
     expensive later.
   - STIP-30 (#11364) becomes "CDC as the first implementation of the Source 
progress contract";
     its startup-mode semantics, accuracy rules and engine collection design 
are unchanged.
   
   ### 8. Changes to the open PRs if this is accepted
   
   - #13239 (API): rename and move the types, drop the two provider interfaces 
in favour of the
     default methods, replace the CDC lifecycle and assignment enums with phase 
constants, add the
     bounded `details` map, add the batch example to the doc.
   - #13240 (connector): adopt the default methods; phase strings instead of 
enums.
   - #11512 (engine): rename, codec gains the `details` block, otherwise 
unchanged.
   - #12137 (MySQL E2E) follows after the three merge.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to