goutamadwant commented on PR #13239:
URL: https://github.com/apache/seatunnel/pull/13239#issuecomment-6842826394
@nzw921rx here is the proposal you asked for, using `SourceReaderMetrics`
and `SourceEnumeratorMetrics`, with a batch example and a CDC example. No
implementation changes until this is settled; I have converted the PR to draft.
### Summary
A Source-level progress contract with two report types and no new
interfaces. A reader or
enumerator that can describe its progress returns an immutable snapshot from
one default method;
everything else keeps returning `null`. Batch and CDC sources use the same
two shapes; what differs
is the values they put in them. CDC is the first implementation; one batch
source is used below to
show the shape is not CDC-specific.
Public surface (all `@Experimental`, package
`org.apache.seatunnel.api.source`):
| Type | Kind | Responsibility |
| --- | --- | --- |
| `SourceReader#getReaderMetrics()` | default method, returns `null` |
Reader-owned facts, pulled by the engine |
| `SourceSplitEnumerator#getEnumeratorMetrics()` | default method, returns
`null` | Enumerator-owned facts, pulled by the engine |
| `SourceReaderMetrics` | final immutable class | Phase, active split,
consumed position, timestamps, bounded details |
| `SourceEnumeratorMetrics` | final immutable class | Phase, split counts,
bounded details |
| `MetricValue<T>` + `MetricAccuracy` | final class + enum | A value and how
much it can be trusted |
| `SourcePosition` | final immutable class | Connector-native coordinates
under a named family |
Five types and two default methods. Compared with #13239 as pushed: no
`SupportCdcProgress`
marker, no `CdcProgressProvider`, no empty `CdcProgressReport`, no per-split
detail list, no
CDC-specific enums in the common contract.
### 1. What each component maintains and exposes
### Reader
```java
public final class SourceReaderMetrics {
String phase; // see §2
String activeSplitId; // nullable before the first
split
MetricValue<SourcePosition> position; // latest consumed position
long lastPositionChangeAt; // epoch millis, 0 if never
Long lastSourceEventAt; // source event time when
known, else null
Map<String, String> details; // connector-specific,
bounded, opaque to the engine
}
```
The reader reports what it has observed, not what has been checkpointed. The
engine adds task
identity, attempt, sequence and observation time when it collects.
### Enumerator
```java
public final class SourceEnumeratorMetrics {
String phase; // see §2
MetricValue<Integer> pendingSplitCount; // created, not yet assigned
MetricValue<Integer> assignedSplitCount; // handed to a reader
(running + completed)
MetricValue<Integer> runningSplitCount; // assigned, completion not
yet known
MetricValue<Integer> completedSplitCount; // assigned and reported
complete
MetricValue<Integer> unsplitUnitCount; // discovered
tables/files/partitions not yet split
Map<String, String> details;
}
```
The enumerator reports only what it knows. Many batch enumerators never
learn that a split
finished; they report `completedSplitCount` and `runningSplitCount` as
`UNSUPPORTED` instead of
guessing.
Both classes validate in the constructor: present counts are non-negative;
`assigned ==
completed + running` is enforced only when all three are `EXACT`; `details`
is copied and capped
(proposed: 32 entries, 256 characters per key and value); `phase` and
`position` are non-null.
### 2. Common semantics versus implementation-specific
| Common (fixed by the contract) | Implementation-specific (carried as
values) |
| --- | --- |
| The two shapes above and their field meanings | Which `phase` strings a
source uses beyond the common ones |
| `MetricAccuracy` and the rule that `EXACT` is only claimed when runtime
state proves it | The `SourcePosition` family and its keys (`MYSQL_BINLOG`,
`POSTGRES_LSN`, `FILE`, ...) |
| Common phases: `UNKNOWN` (no split yet), `READING`, `COMPLETED` (no more
splits and no more elements) for readers; `DISCOVERING`, `ASSIGNING`,
`COMPLETED`, `NOT_APPLICABLE` for enumerators | Finer reader phases: CDC uses
`SNAPSHOT`, `CATCH_UP`, `INCREMENTAL` in place of `READING` |
| Non-blocking, thread-safe, immutable, truthful provider guarantees (§3,
§4) | Anything in `details` |
Phases are strings with the common values as constants on the two classes. A
consumer that does
not know a connector-specific phase treats it as `READING`. An enum would be
safer for consumers
but would put CDC-specific values into a Source-level type or force a second
enum per connector
family; the string keeps the common type small. If the committers prefer an
enum for the common
phases plus a free-form `detailPhase`, that is a mechanical change.
Position families are strings too, for the same reason. The contract fixes
only that a position
carries a family, a schema version and string coordinates, that payloads
never contain
credentials or connection details, and that consumers compare positions
within one family only.
### 3. Unsupported or temporarily unavailable information
Every count and the position are a `MetricValue<T>`:
| Accuracy | Payload | Meaning |
| --- | --- | --- |
| `EXACT` | non-null | Current runtime state proves the value |
| `BEST_EFFORT` | non-null | Useful for diagnosis, precision not guaranteed |
| `UNSUPPORTED` | null | This source cannot provide the value |
| `UNAVAILABLE` | null | Supported, but not available for this observation |
A missing value is never replaced by zero or a guess. Nullable scalars
(`activeSplitId`,
`lastSourceEventAt`) are plain nullables because "unknown" is their only
non-value. Returning
`null` from the default method means "no report yet"; the engine does not
record anything.
### 4. How the engine collects without blocking processing
Unchanged from #11512, with names updated:
- Both methods return a snapshot the component already maintains. They never
perform source,
network, checkpoint or other blocking I/O and never take a lock that
record processing or split
assignment holds. Implementations publish a new immutable instance on
change (volatile field);
the engine reads the field.
- On each worker, `TaskExecutionService` samples `getReaderMetrics()` on the
existing
`print-execution-info-interval` cadence, wraps non-null results in an
envelope (task location,
source vertex, execution attempt, attempt-local sequence, observed-at) and
sends one batch per
worker to the coordinator. A throwing implementation is isolated per task
and logged by class
name only.
- The coordinator asks each worker that hosts a split enumerator for
`getEnumeratorMetrics()` on
the same cadence, at most one request per worker in flight. Enumerators
returning `null` add
nothing to the batch.
- The coordinator keeps the latest report per owner and task; lower attempts
and non-increasing
sequences are ignored; reports are removed with their pipeline.
- Wire format: one fixed schema per report type, enums and phases by name,
bounded entry and
field counts. Because the shapes are fixed, the engine never deserializes
connector classes.
Nothing is exposed through REST, metrics or the CLI in this phase.
### 5. Batch example: Local file source
Enumerator (`FileSourceSplitEnumerator` knows the file list and the assigned
set; it is not told
when a reader finishes a file):
```java
@Override
public SourceEnumeratorMetrics getEnumeratorMetrics() {
int total = filePaths.size();
int assigned = assignedSplit.size();
return new SourceEnumeratorMetrics(
assigned == total ? SourceEnumeratorMetrics.PHASE_COMPLETED
: SourceEnumeratorMetrics.PHASE_ASSIGNING,
MetricValue.exact(total - assigned), // pending
MetricValue.exact(assigned), // assigned
MetricValue.unsupported(), // running: enumerator
is not told
MetricValue.unsupported(), // completed:
enumerator is not told
MetricValue.exact(0), // unsplit units:
files are the splits
Collections.emptyMap());
}
```
Reader (one file at a time; rows emitted since the file was opened are
exact):
```java
@Override
public SourceReaderMetrics getReaderMetrics() {
return latest; // volatile; rebuilt when a file is opened, every N rows,
and on completion
}
private void refreshMetrics(String path, long rowsRead, boolean done) {
Map<String, String> coords = new LinkedHashMap<>();
coords.put("path", path);
coords.put("rows", Long.toString(rowsRead));
latest = new SourceReaderMetrics(
done ? SourceReaderMetrics.PHASE_COMPLETED :
SourceReaderMetrics.PHASE_READING,
path,
MetricValue.exact(new SourcePosition("FILE", 1, coords)),
System.currentTimeMillis(),
null,
Collections.emptyMap());
}
```
What this proves: the shapes hold without any CDC concept; a batch
enumerator can be honest about
not knowing completion; a batch reader's position is a split-local offset
rather than a stream
offset, and `SourcePosition` carries both.
### 6. CDC example: `connector-cdc-base`
Enumerator (`SnapshotSplitAssigner` already keeps an immutable count
snapshot; the incremental
assigner has no snapshot phase):
| Startup mode | Enumerator phase | pending / assigned / running / completed
/ unsplit |
| --- | --- | --- |
| `initial`, `snapshot-only` | `DISCOVERING` → `ASSIGNING` → `COMPLETED` |
all `EXACT` while counts are consistent, `BEST_EFFORT` otherwise; `unsplit` =
tables not yet chunked |
| `earliest`, `latest`, `specific`, `timestamp`, `committed-offset` |
`NOT_APPLICABLE` | all `UNSUPPORTED` |
Reader (`CdcReaderProgressTracker` derives the phase from the split it is
processing):
| Startup mode | Reader phase sequence | position before → after the first
record |
| --- | --- | --- |
| `initial` | `UNKNOWN` → `SNAPSHOT` → `CATCH_UP` → `INCREMENTAL`; idle
readers → `COMPLETED` | `UNAVAILABLE` → `BEST_EFFORT` (min snapshot watermark)
→ `EXACT` |
| `snapshot-only` | `UNKNOWN` → `SNAPSHOT` → `COMPLETED` | `UNAVAILABLE`
throughout |
| incremental-only modes | `UNKNOWN` → `INCREMENTAL` | `BEST_EFFORT`
(configured start) → `EXACT` |
`SNAPSHOT`, `CATCH_UP` and `INCREMENTAL` are CDC's refinement of `READING`.
Position families:
`MYSQL_BINLOG` (file, pos, gtids), others use the plugin name until a family
is defined. The
`details` map stays empty in v1.
### 7. Scope and compatibility
- Default methods: no existing connector or engine changes behaviour. A
connector that does not
override them is simply not observed.
- CDC (`connector-cdc-base`, MySQL family) is the only implementation in
this STIP. The file
example above is a design proof; implementing batch connectors is separate
work, if wanted.
- The contract stays `@Experimental` until at least one more source family
(batch or a second CDC
dialect such as PostgreSQL) has adopted it without changing the shapes.
- Naming: the types carry structured progress, not numeric counters, so they
do not go through
`MetricsContext`; the doc will state that distinction. If
`SourceReaderProgress` /
`SourceEnumeratorProgress` reads clearer alongside `MetricsContext`, the
rename is cheap now and
expensive later.
- STIP-30 (#11364) becomes "CDC as the first implementation of the Source
progress contract";
its startup-mode semantics, accuracy rules and engine collection design
are unchanged.
### 8. Changes to the open PRs if this is accepted
- #13239 (API): rename and move the types, drop the two provider interfaces
in favour of the
default methods, replace the CDC lifecycle and assignment enums with phase
constants, add the
bounded `details` map, add the batch example to the doc.
- #13240 (connector): adopt the default methods; phase strings instead of
enums.
- #11512 (engine): rename, codec gains the `details` block, otherwise
unchanged.
- #12137 (MySQL E2E) follows after the three merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]