88fantasy opened a new pull request, #12209:
URL: https://github.com/apache/seatunnel/pull/12209
> **Stacked on #12208.** This branch is built on top of
`feature/openlineage-core` (PR #12208), so until #12208 merges, this diff also
shows #12208's changes (the `seatunnel-lineage`/`seatunnel-lineage-openlineage`
modules and the new `openlineage_*` options). It will shrink to just the
Zeta-specific commits once #12208 merges into `dev`. Please review this PR's
own commits (`git log feature/openlineage-core..feature/openlineage-zeta`);
#12208's commits are under separate review there.
### Purpose of this pull request
Implements #12205 (part of the umbrella #12207), stacked on #12208. Adds
`ZetaLineageReporter` and wires it into
`JobMaster`/`PhysicalPlan`/`CheckpointCoordinator` so a Zeta job reports
OpenLineage START / throttled RUNNING heartbeat / terminal COMPLETE-FAIL-ABORT
events for each of its sinks when `openlineage_enabled = true`.
### Does this PR introduce _any_ user-facing change?
Yes, but opt-in only (`openlineage_enabled`, default `false`, added in
#12208). When enabled: one extra outbound HTTP call at job start, one per
completed checkpoint (throttled by `openlineage_heartbeat_min_interval_ms`),
and one at job end. When disabled (the default for every existing job), no
change to job execution, checkpointing, or scheduling.
### How was this patch tested?
- Unit tests: `ZetaLineageReporterTest`, `LineageOptionParityTest`
(env-option resolution matches what #12208 defines),
`CheckpointCoordinatorTest` (the heartbeat hook doesn't interfere with
checkpoint completion), `SourceSinkActionCompatibilityTest` (the actions this
PR touches keep a stable `serialVersionUID`).
- End-to-end against a real Gravitino instance, in an isolated standalone
SeaTunnel sandbox cluster not shared with any other workload, built from this
PR's own code:
- `FakeSource (10 rows) → Paimon` batch job: Gravitino received one
`start` and one `complete` event with `outputStatistics: {rowCount: 10, size:
170}`; the Paimon table's data files (schema/manifest/snapshot/parquet) were
inspected directly and match.
- `FakeSource (10 rows) → Doris` batch job: same event shape and
statistics; verified against the actual table with `SELECT COUNT(*)` over the
Doris query port — `10`, matching.
- Both events' facets were checked for the shaded-Jackson corruption issue
described in #12208 — clean, properties inlined correctly, no
`additionalProperties` wrapping.
- `./mvnw -q -DskipTests verify` across the whole reactor — 470 modules,
BUILD SUCCESS. This branch's affected-module unit tests all pass.
### Check list
* [ ] No new JAR
* [x] Documentation updated (`docs/en`, `docs/zh` — Zeta section of the
lineage concept page; one line added to the hybrid-cluster-deployment doc)
* [ ] `incompatible-changes.md` — not applicable
* [ ] Connector-specific checklist items — not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
