88fantasy opened a new issue, #12207: URL: https://github.com/apache/seatunnel/issues/12207
### Motivation Metadata/lineage platforms that speak [OpenLineage](https://openlineage.io/) (Marquez, Gravitino, DataHub, ...) have no way to learn which tables a SeaTunnel job read from and wrote to. This is an umbrella issue for adding opt-in, table-level lineage reporting so a SeaTunnel job can tell such a receiver, in the standard OpenLineage `RunEvent` shape, what it ran and what it touched. ### What this adds - A START event when a job (or one of its pipelines) begins, a terminal COMPLETE/FAIL/ABORT event when it ends, and an optional throttled RUNNING heartbeat for long-running streaming jobs. - Table-level datasets only (no column-level lineage), named per the OpenLineage naming spec. - Output row-count/byte-size statistics on the terminal event where the engine can obtain them. - Everything here is **off by default** and fully opt-in via a new `openlineage_enabled` job option. ### Scope - Engines: **Zeta** and the **Flink 1.20 starter**. Spark and the Flink 13/15 starters are explicitly out of scope (the latter two because `JobStatusHook`, the hook this design relies on, is a Flink 1.16+ API). - Granularity: table-level only. Column-level lineage would require per-transform field-mapping resolution (including SQL lineage parsing for the `Sql` transform) and is a separate, larger effort. ### Design Two small new modules carry the engine-agnostic pieces, so neither engine's code needs to know about OpenLineage directly: - `seatunnel-lineage`: configuration resolution (11 new `openlineage_*` job options), the `LineageEvent`/`LineageDataset` model, dataset naming per the OpenLineage spec, and a pluggable `LineageBackend` SPI. - `seatunnel-lineage-openlineage`: the default backend — OpenLineage 1.29.0 model classes plus a hand-rolled `httpclient`-based transport (neither `openlineage-java`'s own transport nor SeaTunnel's existing shaded Jackson infrastructure could be reused as-is; see PR #{{PR1}} for why). Each engine gets a thin call site on top of that: - Zeta: `ZetaLineageReporter`, invoked from the job master. - Flink (1.20 starter only): `LineageJobStatusHook` (JobManager-side, implements Flink's `JobStatusHook`) plus a client-side registration/statistics path, because attached and detached Flink submissions need different terminal-event handling. ### PR plan Split into 3 PRs, each based on `dev`, to keep each review scope narrow: 1. **PR #{{PR1}}**: the two new modules, config options, dependency/license bookkeeping. No engine wiring. Implements #12204. 2. **PR #{{PR2}}**: the Zeta call site. Built on top of #{{PR1}}'s branch, so its diff includes #{{PR1}}'s changes until that one merges — see the PR description for the exact commit range to review. Implements #12205. 3. **PR #{{PR3}}**: the Flink call site + cluster deployment docs. Same stacking note as #{{PR2}}. Implements #12206. ### Validation Beyond unit/integration tests, this was validated end-to-end against a real Gravitino receiver in a sandbox: Zeta jobs writing to both a Paimon table and a Doris table, and the Flink 1.20 starter submitting the same kind of job in both attached and detached mode — each producing the expected OpenLineage events with row counts that were cross-checked directly against the written tables. Details are in each PR's "How was this patch tested" section. ### Backward compatibility Purely additive and opt-in. No existing configuration option is renamed, removed, or has its default value changed. No existing SPI contract changes. Nothing happens for any job that doesn't set `openlineage_enabled = true`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
