rahul-madaan opened a new issue, #39427: URL: https://github.com/apache/beam/issues/39427
### What would you like to happen? Beam pipelines should be able to emit [OpenLineage](https://openlineage.io) events describing the datasets they read and write, so OpenLineage consumers (Marquez, Atlan, and others) can build lineage graphs for Beam jobs on any runner. This continues #33981 and builds on the pluggable lineage mechanism from #36790. Proposed: a new `sdks/java/extensions/openlineage` module that follows the architecture of the official OpenLineage Spark and Flink integrations: - Configuration extends `io.openlineage.client.OpenLineageConfig` and resolves with the Spark integration's precedence: pipeline options over `openlineage.yml` over `OPENLINEAGE__` environment variables. Transport, facets, dataset and circuit-breaker settings reuse the standard openlineage-java config classes. - `OpenLineageRunner` wraps any runner (`--runner=OpenLineageRunner --openLineageDelegateRunner=FlinkRunner`), extracts datasets from the pipeline graph via classpath-guarded per-IO visitors (PubsubIO, IcebergIO; extensible through a public `LineageProvider` interface), mints a UUIDv7 run id at submission and propagates it via serialized options, and emits START and terminal COMPLETE/ABORT/FAIL events. - A job tracker emits periodic RUNNING events on the Flink integration's `trackingIntervalInSeconds` cadence (default 60s) and sweeps lineage reported through the Beam `Lineage` metrics API. - `OpenLineageLineage` implements the `LineageBase` plugin (`--lineageType`) to capture lineage live in worker JVMs for long-running streaming jobs, teeing FQNs back into the metrics store so runner-native consumers keep working. - Dataset naming strictly follows the [OpenLineage naming conventions](https://openlineage.io/docs/spec/naming/); Iceberg datasets use the physical table location with the catalog identity as a TABLE symlink, matching the Spark integration's IcebergHandler. ### Issue Priority Priority: 2 (default / most feature requests should be filed as P2) ### Issue Components - [X] Component: Java SDK - [ ] Component: Python SDK -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
