bamaer opened a new issue, #9004:
URL: https://github.com/apache/hop/issues/9004
### What would you like to happen?
The lineage hub (`org.apache.hop.lineage`) emits per-transform lifecycle,
file-I/O and relational-I/O events, but no event carries the pipeline's hop
graph, so a sink cannot tell which transform fed which. That matters because
dataset lineage is derived *through* jobs: a job's accumulated inputs are
paired all-to-all with its outputs.
An OpenLineage sink has to choose between two broken options today:
* one job per pipeline (what `plugins/tech/openlineage` does, so Marquez
shows the datasets on the job a user opens): a pipeline that reads `a` and `b`
and writes `x` and `y` produces four dataset edges (`a→x`, `a→y`, `b→x`, `b→y`)
instead of two;
* one job per transform with the I/O each transform emits: every edge
disappears, because a reader's job has only inputs and a writer's job has only
outputs (`dataset → dataset` lineage has to pass through a job that holds both
ends).
Declared `fineGrainedLineages` do not fix dataset-level pairing (measured:
DataHub stores the declaration faithfully and still reports the full product at
dataset level).
**Request:** emit the pipeline's hop graph once per run, as an additive
event kind:
```java
public enum LineageEventKind { ..., PIPELINE_GRAPH }
public final class PipelineGraphLineagePayload implements LineagePayload {
public record Transition(String from, String to) {}
private final List<Transition> transitions; // enabled hops, in pipeline
order
}
```
emitted from `LineageRunLifecycleEmitter.emitPipeline` when `phase ==
STARTED`, with the same context, built from `PipelineMeta.getHops()` filtered
to `hop.isEnabled()` with both ends present. Roughly 80 lines including a test.
With the graph a sink can:
1. attach file/relational I/O to the transform's job instead of the
pipeline's, and
2. propagate: `inputs(T) = ownReads(T) ∪ ⋃ outputs(U)` for `U ∈ prev(T)`,
transitively,
which gives the true dataset edges for chains (including pass-through
transforms that do no I/O of their own), fan-out and fan-in. The pipeline job
stays the run-level entry point without datasets; the transform jobs carry the
pairs.
This is additive: no existing sink behaves differently until it reads the
new kind. The mapping change in the OpenLineage sink would be gated by a flag
and measured before it becomes a default, because it also moves the datasets
from the pipeline job to its transforms in Marquez.
### Issue Priority
Medium (needed before lineage can be correct for multi-input/multi-output
pipelines; nothing else depends on it)
### Issue Component
Engine, Plugins
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
