bamaer opened a new issue, #9034: URL: https://github.com/apache/hop/issues/9034
### What would you like to happen? Add a lineage sink that writes Hop lineage to [DataHub](https://datahubproject.io/), as a new plugin `plugins/tech/datahub` next to the OpenLineage sink. In DataHub this gives: - each pipeline and workflow as a DataHub pipeline (DataFlow) on an `Apache Hop` platform, with its transforms or actions as tasks (DataJobs); the hop graph becomes the task graph - each run of a pipeline, workflow, transform or action as a run with a readable name, its outcome and its parent run; a pipeline started by a workflow action is linked to that action - the files and tables each transform or action reads and writes, named the way DataHub's own sources name them, so Hop lineage lands on the datasets those sources ingest - column lineage, column types (only where DataHub has no schema yet) and descriptions from the Hop metadata - copied and moved files linked to their source; optionally, deleted and moved files deprecated - transforms or actions removed from a pipeline or workflow retired from DataHub The sink writes to DataHub's metadata API (GMS) directly, not to DataHub's OpenLineage endpoint. That endpoint needs server-wide settings that also change how other OpenLineage producers are ingested, it ingests asynchronously, and it turns every transform into a pipeline of its own. It reuses the OpenLineage plugin's mapping, asynchronous delivery and spool, so delivery is the same: off the pipeline thread, retried, and durable across a DataHub outage when a spool folder is set. Configuration is through engine variables, like the other lineage settings: HOP_LINEAGE_ENABLED=Y HOP_LINEAGE_SINK_IDS=datahub HOP_LINEAGE_DATAHUB_URL=http://localhost:8080 HOP_LINEAGE_DATAHUB_TOKEN=<personal access token, optionally hop-encrypt'ed> Builds on #9004 (pipeline hop graph) and #9005 (lineage of short `hop-run` runs). Tested against DataHub 1.0 and 1.7. ### Issue Priority Priority: 2 ### Issue Component Component: Other -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
