bamaer opened a new pull request, #9545: URL: https://github.com/apache/hop/pull/9545
Items 1, 2 and 4–9 of #9324 (item 3 was #9479), one commit per item, followed by fixes from review. - **Lineage sinks as metadata (1).** A `lineage-sink` metadata type, OpenLineage or DataHub, selected in a pipeline or workflow run configuration. A child run also reports to its parent's sink. The DataHub sink takes its URL, token and environment from a DataHub data catalog. The `HOP_LINEAGE_*` variables work as before. - **Sink health (2).** Tools > Lineage sinks in Hop GUI, and a section on the Hop Server status page: events sent, refused, waiting and spooled, and the last error. - **Transformation logic (4).** The SQL of Table input and Execute SQL goes to the OpenLineage `sql` facet and to DataHub's transformation logic. Formula and Calculator expressions become task properties `logic.<field>`, because DataHub only takes SQL there. - **Platform instance per connection (5).** The DataHub data catalog maps relational connections to platform instances. Their tables are then named `<instance>.<db>.<schema>.<table>`, as DataHub's SQL sources name them. - **Documentation (6).** Publishing can include what `hop doc` writes, as DataHub documents (1.7 and later). They are related to the pipeline and its tasks, under a document for the project. Parameters and notes are only included when asked for. - **Publishing without running (7).** `hop datahub-publish` and a new **Publish to DataHub** action publish pipelines and workflows: their tasks, the tables and files they declare, and file layouts as schemas. Names resolve as a run of each file resolves them, and no database is queried. - **Unit tests as assertions (8).** Each golden data set becomes an assertion run (passed, failed or error) on the datasets the tested transform writes, and an OpenLineage `hopUnitTest` run facet. The test's own run, of a modified copy of the pipeline, doesn't go to sinks selected in a run configuration. - **AI Assistant (9).** A sharing option that adds what a data catalog says about the pipeline's tables and files: descriptions, owners, tags, glossary terms and deprecation. The catalog is read in the background. Tested with unit tests in the engine and the affected plugins, and with a new IT project, `integration-tests/lineage-sinks`, which runs a DataHub stand-in in the container. It covers sink routing and inheritance, publishing, documents, unit test assertions, transformation logic and platform instances. The `openlineage` IT passes too, and I checked against a live DataHub 1.7. Not in this PR: - Child workflows of a unit test run still report their lineage. - Results of the Execute tests transform don't reach run-configuration sinks. Fixes #9324 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
