bamaer opened a new issue, #9324:
URL: https://github.com/apache/hop/issues/9324
### What would you like to happen?
### What would you like to happen?
Follow-ups to the lineage engine (#9322), the OpenLineage and DataHub sinks
(#9034) and
reading metadata from DataHub (#9323). Each item can be its own PR.
**Lineage sinks in the GUI**
1. **Configure sinks in the GUI.** A `lineage-sink` metadata type with a
dialog per sink
(OpenLineage, DataHub), selectable per project or run configuration. The
DataHub sink
uses the connection from #9323. The `HOP_LINEAGE_*`, `HOP_OPENLINEAGE_*`
and
`HOP_DATAHUB_*` variables keep working.
2. **Sink health.** Show events queued, spooled, sent and rejected, and the
last error, per
sink, in Hop GUI and on the Hop Server status page. A wrong token or an
unreachable
backend should not go unnoticed.
**Richer lineage**
3. **Row counts and write operations.** Rows read, written, rejected and
errors per transform
and per run. OpenLineage: input/output statistics facets. DataHub: run
properties plus an
`operation` aspect on written datasets, which also drives DataHub's
freshness.
4. **Transformation logic.** Send the logic a transform applies where it is
cheap to get:
the SQL of Table Input and Execute SQL, the expressions of Formula and
Calculator.
OpenLineage: the `sql` job facet. DataHub: the DataJob's
transformation-logic aspect,
on the versions that support it.
5. **Platform instance per relational connection.** So two servers with the
same database and
table names do not merge into one dataset. It must match the instance
DataHub's own
ingestion source uses.
**Publishing project metadata to DataHub**
6. **Hop Doc to Context Documents.** Publish what `hop doc` generates for a
project as
DataHub Context Documents, linked to the matching DataFlows and DataJobs.
7. **Design-time project structure.** Publish pipelines and workflows as
DataFlows and DataJobs,
with their declared inputs and outputs, without running them: a `hop`
command and a
workflow action. Include file layouts defined in Hop (for example Text
file input fields)
as dataset schemas, since DataHub cannot crawl those. Out of scope:
crawling database
tables, which DataHub's own sources already do.
8. **Unit test results as assertions.** Publish Hop unit test results (data
sets and golden
data) as DataHub assertion runs on the datasets they test.
**AI Assistant**
9. **DataHub context.** Let the AI Assistant include DataHub context
(descriptions, owners,
tags, glossary terms, deprecation) for the datasets a pipeline touches.
### Issue Priority
Priority: 3
### Issue Component
Component: Other
### Issue Priority
Priority: 2
### Issue Component
Component: Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]