bamaer opened a new issue, #9324:
URL: https://github.com/apache/hop/issues/9324

   ### What would you like to happen?
   
   ### What would you like to happen?
   
   Follow-ups to the lineage engine (#9322), the OpenLineage and DataHub sinks 
(#9034) and
   reading metadata from DataHub (#9323). Each item can be its own PR.
   
   **Lineage sinks in the GUI**
   
   1. **Configure sinks in the GUI.** A `lineage-sink` metadata type with a 
dialog per sink
      (OpenLineage, DataHub), selectable per project or run configuration. The 
DataHub sink
      uses the connection from #9323. The `HOP_LINEAGE_*`, `HOP_OPENLINEAGE_*` 
and
      `HOP_DATAHUB_*` variables keep working.
   2. **Sink health.** Show events queued, spooled, sent and rejected, and the 
last error, per
      sink, in Hop GUI and on the Hop Server status page. A wrong token or an 
unreachable
      backend should not go unnoticed.
   
   **Richer lineage**
   
   3. **Row counts and write operations.** Rows read, written, rejected and 
errors per transform
      and per run. OpenLineage: input/output statistics facets. DataHub: run 
properties plus an
      `operation` aspect on written datasets, which also drives DataHub's 
freshness.
   4. **Transformation logic.** Send the logic a transform applies where it is 
cheap to get:
      the SQL of Table Input and Execute SQL, the expressions of Formula and 
Calculator.
      OpenLineage: the `sql` job facet. DataHub: the DataJob's 
transformation-logic aspect,
      on the versions that support it.
   5. **Platform instance per relational connection.** So two servers with the 
same database and
      table names do not merge into one dataset. It must match the instance 
DataHub's own
      ingestion source uses.
   
   **Publishing project metadata to DataHub**
   
   6. **Hop Doc to Context Documents.** Publish what `hop doc` generates for a 
project as
      DataHub Context Documents, linked to the matching DataFlows and DataJobs.
   7. **Design-time project structure.** Publish pipelines and workflows as 
DataFlows and DataJobs,
      with their declared inputs and outputs, without running them: a `hop` 
command and a
      workflow action. Include file layouts defined in Hop (for example Text 
file input fields)
      as dataset schemas, since DataHub cannot crawl those. Out of scope: 
crawling database
      tables, which DataHub's own sources already do.
   8. **Unit test results as assertions.** Publish Hop unit test results (data 
sets and golden
      data) as DataHub assertion runs on the datasets they test.
   
   **AI Assistant**
   
   9. **DataHub context.** Let the AI Assistant include DataHub context 
(descriptions, owners,
      tags, glossary terms, deprecation) for the datasets a pipeline touches.
   
   ### Issue Priority
   
   Priority: 3
   
   ### Issue Component
   
   Component: Other
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Other


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to