bamaer opened a new pull request, #9457: URL: https://github.com/apache/hop/pull/9457
Items 1 and 2 of #9323: Hop reads dataset metadata from a data catalog, and a workflow can check datasets before it goes on. - A `data-catalog` metadata type with a pluggable catalog type, like execution information locations. DataHub is the first type, in the DataHub plugin: URL, token, environment, timeouts, trust store, and a Test button in the editor. - A new action, **Check data catalog**: for a list of tables, files or catalog ids it checks that the catalog has them, that they aren't deprecated, that their data changed within a maximum age, and that their latest assertion runs passed. A failed check is a false result with 0 errors; a configuration problem is an error. What happens when the catalog can't answer is an option. - Tables and files resolve to the same dataset ids the lineage sinks write, through the engine's lineage identity code. The API (`org.apache.hop.datacatalog`) and the action are marked experimental. Points worth checking: - The DataHub queries use fields DataHub has had since 1.0 (checked against the 1.0, 1.7 and 1.8 schemas). - Freshness comes from DataHub operations. The DataHub lineage sink doesn't send those yet, so a table that only Hop writes has no last change; follow-up for #9324. - The Data Catalog type is in the default client; DataHub comes with the full client or the marketplace. Tests: unit tests in engine, the DataHub plugin and the action. IT `datacatalog` runs the action against a DataHub GraphQL stub (8 cases, plus a check of what was asked). Also tried against a local DataHub 1.7. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
