GitHub user clhubert created a discussion: Build Trust in Data and Enhance Semantic Layer for AI via Data Provenance
BRIEFLY Discussed this with @rusackas / @michael-s-molina in the 7/1/2026 Super Set SIP Review. Looking for feedback on usefulness and potentially clarity on an a future extension/SIP. **Use Case 1** : Build Trust in Data AND Enhance Semantic Layer for AI via Data Provenance - The source of the data in a superset dataset can be used to ENHANCE AI's semantic awareness via its ability to trace the linage AND provenance of data. - For the report/dashboard end user who want to know WHERE the data come from, like a reference, footnote, etc... Build trust in data provided in dashboard. **Use Case 2** - Visualize: Report Author/Viewer wants to see "**_provenance_**" of data from a dataset visually like a sql trace. - Linage is already supported with what’s available today: community extension. By @michael-s-molina - https://superset.apache.org/developer-docs/extensions/registry/, https://github.com/msyavuz/superset-sql-visualizer - Including data provenance leads to more COT (Chain of Thought), for the AI. **Use Case 3:** Pull in EXTENDED Data Linage/Provenance MetaData - This is beyond where the information is at rest in data set, but where the data that was inserted into the dataset came from (Provenance). - This data may already be accessible via a Data Catalog, Glossary, MDM and or Data Governance Platform such as Open Meta Data - Open MetaData - pulls in metadata from multiple sources -- Data Dictionary terms in glossary can be associated with several fields. In OpenMetaData this would be associated an asset(s) to a term. - https://docs.open-metadata.org/v1.12.x/how-to-guides/data-governance/glossary/glossary-term . Other MDMs and Catalogs (Talend Data Catalog), have something similar. --- OpenMetaData uses Apache Airflow for Ingestion - https://docs.open-metadata.org/v1.12.x/connectors/pipeline/airflow --- **Lineage Workflow** - https://docs.open-metadata.org/v1.12.x/how-to-guides/data-lineage/workflow --- **Lineage Ingestion** - https://docs.open-metadata.org/v1.12.x/connectors/ingestion/lineage --- Apache Nifi is one of the Source - https://docs.open-metadata.org/v1.12.x/connectors/pipeline/nifi, which exposes its internal data provenance information to OpenMetaData. **Notable:** - Not clear if the Data Governance use case mentioned above can be accomplished using an MCP enabled Data Governance Platform. -- Open MetaData has MCP integration into their semantic layer. https://open-metadata.org/mcp - Avoid building a federated query engine - Denodo, TIbco Data Virtualization that does query plans across disparate data sources. GitHub link: https://github.com/apache/superset/discussions/41633 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
