[
https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227039#comment-16227039
]
Herman van Hovell commented on SPARK-22405:
-------------------------------------------
For some context. When implementing this we just wanted to have a way to know
that some metadata was about to change. A consumer could always retrieve more
information about the the (to-be) changed by querying the catalog (assuming
that a pre-event does not want to inspect the change itself). Propagating the
definition is very heavy weight and has an added problem that this sort of
implies that we should stabilize that class (hierarchy); so I opted not to do
that.
An additional problem with tracking metadata is if you use multiple clusters,
that you need to be able to track all metadata changes in all clusters running.
> Enrich the event information and add new event of ExternalCatalogEvent
> ----------------------------------------------------------------------
>
> Key: SPARK-22405
> URL: https://issues.apache.org/jira/browse/SPARK-22405
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Saisai Shao
> Priority: Minor
>
> We're building a data lineage tool in which we need to monitor the metadata
> changes in {{ExternalCatalog}}, current {{ExternalCatalog}} already provides
> several useful events like "CreateDatabaseEvent" for custom SparkListener to
> use. But the information provided by such event is not rich enough, for
> example {{CreateTablePreEvent}} only provides "database" name and "table"
> name, not all the table metadata, which is hard for user to get all the table
> related useful information.
> So here propose to and new {{ExternalCatalogEvent}} and enrich the current
> existing events for all the catalog related updates.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
