Thanks @Jocean for starting this discussion, I have some comments 1. About the public interfaces in the PIP, we should add @Public for them such as `Event`, `Listener` and even `CommitEvent` and other events. But for `Listeners`, I don't think it should be a public interface. All fields in the public interface for users should be `Public` too, but I found the information such as `ManifestEntry` in `CommitEvent` is not a public interface. I think you may need to reconsider which interfaces need to be marked with @Public and which are not.
2. In general, it is better to give a `Factory` to create `Listener` which should be all marked as `@Public` and you can see `CatalogFactory`->`Catalog` as an example. 3. Currently Flink and Spark support meta-data listeners and we can support reporting ddl information there, should we need to add the same listener in Paimon? 4. Should we need to report the events such as commit/compact to an external system? Currently we have some system tables and users can get these information by SQL or API, should the external system query these information regularly instead of a listener to push them? Best, Shammon FY On Tue, Aug 15, 2023 at 11:08 AM Jocean shi <[email protected]> wrote: > Hi devs: > > We would like to start a discussion about PIP-8: Introduce listeners > for Paimon[1]. > > In production environments, users often need to perceive the state > changes of Paimon table, > such as whether a new file has been committed to the table, in which > partitions the committed files are, > the size and number of the committed files, the status and type of > compaction, operations like table creation, deletion, and schema > changes, etc. > So, we introduce a Listener system for Paimon. > Looking forward to hearing from you. > > [1] > https://cwiki.apache.org/confluence/display/PAIMON/PIP-8%3A+Introduce+listeners+for+Paimon > > Best > shidayang >
