Hi everyone, I would like to start some initial discussions around Iceberg event notification support, because we might have some engineering resources to work on Iceberg notification integration with AWS services such as SNS, SQS, CloudWatch.
As of today, we have a Listener interface and 3 events ScanEvent, IncrementalScanEvent, CreateSnapshotEvent. There is a static registry called Listeners that registers the event listeners in the JVM. However, when I read the related code paths, my thought is that it might be better to register listeners per-table, based on the following observations: 1. Iceberg events are all table or sub-table level events. For any catalog or global level events, the catalog service can provide notifications, Iceberg can be out of the picture. 2. A user might have multiple Iceberg catalogs defined, pointing to different catalog services. (e.g. one to AWS Glue, one to a Hive metastore). The notifications from tables of these different catalogs should be directed to different listeners at least per catalog, instead of the same set of listeners that are registered globally. 3. Event listener configurations are usually static. It makes more sense to me to define it once and then repeatedly use it, instead of re-registering it every time I start an application. If we register the listeners at table level, we can add a hook in TableOperations to get a set of listeners to emit specific events. The listeners could be defined and serialized as a part of the table properties, or maybe even a part of the Iceberg spec. This is really just my brainstorming. Maybe it's a bit overkill, maybe I am missing the correct way to use the Listeners static registry. It would be great if anyone could provide more contexts or thoughts around this topic. Best, Jack Ye
