Thanks Jocean. So should we need to introduce trigger-based metrics for some special events such as commit/compaction? Maybe we can hear from others cc @Caizhi Weng <[email protected]> , @Jingsong Li <[email protected]> , what do you think?
Best, Shammon FY On Fri, Sep 1, 2023 at 11:29 AM Jocean shi <[email protected]> wrote: > Hi Shammon FY > > Continuing from the previous discussion. I would like to discuss the > abstract model of "Listener," "Metric," and "MetricReport" further. > Firstly, it is important to clarify that the discussed compaction and > commit operations have clear boundaries, making them events. These > events, in addition to basic information, also include metrics > generated during the execution process, such as execution time and CPU > consumption. Therefore, an event consists of both base information and > base metrics. Users can obtain these events through Listeners, and > they can construct the desired new Metric using these events. They can > then report the Metric periodically via MetricReport. Thus, I believe > the Metric system is a use case of the Listener. > > Some similar implementations: > > SparkListener: The SparkListenerTaskEnd event includes TaskInfo and > TaskMetrics, and users can subscribe to SparkListenerTaskEnd to obtain > the desired metrics. > Iceberg: Iceberg's CommitMetric is also generated through the > CreateSnapshotEvent and reported via a reporter. The difference is > that Iceberg's reporting is trigger-based, while Paimon performs it on > a schedule. > > Best > shidayang > > Jocean shi <[email protected]> 于2023年8月24日周四 17:41写道： > > > > Hi Shammon FY > > > > Thanks for your comment. > > > > 1. DDL events > > Many behaviors of the Table service are related to the options of > > tables, such as whether the table has enabled full-compaction and the > > triggering conditions for compaction. If the options of a table are > > changed, the Table service needs to perceive it in a timely manner and > > make corresponding adjustments to the behavior of the table. Without a > > listener mechanism, the Table service needs to constantly poll the > > table to determine if its configuration has changed, which increases > > the pressure on Hive and the Table service. If we can listen to the > > AlterTableEvent, we won't need to poll the options of the table. > > > > 2. Why not metric > > Metric is mainly processed statistical indicators that are usually > > measured at regular intervals, and multiple reported values may be the > > same. This is quite different from events. For example, for 'commit', > > Metric usually measures the size, quantity, and duration of recently > > committed files, and the results obtained from multiple retrievals may > > be the same. It can be imagined that replacing the currently existing > > CommitCallback with Metric would be very troublesome. > > > > Best > > shidayang > > > > Shammon FY <[email protected]> 于2023年8月23日周三 10:53写道： > > > > > > Hi Jocean > > > > > > Thanks for your answer. I think there are two types of the information > you > > > want to report: the ddl events and the runtime events such as commit, > > > compaction. > > > > > > For the ddl events, I don't quite understand why you need to poll the > table > > > information regularly? As we all know that Paimon is really a storage > which > > > has all meta information in it, and even when you poll the information > from > > > Paimon, you need to store it somewhere. I think you can just use > Paimon as > > > the storage itself. If the performance of obtaining Paimon tables is > > > relatively low, such as the large number of tables you mentioned, I > think > > > we should improve this, for example, add a table cache? > > > > > > For the runtime events, I understand that they are indeed necessary to > > > report to a system like `Table Service`. But my issue is: can we do > this in > > > the existing metrics mechanism? For example, reporting relevant > metrics to > > > the `Table Service` instead of adding a new `listener`? If the metrics > > > information is not complete enough, we can continue to add information > in > > > it. > > > > > > Best, > > > Shammon FY > > > > > > On Tue, Aug 22, 2023 at 2:20 PM Jocean shi <[email protected]> > wrote: > > > > > > > Hi Shammon FY, > > > > > > > > I get your point, but the role of a Listener is more towards > > > > notification. For example, as you mentioned, we can query the > relevant > > > > information through APIs for DDL and commit information. However, > when > > > > we want to know if there have been any changes to the relevant > > > > information, we need to constantly poll the tables. This mechanism > can > > > > be resource-intensive, especially when there are many tables. With a > > > > Listener, we can promptly detect changes in status. Consider a > > > > separate Table service that has a requirement to compact all tables, > > > > and the compact parameters are stored in the options. When there is a > > > > change in the options of a table, the Table Service needs to be > > > > notified promptly to determine whether to immediately compact the > > > > table. When there is new data committed to a table, it needs to be > > > > promptly detected to determine whether to compact it. Also, users > need > > > > the assistance of CommitEvent to trigger downstream tasks based on > the > > > > watermark of a table. > > > > Querying compact information through SQL or APIs is indeed a good > way. > > > > It is relatively simple to query historical compact records. However, > > > > if you want to know the current compact status of a table, using a > > > > Listener may be simpler. > > > > > > > > Best > > > > Shidayang > > > > > > > > Shammon FY <[email protected]> 于2023年8月21日周一 23:24写道： > > > > > > > > > > Hi Jocean, > > > > > > > > > > Thanks for your explanation. I still have some issues > > > > > > > > > > 1. What are the ddl events for Paimon used for? If you need to show > > > > tables > > > > > for paimon in your system, I think it's better to define table > related > > > > > interfaces, and then you can implement them for Paimon, Iceberg > and Hudi > > > > > instead of adding a ddl listener in them. It's more general and > you can > > > > > even manage other tables such as databases, mongodb and hive. > > > > > > > > > > 2. If some system information in `CompactEvent` is currently > missing or > > > > > there's no information about `compact`, I think a better way is > to add > > > > > this system information in Paimon, rather than adding a listener > and > > > > > creating an event with the information. Then the external system > can get > > > > > the information by SQL or API directly, this is a more reasonable > > > > approach. > > > > > > > > > > 3. Also what is the `CommitEvent` used for? Currently we have > metrics for > > > > > `Commit` and jobs can report them. How about adding a customized > reporter > > > > > for metrics instead of a listener for `CommitEvent`? > > > > > > > > > > Best, > > > > > Shammon FY > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Aug 21, 2023 at 5:16 PM Jocean shi <[email protected]> > wrote: > > > > > > > > > > > Hi Shammon FY, > > > > > > > > > > > > Thanks for your comments. I’d like to share my thoughts about > your > > > > > > comments. > > > > > > > > > > > > 1. Public Interface > > > > > > Thank you for the reminder. I overlooked the correspondence > between > > > > > > the Public Interface of PIP and the "@Public" annotation. > > > > > > My idea was that Event, Listener, and ListenerFactory are public, > > > > > > while the others are non-public. > > > > > > > > > > > > 2. Add `Factory` to create `Listener` > > > > > > Great suggestion, I have already added the ListenerFactory to > PIP. > > > > > > > > > > > > 3. Flink and Spark support meta-data listeners > > > > > > It will be very inconvenient for users to obtain DDL information > > > > > > through engines. Firstly, there are many implementations of > various > > > > > > engines that need to be connected. Secondly, in addition to > Flink and > > > > > > Spark, many engines do not support meta-data listeners. As a > general > > > > > > data lake, Paimon should have its own mechanism for meta-data > > > > > > listeners. > > > > > > > > > > > > 4. report events such as commit/compact to an external system > > > > > > CompactEvent: Currently, the compact state is a black box, and > users > > > > > > cannot obtain the information through SQL or API. > > > > > > CommitEvent: Currently, the methods of querying through SQL or > API are > > > > > > based on polling, which makes it difficult for users to perceive > > > > > > commit operations in a timely manner and consumes a lot of > resources. > > > > > > > > > > > > Best > > > > > > Shidayang > > > > > > > > > > > > Shammon FY <[email protected]> 于2023年8月18日周五 14:07写道： > > > > > > > > > > > > > > Thanks @Jocean for starting this discussion, I have some > comments > > > > > > > > > > > > > > 1. About the public interfaces in the PIP, we should add > @Public for > > > > them > > > > > > > such as `Event`, `Listener` and even `CommitEvent` and other > events. > > > > But > > > > > > > for `Listeners`, I don't think it should be a public > interface. All > > > > > > fields > > > > > > > in the public interface for users should be `Public` too, but I > > > > found the > > > > > > > information such as `ManifestEntry` in `CommitEvent` is not a > public > > > > > > > interface. I think you may need to reconsider which interfaces > need > > > > to be > > > > > > > marked with @Public and which are not. > > > > > > > > > > > > > > 2. In general, it is better to give a `Factory` to create > `Listener` > > > > > > which > > > > > > > should be all marked as `@Public` and you can see > > > > > > > `CatalogFactory`->`Catalog` as an example. > > > > > > > > > > > > > > 3. Currently Flink and Spark support meta-data listeners and > we can > > > > > > support > > > > > > > reporting ddl information there, should we need to add the same > > > > listener > > > > > > in > > > > > > > Paimon? > > > > > > > > > > > > > > 4. Should we need to report the events such as commit/compact > to an > > > > > > > external system? Currently we have some system tables and > users can > > > > get > > > > > > > these information by SQL or API, should the external system > query > > > > these > > > > > > > information regularly instead of a listener to push them? > > > > > > > > > > > > > > Best, > > > > > > > Shammon FY > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 15, 2023 at 11:08 AM Jocean shi < > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > Hi devs: > > > > > > > > > > > > > > > > We would like to start a discussion about PIP-8: Introduce > > > > listeners > > > > > > > > for Paimon[1]. > > > > > > > > > > > > > > > > In production environments, users often need to perceive the > state > > > > > > > > changes of Paimon table, > > > > > > > > such as whether a new file has been committed to the table, > in > > > > which > > > > > > > > partitions the committed files are, > > > > > > > > the size and number of the committed files, the status and > type of > > > > > > > > compaction, operations like table creation, deletion, and > schema > > > > > > > > changes, etc. > > > > > > > > So, we introduce a Listener system for Paimon. > > > > > > > > Looking forward to hearing from you. > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/PAIMON/PIP-8%3A+Introduce+listeners+for+Paimon > > > > > > > > > > > > > > > > Best > > > > > > > > shidayang > > > > > > > > > > > > > > > > > > >
