gardenia commented on PR #10515: URL: https://github.com/apache/ozone/pull/10515#issuecomment-4912949350
> @gardenia , the progress indicator is the txID, key is "notification-checkpoint", value is txID. How about use the OM metadata table to persist this value? metadata is defined in OMDBDefinition.java, currently it's used to persist data like OM layout version, ranger ozone service version, it can perfectly save the "notification-checkpoint" -> txID data, Or we can introduce a new table to save it, just like the statefulServiceConfig table in SCM, which can be used by any feature of SCM. > > Second, this indicator tries to prevent OM to handle events which have already been handled before OM start or OM leader transfer, but since the event model is at least once, the receiver need to have built-in capability to handle resent event notification. So the checkpoint loading or saving fails, we should still go ahead and send events out. Checkpoint is nice to have, not mandatory, it's failure should lead to the event process task function fail. Also we need to consider if event notification feature is enabled -> checkpoint is created and saved -> event notification is disabled, who is going to clean the checkpoint. If later event notification is enabled again, what will be the impact of this uncleaned checkpoint? > > Third, the event framework supports multiple plugins, each plugin runs independently. The current checkpoint and #10516 are specific to Kafka. If there is another plugin loaded, this plugin may also has it's checkpoint, and it needs to coordinate with Kafka about the purge, in case one plugin moving fast, while another one moving relatively slow. It looks like the current framework doesn't support this if we really want to support multiple plugins running together. > > BTW, I turned https://issues.apache.org/jira/browse/HDDS-13513 into an umbrella JIRA, and converted all existing event notification JIRA as the sub JIRA of [HDDS-13513](https://issues.apache.org/jira/browse/HDDS-13513). Let's create new sub JIRAs of [HDDS-13513](https://issues.apache.org/jira/browse/HDDS-13513) in future, so that we can easily track all event notification patches under one page. @ChenSammi I have opened a new PR which intends to implement the strategy you suggest for using RocksDB to store the checkpoint and address the other issues you mentioned (different checkpoint names per plugin and progress not stalled by a checkpoint write failure): https://github.com/apache/ozone/pull/10689 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
