donalmag commented on code in PR #8871: URL: https://github.com/apache/ozone/pull/8871#discussion_r2336134450
########## hadoop-hdds/docs/content/design/event-notifications.md: ########## @@ -0,0 +1,255 @@ +--- +title: Event notification support in Ozone +summary: Event notifications for all bucket/event types in ozone +date: 2025-06-28 +jira: HDDS-13513 +status: design +author: Donal Magennis, Colm Dougan +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Abstract + +Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem. +This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets. +This document proposes a potential solution and discusses some of the challenges/open questions. + +## Introduction + +Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications. +These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone. +These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts. + +### Goals + +Durable event log within each OM containing relevant OMRequest information for notification purposes. +Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM. +Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls. +Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable. +Read-only access for plugins to notification table. + +### Non-Goals + +Exactly-once end-to-end semantics to external systems. +Filtering of events or paths/buckets. +Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone. +Retrofitting historical events prior to feature enablement. + +### Supported OMRequests + +OMDirectoryCreateRequest +OMKeyCommitRequest +OMKeyDeleteRequest +OMKeyRenameRequest +OMKeyAddAclRequest +OMKeyRemoveAclRequest +OMKeySetAclRequest +OMKeySetTimesRequest + +# Design + +## Overview + +Introduce an Event Notification Pipeline for Apache Ozone with two +logical pieces: + +1. event data capture + +* OM captures the required details of selected OMRequest write + operations post metadata update and persists them to a dedicated RocksDB + completed operations "ledger" table keyed by the Ratis Txn Id +* each OM independently produces items to its local ledger table. The + ledger table should be integrated into OM Snapshots so that all OM's Review Comment: @errose28 - I meant to mention this on the community call. We are suggesting that this ledger has to be "included in the snapshot process". The idea being that, we are relying on the ratis logs to update the ledger table, if a node loads a snapshot currently, it will not see all the ratis logs, therefore it's ledger will be missing events which the leader would have seen, making it unable to fufil the 'at-least-once' guarantee after a failover. The snapshot process doesn't seem readily extendible to include this, would this be the correct approach, or do you have any other thoughts on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org For additional commands, e-mail: issues-h...@ozone.apache.org
