chungen0126 commented on code in PR #8449: URL: https://github.com/apache/ozone/pull/8449#discussion_r2103362852
########## hadoop-hdds/docs/content/design/s3-event-notification.md: ########## @@ -0,0 +1,284 @@ +--- +title: S3 Event Notifications +summary: S3 Event Notifications support, similar like AWS +date: 2025-05-14 +jira: HDDS-5984 +status: design +author: Chung En Lee, Wei-Chiu Chuang + +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3 Event Notifications +## Overview + +This document proposes the design of an event notification system for Apache Ozone. The goal is to enable external consumers to subscribe to and consume events that occur within the Ozone cluster. This requirement closely resembled the functionality provided by ([AWS S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html)). + +## Motivation + +Apache Ozone doesn’t support event notifications, which limits its integration with downstream systems like real-time data pipelines, auditing tools, or metadata indexers. Adding this feature will enable external services to react to key operations (e.g., PUT, DELETE) in near real-time, supporting use cases such as analytics, compliance, and monitoring. It also helps bring Ozone closer to feature parity with object stores like Amazon S3. + +## Use Cases + +- **Triggering downstream data processing:** e.g., trigger a Spark or Flink job on object creation. +- **Data Backup and Replication** +- **Monitoring and Alerts** + +## Design Principles + +- **Cloud-Agnostic Integration:** Inspired by MinIO and Ceph, this design avoids dependence on managed services like AWS SNS/SQS, and instead supports self-managed targets (e.g., Kafka, RabbitMQ). +- **Separation of Concerns:** Event delivery is decoupled from core Ozone operations. Ozone handles event generation and persistence; external systems handle delivery and processing. + +## Goals & Non-Goals + +These design choices are guided by two main considerations: + +1. Refer to MinIO and Ceph: The design favors deployment in on-premises or cloud-agnostic environments by avoiding reliance on managed cloud services. +2. Decoupling event delivery from Ozone internals. + +### Goals + +- Provide Java APIs to configure bucket-level notifications and target configurations. +- Support CLI tools to manage notification settings. +- Support initial delivery targets: Kafka and RabbitMQ. +- Enable filtering of notifications via object prefixes. +- Support event types: object creation, deletion, and tagging. +- Support for AWS-compatible REST APIs (e.g., GET/PUT Bucket Notification). + +### Non-Goals + +- Only a subset of Ozone-supported events can be configured for notifications. +- The design does not guarantee any delivery semantics (except for synchronous failure responses). + +## Event Types + +### Support + +Currently, Apache Ozone can publish notifications for the following events: +- New object created events +- Object removal events +- Object tagging events + +### Not Support + +Those can be published by Amazon S3 but are not supported in Ozone. See [S3 Event Notification](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html#notification-how-to-overview). + +- Reduced Redundancy Storage (RRS) object lost events +- Replication events +- Restore object events +- S3 Lifecycle expiration/transition events +- S3 Intelligent-Tiering automatic archival events +- Object ACL PUT events + +### Mapping Table +Here is the mapping between OM audit actions and S3 event types. + +| Ozone Manager Action | S3 Event Type | Notes | +|-------------------------------|------------------------------------------|--------------------------------| +| COMMIT_KEY (via PUT/Copy) | `s3:ObjectCreated:Put`, `Copy` | No distinction between them | +| COMPLETE_MULTIPART_UPLOAD | `s3:ObjectCreated:CompleteMultipartUpload`| | +| DELETE_KEY | `s3:ObjectRemoved:Delete` | | +| PUT_OBJECT_TAGGING | `s3:ObjectTagging:Put` | | +| DELETE_OBJECT_TAGGING | `s3:ObjectTagging:Delete` | | + +--- + +## Design + +### Overview + +A callback is introduced post-Ratis commit and pre-client response to handle event notification logic. Review Comment: No, actually the callback will run after the request has been completed. Maybe I wasn't clear enough. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
