ivandika3 commented on code in PR #6589: URL: https://github.com/apache/ozone/pull/6589#discussion_r1580578213
########## hadoop-hdds/docs/content/design/lifecycle-configurations.md: ########## @@ -0,0 +1,251 @@ +--- +title: AWS S3 Lifecycle Configurations +summary: Enables users to manage lifecycle configurations for buckets, allowing automated deletion of keys based on predefined rules. +date: 2024-04-25 +jira: HDDS-8342 +status: draft +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Lifecycle Management + +## Introduction +I encountered the need for a retention solution within my cluster, specifically the ability to delete keys in specific paths after a certain time period. +This requirement closely resembled the functionality provided by AWS S3 Lifecycle configurations, particularly the Expiration part ([AWS S3 Lifecycle Configuration Examples](https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html)). +After designing and implementing a solution, I am eager to contribute it back to the Apache Ozone community. + +## Overview + +### Functionality +- User should be able to create/remove/fetch lifecycle configurations for a specific S3 bucket. +- The lifecycle configurations will be executed periodically. +- Depending on the rules of the lifecycle configuration there could be different actions or even multiple actions. +- At the moment only expiration is supported (keys get deleted). +- The lifecycle configurations supports all buckets not only S3 buckets. + + +### Components + +- Lifecycle configurations (will be stored in DB) consists of volumeName, bucketName and a list of rules + - A rule contains prefix (string), Expiration and an optional Filter. Review Comment: Currently, @xichen01 and I are working on the object tagging integrations for bucket lifecycle configuration. We can add it in the scope of this feature. ########## hadoop-hdds/docs/content/design/lifecycle-configurations.md: ########## @@ -0,0 +1,251 @@ +--- +title: AWS S3 Lifecycle Configurations +summary: Enables users to manage lifecycle configurations for buckets, allowing automated deletion of keys based on predefined rules. +date: 2024-04-25 +jira: HDDS-8342 +status: draft +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Lifecycle Management + +## Introduction +I encountered the need for a retention solution within my cluster, specifically the ability to delete keys in specific paths after a certain time period. +This requirement closely resembled the functionality provided by AWS S3 Lifecycle configurations, particularly the Expiration part ([AWS S3 Lifecycle Configuration Examples](https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html)). +After designing and implementing a solution, I am eager to contribute it back to the Apache Ozone community. + +## Overview + +### Functionality +- User should be able to create/remove/fetch lifecycle configurations for a specific S3 bucket. +- The lifecycle configurations will be executed periodically. +- Depending on the rules of the lifecycle configuration there could be different actions or even multiple actions. +- At the moment only expiration is supported (keys get deleted). +- The lifecycle configurations supports all buckets not only S3 buckets. + + +### Components + +- Lifecycle configurations (will be stored in DB) consists of volumeName, bucketName and a list of rules + - A rule contains prefix (string), Expiration and an optional Filter. + - Expiration contains either days (integer) or Date (long) + - Filter contains prefix (string). +- S3G bucket endpoint needs few updates to accept ?/lifecycle +- ClientProtocol and all implementers provides (get, list, delete and create) lifecycle configuration +- RetentionManager will be running periodically. + - Fetches a lifecycle configurations list with the help of OM + - Executes each lifecycle configuration on a specific bucket + - Lifecycle configurations will be running on parallel (each one against different bucket). + + +### Flow +1. Users interact with lifecycle configurations via S3Gateway. +2. Configuration details are processed by a handler. +3. Configurations are saved/fetched from the database. +4. RetentionManager, running periodically in the Leader OM, executes lifecycle configurations and issues deletions for eligible keys. + +## Limitations +- The current solution lacks certain features: + - Filter doesn't support `AND` operations. Review Comment: I think we can remove this, @xichen01 is currently working on the `AND` operations. We can include it in the scope of this feature and added it in `Components` section. @xichen01 Could you help to update the `AND` operation protobuf definitions? ########## hadoop-hdds/docs/content/design/lifecycle-configurations.md: ########## @@ -0,0 +1,251 @@ +--- +title: AWS S3 Lifecycle Configurations +summary: Enables users to manage lifecycle configurations for buckets, allowing automated deletion of keys based on predefined rules. +date: 2024-04-25 +jira: HDDS-8342 +status: draft +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Lifecycle Management + +## Introduction +I encountered the need for a retention solution within my cluster, specifically the ability to delete keys in specific paths after a certain time period. +This requirement closely resembled the functionality provided by AWS S3 Lifecycle configurations, particularly the Expiration part ([AWS S3 Lifecycle Configuration Examples](https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html)). +After designing and implementing a solution, I am eager to contribute it back to the Apache Ozone community. + +## Overview + +### Functionality +- User should be able to create/remove/fetch lifecycle configurations for a specific S3 bucket. +- The lifecycle configurations will be executed periodically. +- Depending on the rules of the lifecycle configuration there could be different actions or even multiple actions. +- At the moment only expiration is supported (keys get deleted). +- The lifecycle configurations supports all buckets not only S3 buckets. + + +### Components + +- Lifecycle configurations (will be stored in DB) consists of volumeName, bucketName and a list of rules + - A rule contains prefix (string), Expiration and an optional Filter. + - Expiration contains either days (integer) or Date (long) + - Filter contains prefix (string). +- S3G bucket endpoint needs few updates to accept ?/lifecycle +- ClientProtocol and all implementers provides (get, list, delete and create) lifecycle configuration +- RetentionManager will be running periodically. + - Fetches a lifecycle configurations list with the help of OM + - Executes each lifecycle configuration on a specific bucket + - Lifecycle configurations will be running on parallel (each one against different bucket). + + +### Flow +1. Users interact with lifecycle configurations via S3Gateway. +2. Configuration details are processed by a handler. +3. Configurations are saved/fetched from the database. +4. RetentionManager, running periodically in the Leader OM, executes lifecycle configurations and issues deletions for eligible keys. + +## Limitations +- The current solution lacks certain features: + - Filter doesn't support `AND` operations. + - Only expiration actions are supported. + - Lack of CLI support for managing lifecycle configurations across all buckets (S3G is the only supported entry point). + - Review Comment: Nit: Remove this bullet point. ########## hadoop-hdds/docs/content/design/lifecycle-configurations.md: ########## @@ -0,0 +1,251 @@ +--- +title: AWS S3 Lifecycle Configurations +summary: Enables users to manage lifecycle configurations for buckets, allowing automated deletion of keys based on predefined rules. +date: 2024-04-25 +jira: HDDS-8342 +status: draft +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Lifecycle Management + +## Introduction +I encountered the need for a retention solution within my cluster, specifically the ability to delete keys in specific paths after a certain time period. +This requirement closely resembled the functionality provided by AWS S3 Lifecycle configurations, particularly the Expiration part ([AWS S3 Lifecycle Configuration Examples](https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html)). +After designing and implementing a solution, I am eager to contribute it back to the Apache Ozone community. Review Comment: Nit: I think line 25 is unnecessary for a design doc, we can remove it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
