This is an automated email from the ASF dual-hosted git repository.
sivabalan pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new e356a32 [HUDI-3496] Add note for S3 Versioned Bucket (#4900)
e356a32 is described below
commit e356a32f5308a7812ad278f2e4dac33fe9117b8b
Author: Vinay Patil <[email protected]>
AuthorDate: Fri Feb 25 09:50:58 2022 +0530
[HUDI-3496] Add note for S3 Versioned Bucket (#4900)
---
website/docs/s3_hoodie.md | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/website/docs/s3_hoodie.md b/website/docs/s3_hoodie.md
index c39c73c..a67de25 100644
--- a/website/docs/s3_hoodie.md
+++ b/website/docs/s3_hoodie.md
@@ -77,4 +77,11 @@ AWS hadoop libraries to add to our classpath
AWS glue data libraries are needed if AWS glue data is used
- com.amazonaws.glue:aws-glue-datacatalog-hive2-client:1.11.0
- - com.amazonaws:aws-java-sdk-glue:1.11.475
\ No newline at end of file
+ - com.amazonaws:aws-java-sdk-glue:1.11.475
+
+## AWS S3 Versioned Bucket
+
+With versioned buckets any object deleted creates a [Delete
Marker](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DeleteMarker.html),
as Hudi cleans up files using [Cleaner
utility](https://hudi.apache.org/docs/hoodie_cleaner) the number of Delete
Markers increases over time.
+It is important to configure the [Lifecycle
Rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html)
correctly
+to clean up these delete markers as the List operation can choke if the number
of delete markers reaches 1000.
+We recommend cleaning up Delete Markers after 1 day in Lifecycle Rule.
\ No newline at end of file
