[
https://issues.apache.org/jira/browse/HADOOP-19536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18010306#comment-18010306
]
Ayush Shukla commented on HADOOP-19536:
---------------------------------------
Hey [~ste...@apache.org] , I will be working on this ticket.
To answer you questions this will be a feature of S3A which will be operated
through configs. Any component when interacting to S3 using S3A, if provided
with relevant configs (which will be newly created) then the tags will be
applied in S3. These configs can be added while creating an object or deleting
an object in S3.
Config for adding the tag : fs.s3a.object.tag.* (comma separated key and
values) or fs.s3a.object.tag.\{TAG_NAME} (separate configs)
Config for deleting the tag : {{fs.s3a.soft.delete.enabled=true}}
Example :
1. hadoop fs -Dfs.s3a.object.tag.department=finance,project=alpha -put file.txt
s3a://bucket/path/
or adding conf in different lines : --conf
spark.hadoop.fs.s3a.object.tag.department=finance \
--conf spark.hadoop.fs.s3a.object.tag.project=alpha \
This command will add tags with key as department and project and value as
finance and alpha respectively.
2. hadoop fs \
-Dfs.s3a.soft.delete.enabled=true \
-Dfs.s3a.soft.delete.tag.key=archive \
-Dfs.s3a.soft.delete.tag.value=true \
-rm s3a://ayshukla-emr-dev/tagged-file27.txt
In this tagged-file27.txt will not be deleted. Instead a tag will be added with
key as archive and value as true (since those are defined by user).
Here if no key and value tag are added then a default delete tag can be added.
For example key is status and value as deleted.
Documented this in the attached pdf.
> S3A : Add option for custom S3 tags while writing and deleting S3 objects
> -------------------------------------------------------------------------
>
> Key: HADOOP-19536
> URL: https://issues.apache.org/jira/browse/HADOOP-19536
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Reporter: Syed Shameerur Rahman
> Priority: Major
>
> Custom S3 object
> [tags|https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html]
> can be added to S3 objects while writing and deleting.
> *Use Case:*
> S3 tags can be used to categorize the
> [objects|https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html]
> and potentially apply bucket level polices to take some actions.
> For example : objects can be marked as "to-be-glacier" and based on some
> bucket policy the written objects can be moved to Glacier tier after sometime
> for cost savings.
> Apache iceberg's [S3FileIO|#s3-tags]] also uses S3 Tags for soft deletes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
