[
https://issues.apache.org/jira/browse/HADOOP-17966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-17966.
-------------------------------------
Resolution: Duplicate
> S3A SSE-KMS inconsistency issue during rename
> ---------------------------------------------
>
> Key: HADOOP-17966
> URL: https://issues.apache.org/jira/browse/HADOOP-17966
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
> Affects Versions: 3.1.2
> Reporter: Dong0829
> Priority: Major
>
> According to the document:
>
> [https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/encryption.html#S3_Default_Encryption]
> "Organizations may define a default key in the Amazon KMS; if a default key
> is set, then it will be used whenever SSE-KMS encryption is chosen and the
> value of fs.s3a.server-side-encryption.key is empty."
> So basically two conditions to make the object with default KMS: 1. Set
> SSE-KMS encryption 2. Did not set fs.s3a.server-side-encryption.key
> But there is another confusing scenario below:
> 1. User want to rely on s3 bucket side encryption using their customer KMS
> key(kms-keyA, for example), so user did not set
> fs.s3a.server-side-encryption-algorithm or fs.s3a.server-side-encryption.key,
> and the files uploaded to this bucket will use bucket custom KMS key kms-keyA
> 2. Next step, user want to copy the file to other file using s3a, the
> process will invoke copyFile() in S3AFileSystem, during the copy, s3a will
> clone the meta data of the source in cloneObjectMetadata(), in the clone,
> there is copy of SSE algorithm but no specific kms key copy for the SSE-KMS,
> it will cause the destination using SSE-KMS without any key id, the final
> file will use account level default key under aws/s3(
> [https://docs.aws.amazon.com/cli/latest/reference/s3api/put-object.html),]
> lets say its kms-keyB.
> It means when ever there is a copy, the kms key will be changed from customer
> key kms-keyA to kms-keyB, which will cause inconsistency, for example:
> hdfs dfs -put test s3://ssetest/
> During this put, there will be rename processing from test.__COPYING__ to
> test, it will cause the final test file encrypted with account default key
> kms-keyB instead of s3 bucket customer key kms-keyA which is expected. Should
> we consider to clone the KMS key id also to keep the consistency?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
