[
https://issues.apache.org/jira/browse/HADOOP-19197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17901595#comment-17901595
]
ASF GitHub Bot commented on HADOOP-19197:
-----------------------------------------
raphaelazzolini opened a new pull request, #7193:
URL: https://github.com/apache/hadoop/pull/7193
Add the property fs.s3a.encryption.context that allow users to specify the
AWS KMS Encryption Context to be used in S3A.
The value of the encryption context is a key/value string that will be
Base64 encoded and set in the parameter ssekmsEncryptionContext from the S3
client.
Contributed by Raphael Azzolini
### Description of PR
This code change adds a new property to S3A: fs.s3a.encryption.context\
The property's value accepts a set of key/value attributes to be set on S3's
encryption context. The value of the property will be base64 encoded and set in
the parameter ssekmsEncryptionContext from the S3 client.
This change was merged to trunk by the pull request
https://github.com/apache/hadoop/pull/6874. This request is to merge the code
change to branch-3.4.
### How was this patch tested?
Tested in us-east-1 with `mvn -Dparallel-tests -DtestsThreadCount=16 clean
verify`
I added a new test `ITestS3AEncryptionSSEKMSWithEncryptionContext`. However,
S3's head-object response doesn't contain the object encryption key. Therefore,
I enabled CloudTrails data logs in my bucket to verify that the tests were
passing the encryption context to the request.
I added this property to `auth-keys.xml`
```
<property>
<name>fs.s3a.encryption.context</name>
<value>
project=hadoop,
jira=HADOOP-19197,
component=fs/s3
</value>
</property>
```
Then I executed the following tests:
```
mvn clean verify -Dit.test=ITestS3AEncryption* -Dtest=none
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running
org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKeyWithEncryptionContext
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.10
s
> S3A: Support AWS KMS Encryption Context
> ---------------------------------------
>
> Key: HADOOP-19197
> URL: https://issues.apache.org/jira/browse/HADOOP-19197
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Raphael Azzolini
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> S3A properties allow users to choose the AWS KMS key
> ({_}fs.s3a.encryption.key{_}) and S3 encryption algorithm to be used
> (f{_}s.s3a.encryption.algorithm{_}). In addition to the AWS KMS Key, an
> encryption context can be used as non-secret data that adds additional
> integrity and authenticity to check the encrypted data. However, there is no
> option to specify the [AWS KMS Encryption
> Context|https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context]
> in S3A.
> In AWS SDK v2 the encryption context in S3 requests is set by the parameter
> [ssekmsEncryptionContext.|https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/model/CreateMultipartUploadRequest.Builder.html#ssekmsEncryptionContext(java.lang.String)]
> It receives a base64-encoded UTF-8 string holding JSON with the encryption
> context key-value pairs. The value of this parameter could be set by the user
> in a new property {_}*fs.s3a.encryption.context*{_}, and be stored in the
> [EncryptionSecrets|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/EncryptionSecrets.java]
> to later be used when setting the encryption parameters in
> [RequestFactoryImpl|https://github.com/apache/hadoop/blob/f92a8ab8ae54f11946412904973eb60404dee7ff/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
