raphaelazzolini opened a new pull request, #6874:
URL: https://github.com/apache/hadoop/pull/6874
Add the property fs.s3a.encryption.context that allow users to specify the
AWS KMS Encryption Context to be used in S3A.
The value of the encryption context is a key/value string that will be
Base64 encoded and set in the parameter ssekmsEncryptionContext from the S3
client.
Contributed by Raphael Azzolini
### Description of PR
This code change adds a new property to S3A: fs.s3a.encryption.context\
The property's value accepts a set of key/value attributes to be set on S3's
encryption context. The value of the property will be base64 encoded and set in
the parameter ssekmsEncryptionContext from the S3 client.
### How was this patch tested?
S3's head-object response doesn't contain the object encryption key.
Therefore, I enabled CloudTrails data logs in my bucket to verify that the
tests were passing the encryption context to the request.
I added this property to `auth-keys.xml`
```
<property>
<name>fs.s3a.encryption.context</name>
<value>
project=hadoop,
jira=HADOOP-19197,
component=fs/s3
</value>
</property>
```
Then I executed the following tests:
```
mvn clean verify -Dit.test=ITestS3AEncryption* -Dtest=none
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running
org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKeyWithEncryptionContext
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.10
s -- in
org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKeyWithEncryptionContext
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
48.17 s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0
s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[INFO] Running
org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKeyWithEncryptionContext
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.575
s -- in
org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKeyWithEncryptionContext
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.246
s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[INFO] Running
org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed:
2.600 s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[INFO] Running
org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.414
s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.680
s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Running
org.apache.hadoop.fs.s3a.ITestS3AEncryptionDSSEKMSUserDefinedKeyWithEncryptionContext
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.538
s -- in
org.apache.hadoop.fs.s3a.ITestS3AEncryptionDSSEKMSUserDefinedKeyWithEncryptionContext
[INFO] Running
org.apache.hadoop.fs.s3a.ITestS3AEncryptionDSSEKMSUserDefinedKey
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.425
s -- in org.apache.hadoop.fs.s3a.ITestS3AEncryptionDSSEKMSUserDefinedKey
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 53, Failures: 0, Errors: 0, Skipped: 6
```
```
mvn clean verify -Dit.test=TestMarshalledCredentials -Dtest=none
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.auth.TestMarshalledCredentials
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.133
s -- in org.apache.hadoop.fs.s3a.auth.TestMarshalledCredentials
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
```
```
mvn clean verify -Dit.test=ITestS3AHugeFilesEncryption -Dtest=none
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesEncryption
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
33.04 s -- in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesEncryption
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
```
Finally, I verified in the CloudTrail logs, that the encryption context was
set for both `aws:kms` and `aws:kms:dsse`.
```
(...)
{
"eventTime": "2024-06-08T03:49:49Z",
"eventSource": "s3.amazonaws.com",
"eventName": "PutObject",
"awsRegion": "us-west-1",
"userAgent": "[Hadoop 3.5.0-SNAPSHOT, aws-sdk-java/2.24.6
Linux/5.10.217-183.860.amzn2int.x86_64 OpenJDK_64-Bit_Server_VM/25.412-b08
Java/1.8.0_412 vendor/Private_Build io/sync http/Apache cfg/retry-mode/adaptive
hll/cross-region ft/s3-transfer]",
"requestParameters": {
"bucketName": "azzolini-us-west-1",
"x-amz-server-side-encryption-aws-kms-key-id":
"arn:aws:kms:us-west-1:809092095835:key/03056d93-8d15-465f-8f66-c4e5b05a7bbc",
"Host": "azzolini-us-west-1.s3.us-west-1.amazonaws.com",
"x-amz-server-side-encryption": "aws:kms:dsse",
"x-amz-server-side-encryption-context":
"eyJjb21wb25lbnQiOiJmcy9zMyIsInByb2plY3QiOiJoYWRvb3AiLCJqaXJhIjoiSEFET09QLTE5MTk3In0=",
"key": "test/"
},
(...)
```
```
(...)
"awsRegion": "us-west-1",
"sourceIPAddress": "204.246.162.39",
"userAgent": "[Hadoop 3.5.0-SNAPSHOT, aws-sdk-java/2.24.6
Linux/5.10.217-183.860.amzn2int.x86_64 OpenJDK_64-Bit_Server_VM/25.412-b08
Java/1.8.0_412 vendor/Private_Build io/sync http/Apache cfg/retry-mode/adaptive
hll/cross-region ft/s3-transfer]",
"requestParameters": {
"bucketName": "azzolini-us-west-1",
"x-amz-server-side-encryption-aws-kms-key-id":
"arn:aws:kms:us-west-1:809092095835:key/03056d93-8d15-465f-8f66-c4e5b05a7bbc",
"Host": "azzolini-us-west-1.s3.us-west-1.amazonaws.com",
"x-amz-server-side-encryption": "aws:kms",
"x-amz-server-side-encryption-context":
"eyJjb21wb25lbnQiOiJmcy9zMyIsInByb2plY3QiOiJoYWRvb3AiLCJqaXJhIjoiSEFET09QLTE5MTk3In0=",
"key": "test/testEncryptionOverRename-0400"
},
(...)
```
```
echo
eyJjb21wb25lbnQiOiJmcy9zMyIsInByb2plY3QiOiJoYWRvb3AiLCJqaXJhIjoiSEFET09QLTE5MTk3In0=
| base64 --decode
{"component":"fs/s3","project":"hadoop","jira":"HADOOP-19197"}%
```
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [X] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [-] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [-] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
