[
https://issues.apache.org/jira/browse/HADOOP-19197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865988#comment-17865988
]
ASF GitHub Bot commented on HADOOP-19197:
-----------------------------------------
ahmarsuhail commented on code in PR #6874:
URL: https://github.com/apache/hadoop/pull/6874#discussion_r1675660341
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java:
##########
@@ -1402,6 +1406,79 @@ public static String getS3EncryptionKey(
}
}
+ /**
+ * Get any SSE context, without propagating exceptions from
+ * JCEKs files.
+ * @param bucket bucket to query for
+ * @param conf configuration to examine
+ * @return the encryption context value or ""
+ * @throws IllegalArgumentException bad arguments.
+ */
+ public static String getS3EncryptionContext(
Review Comment:
looks like this method is only used in tests?
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/EncryptionSecretOperations.java:
##########
@@ -61,4 +61,21 @@ public static Optional<String> getSSEAwsKMSKey(final
EncryptionSecrets secrets)
return Optional.empty();
}
}
+
+ /**
+ * Gets the SSE-KMS context if present, else don't set it in the S3 request.
+ *
+ * @param secrets source of the encryption secrets.
+ * @return an optional AWS KMS encryption context to attach to a request.
+ */
+ public static Optional<String> getSSEAwsKMSEncryptionContext(final
EncryptionSecrets secrets) {
+ if ((secrets.getEncryptionMethod() == S3AEncryptionMethods.SSE_KMS
+ || secrets.getEncryptionMethod() == S3AEncryptionMethods.DSSE_KMS)
+ && secrets.hasEncryptionKey()
Review Comment:
can't you have encryption context without defining a secret key, if you're
just using an AWS KMS key? so then you shouldn't have this condition?
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionDSSEKMSUserDefinedKeyWithEncryptionContext.java:
##########
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
Review Comment:
your ITests are not very useful as they don't assert on anything. I also
don't think you need 3 new ITests that extend AbstractTestS3AEncryption.
I recommend that you add a single ITest, that writes an object with
encryption context. parameterize it for SSE-KMS default key, SS-KMS customer
key, DSSE-KMS. Then do a GET on the object and assert on the encryption
context.
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java:
##########
@@ -1493,7 +1570,10 @@ public static EncryptionSecrets
buildEncryptionSecrets(String bucket,
LOG.debug("Data is unencrypted");
break;
}
- return new EncryptionSecrets(encryptionMethod, encryptionKey);
+
+ String encryptionContext = getS3EncryptionContextBase64Encoded(bucket,
conf,
Review Comment:
nit: move this statement above the switch
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestSSEConfiguration.java:
##########
@@ -147,39 +151,42 @@ void setProviderOption(final Configuration conf,
}
/**
- * Assert that the exception text from {@link #getAlgorithm(String, String)}
+ * Assert that the exception text from {@link #getAlgorithm(String, String,
String)}
* is as expected.
* @param expected expected substring in error
* @param alg algorithm to ask for
* @param key optional key value
+ * @param context optional encryption context value
* @throws Exception anything else which gets raised
*/
public void assertGetAlgorithmFails(String expected,
- final String alg, final String key) throws Exception {
+ final String alg, final String key, final String context) throws
Exception {
intercept(IOException.class, expected,
- () -> getAlgorithm(alg, key));
+ () -> getAlgorithm(alg, key, context));
}
private S3AEncryptionMethods getAlgorithm(S3AEncryptionMethods algorithm,
- String key)
+ String key,
+ String encryptionContext)
throws IOException {
- return getAlgorithm(algorithm.getMethod(), key);
+ return getAlgorithm(algorithm.getMethod(), key, encryptionContext);
}
- private S3AEncryptionMethods getAlgorithm(String algorithm, String key)
+ private S3AEncryptionMethods getAlgorithm(String algorithm, String key,
String encryptionContext)
throws IOException {
- return getEncryptionAlgorithm(BUCKET, buildConf(algorithm, key));
+ return getEncryptionAlgorithm(BUCKET, buildConf(algorithm, key,
encryptionContext));
}
/**
* Build a new configuration with the given S3-SSE algorithm
* and key.
* @param algorithm algorithm to use, may be null
* @param key key, may be null
+ * @param context encryption context, may be null
* @return the new config.
*/
@SuppressWarnings("deprecation")
- private Configuration buildConf(String algorithm, String key) {
+ private Configuration buildConf(String algorithm, String key, String
context) {
Review Comment:
nit: rename to encryptionContext
> S3A: Support AWS KMS Encryption Context
> ---------------------------------------
>
> Key: HADOOP-19197
> URL: https://issues.apache.org/jira/browse/HADOOP-19197
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Raphael Azzolini
> Priority: Major
> Labels: pull-request-available
>
> S3A properties allow users to choose the AWS KMS key
> ({_}fs.s3a.encryption.key{_}) and S3 encryption algorithm to be used
> (f{_}s.s3a.encryption.algorithm{_}). In addition to the AWS KMS Key, an
> encryption context can be used as non-secret data that adds additional
> integrity and authenticity to check the encrypted data. However, there is no
> option to specify the [AWS KMS Encryption
> Context|https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context]
> in S3A.
> In AWS SDK v2 the encryption context in S3 requests is set by the parameter
> [ssekmsEncryptionContext.|https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/model/CreateMultipartUploadRequest.Builder.html#ssekmsEncryptionContext(java.lang.String)]
> It receives a base64-encoded UTF-8 string holding JSON with the encryption
> context key-value pairs. The value of this parameter could be set by the user
> in a new property {_}*fs.s3a.encryption.context*{_}, and be stored in the
> [EncryptionSecrets|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/EncryptionSecrets.java]
> to later be used when setting the encryption parameters in
> [RequestFactoryImpl|https://github.com/apache/hadoop/blob/f92a8ab8ae54f11946412904973eb60404dee7ff/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
