liuml07 commented on a change in pull request #1875: HADOOP-16794. S3A reverts
KMS encryption to the bucket's default KMS …
URL: https://github.com/apache/hadoop/pull/1875#discussion_r389237849
##########
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractTestS3AEncryption.java
##########
@@ -42,11 +46,24 @@
protected Configuration createConfiguration() {
Configuration conf = super.createConfiguration();
S3ATestUtils.disableFilesystemCaching(conf);
- conf.set(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM,
- getSSEAlgorithm().getMethod());
+ patchConfigurationEncryptionSettings(conf);
return conf;
}
+ /**
+ * This removes the encryption settings from the
Review comment:
OK, I did some investigation, and the observation is as following:
1. The failing tests are just about tests. The fix in copy options should
work as expected. Specially, some encryption tests fail only when bucket level
encryption is set.
1. The inconsistent encryption algorithm and key situation in conf is
*partially* because of the `addConfResource(CONTRACT_XML);` function after
creating and patching the conf object, see
[here](https://github.com/apache/hadoop/blob/9cad3e235026dbe4658705ca85d263d0edf14521/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/S3AContract.java#L36).
1. The call to `addConfResource(CONTRACT_XML);` should be considered a
tricky unrelated bug, not brought by this patch. Most likely it is fixed by
HADOOP-16626 in `trunk`. Guess that's why we don't see test failures there. I'm
wondering how `trunk` is not failing this...? Was HADOOP-16626 enough?
1. Specially, there are some discussions about the Configuration loading
resources "after" the conf object has been patched when setting up the S3
tests. In our test case, if you have the change to call
`removeBucketOverrides()`, and you have the change in `S3AContract` not calling
`addConfResource(CONTRACT_XML);`, then you can test with this: you add multiple
`testEncryption()` and multiple `testEncryptionOverRename()` test cases in
`AbstractTestS3AEncryption`
([example](https://github.com/liuml07/hadoop/commit/1f9fc1b8710af56cde2fbd362556086c6b6f0f32)).
Guess what, only one test fail and all others are failing...How interesting?
This seems very related to static resource loading when creating
S3AFileSystem...
I'll stop here since I for now have no more context. I will request
@steveloughran to provide more help. Shall we backport HADOOP-16626 to
{{branch-3.2}} and see?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
