sahilTakiar opened a new pull request #1764: HADOOP-16346: Stabilize S3A OpenSSL support URL: https://github.com/apache/hadoop/pull/1764 [HADOOP-16346: Stabilize S3A OpenSSL support](https://issues.apache.org/jira/browse/HADOOP-16346) Related JIRAs: * [HADOOP-15669: ABFS: Improve HTTPS Performance](https://issues.apache.org/jira/browse/HADOOP-15669) * [HADOOP-16050: s3a SSL connections should use OpenSSL](https://issues.apache.org/jira/browse/HADOOP-16050) * [HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8](https://issues.apache.org/jira/browse/HADOOP-16371) Introduces `openssl` as an option for `fs.s3a.ssl.channel.mode`. The new option is documented and marked as experimental. Summary of changes: * Made `wildfly-openssl` a runtime dependency (rather than a compile time dependency for ABFS and S3A); only `wildfly-openssl-java` is needed as a compile time dependency * Fixed HADOOP-15851 / HADOOP-16174 when `SSLChannelMode.OpenSSL` is specified in `DelegatingSSLSocketFactory` * Docs updates to `core-default.xml` and S3A's `performance.md` Testing Summary: * Since this patch caused several issues last time it was merged, most of my effort has been focused on testing * So far, I've ran all the S3A tests, ran some manual S3A CLI tests, and ran tests with Impala (IMPALA-8577); so far everything is looking good * The default value of `fs.s3a.ssl.channel.mode` hasn't changed, so this patch doesn't expose any new default changes to users * I'm planning to run ABFS tests next; and will try to run some basic Hive, MR, Spark tests, however, would appreciate any helping in testing this patch as I don't have an env for Hive, MR, Spark, etc. handy Completed Testing: * Confirmed that if native libraries are not build, the OpenSSL tests are skipped. * Unit, integration, and scale tests run with S3Guard enabled (DynamoDB, auth mode) and `test.fs.s3a.sts.enabled = true` `mvn clean verify -Ds3guard -Ddynamodb -Dscale -Dauth` ``` [INFO] Running org.apache.hadoop.mapreduce.filecache.TestS3AResourceScope [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.102 s - in org.apache.hadoop.mapreduce.filecache.TestS3AResourceScope [INFO] Running org.apache.hadoop.fs.s3a.impl.TestNeworkBinding [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.162 s - in org.apache.hadoop.fs.s3a.impl.TestNeworkBinding [INFO] Running org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.629 s - in org.apache.hadoop.fs.s3a.impl.TestPartialDeleteFailures [INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestObjectChangeDetectionAttributes [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.675 s - in org.apache.hadoop.fs.s3a.s3guard.TestObjectChangeDetectionAttributes [INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestLocalMetadataStore [INFO] Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.363 s - in org.apache.hadoop.fs.s3a.s3guard.TestLocalMetadataStore [INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestPathMetadataDynamoDBTranslation [INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.004 s - in org.apache.hadoop.fs.s3a.s3guard.TestPathMetadataDynamoDBTranslation [INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestNullMetadataStore [WARNING] Tests run: 29, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 3.926 s - in org.apache.hadoop.fs.s3a.s3guard.TestNullMetadataStore [INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestDynamoDBMiscOperations [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.698 s - in org.apache.hadoop.fs.s3a.s3guard.TestDynamoDBMiscOperations [INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestDirListingMetadata [INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 s - in org.apache.hadoop.fs.s3a.s3guard.TestDirListingMetadata [INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestS3GuardCLI [INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.165 s - in org.apache.hadoop.fs.s3a.s3guard.TestS3GuardCLI [INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestPathOrderComparators [INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124 s - in org.apache.hadoop.fs.s3a.s3guard.TestPathOrderComparators [INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestS3Guard [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.771 s - in org.apache.hadoop.fs.s3a.s3guard.TestS3Guard [INFO] Running org.apache.hadoop.fs.s3a.TestSSEConfiguration [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.839 s - in org.apache.hadoop.fs.s3a.TestSSEConfiguration [INFO] Running org.apache.hadoop.fs.s3a.TestListing [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.193 s - in org.apache.hadoop.fs.s3a.TestListing [INFO] Running org.apache.hadoop.fs.s3a.TestS3AInputPolicies [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.124 s - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies [INFO] Running org.apache.hadoop.fs.s3a.TestDataBlocks [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in org.apache.hadoop.fs.s3a.TestDataBlocks [INFO] Running org.apache.hadoop.fs.s3a.TestS3AGetFileStatus [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.371 s - in org.apache.hadoop.fs.s3a.TestS3AGetFileStatus [INFO] Running org.apache.hadoop.fs.s3a.commit.TestMagicCommitPaths [INFO] Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.121 s - in org.apache.hadoop.fs.s3a.commit.TestMagicCommitPaths [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestDirectoryCommitterScale [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.473 s - in org.apache.hadoop.fs.s3a.commit.staging.TestDirectoryCommitterScale [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.481 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedTaskCommit [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingCommitter [INFO] Tests run: 48, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.554 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingCommitter [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingDirectoryOutputCommitter [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.195 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingDirectoryOutputCommitter [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestPaths [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.442 s - in org.apache.hadoop.fs.s3a.commit.staging.TestPaths [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.139 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedFileListing [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedJobCommit [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.786 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingPartitionedJobCommit [INFO] Running org.apache.hadoop.fs.s3a.commit.TestTasks [INFO] Tests run: 60, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.984 s - in org.apache.hadoop.fs.s3a.commit.TestTasks [INFO] Running org.apache.hadoop.fs.s3a.TestS3ABlockOutputStream [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.322 s - in org.apache.hadoop.fs.s3a.TestS3ABlockOutputStream [INFO] Running org.apache.hadoop.fs.s3a.TestS3AMultipartUploaderSupport [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.123 s - in org.apache.hadoop.fs.s3a.TestS3AMultipartUploaderSupport [INFO] Running org.apache.hadoop.fs.s3a.TestStreamChangeTracker [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.144 s - in org.apache.hadoop.fs.s3a.TestStreamChangeTracker [INFO] Running org.apache.hadoop.fs.s3a.TestS3AUnbuffer [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.397 s - in org.apache.hadoop.fs.s3a.TestS3AUnbuffer [INFO] Running org.apache.hadoop.fs.s3a.TestInvoker [INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.796 s - in org.apache.hadoop.fs.s3a.TestInvoker [INFO] Running org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider [INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.211 s - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider [INFO] Running org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation [INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.15 s - in org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation [INFO] Running org.apache.hadoop.fs.s3a.auth.TestMarshalledCredentials [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.205 s - in org.apache.hadoop.fs.s3a.auth.TestMarshalledCredentials [INFO] Running org.apache.hadoop.fs.s3a.auth.TestSignerManager [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.512 s - in org.apache.hadoop.fs.s3a.auth.TestSignerManager [INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.TestS3ADelegationTokenSupport [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.301 s - in org.apache.hadoop.fs.s3a.auth.delegation.TestS3ADelegationTokenSupport [INFO] Running org.apache.hadoop.fs.s3native.TestS3xLoginHelper [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.115 s - in org.apache.hadoop.fs.s3native.TestS3xLoginHelper [INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3Select [INFO] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 286.283 s - in org.apache.hadoop.fs.s3a.select.ITestS3Select [INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectCLI [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.11 s - in org.apache.hadoop.fs.s3a.select.ITestS3SelectCLI [INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectMRJob [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.307 s - in org.apache.hadoop.fs.s3a.select.ITestS3SelectMRJob [INFO] Running org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.101 s - in org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat [INFO] Running org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes [WARNING] Tests run: 24, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 17.655 s - in org.apache.hadoop.fs.s3a.impl.ITestPartialRenamesDeletes [INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract [INFO] Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 105.554 s - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract [INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.001 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation [INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal [WARNING] Tests run: 33, Failures: 0, Errors: 0, Skipped: 33, Time elapsed: 33.82 s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal [INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck [INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.194 s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardFsck [INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.055 s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps [INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB [INFO] Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.832 s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB [INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale [WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 47.689 s - in org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStoreScale [INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations [WARNING] Tests run: 7, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 28.767 s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardDDBRootOperations [INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.015 s - in org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore [INFO] Running org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.703 s - in org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService [INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.245 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray [INFO] Running org.apache.hadoop.fs.s3a.ITestS3ACopyFromLocalFile [WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 9.667 s - in org.apache.hadoop.fs.s3a.ITestS3ACopyFromLocalFile [INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC [WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 27.269 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC [INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk [WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 11.255 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk [INFO] Running org.apache.hadoop.fs.s3a.ITestLocatedFileStatusFetcher [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.688 s - in org.apache.hadoop.fs.s3a.ITestLocatedFileStatusFetcher [INFO] Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey [WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.057 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey [INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext [INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI [WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 94.633 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI [INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.603 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics [INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.29 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir [INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations [WARNING] Tests run: 73, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 304.863 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations [INFO] Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.987 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil [INFO] Running org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations [INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 574.584 s - in org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations [INFO] Running org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 142.813 s - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency [INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.276 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer [INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost [WARNING] Tests run: 20, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 55.194 s - in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost [INFO] Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration [INFO] Tests run: 30, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.533 s - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration [INFO] Running org.apache.hadoop.fs.s3a.commit.ITestCommitOperations [INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 90.13 s - in org.apache.hadoop.fs.s3a.commit.ITestCommitOperations [INFO] Running org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob [ERROR] Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 216.553 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob [ERROR] test_200_execute[magic](org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob) Time elapsed: 97.74 s <<< FAILURE! java.lang.AssertionError: Job job_1576269085620_0003 failed in state FAILED with cause Job commit failed: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@23864d21 rejected from org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor@47e32f65[Terminated, p ool size = 0, active threads = 0, queued tasks = 0, completed tasks = 10] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.apache.hadoop.fs.s3a.commit.Tasks$Builder.runParallel(Tasks.java:313) at org.apache.hadoop.fs.s3a.commit.Tasks$Builder.run(Tasks.java:148) at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.commitPendingUploads(AbstractS3ACommitter.java:480) at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.commitJobInternal(AbstractS3ACommitter.java:620) at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.commitJob(AbstractS3ACommitter.java:722) at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:286) at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:238) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) . Consult logs under[...]/hadoop/hadoop-tools/hadoop-aws/target/yarn-2019-12-13-20.31.23.45 at org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob.test_200_execute(ITestS3ACommitterMRJob.java:304) [INFO] Running org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.719 s - in org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocol [INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 259.035 s - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITestStagingCommitProtocol [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitProtocol [INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 270.709 s - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITestDirectoryCommitProtocol [INFO] Running org.apache.hadoop.fs.s3a.commit.staging.integration.ITestPartitionedCommitProtocol [WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 247.788 s - in org.apache.hadoop.fs.s3a.commit.staging.integration.ITestPartitionedCommitProtocol [INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol [INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 226.317 s - in org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol [INFO] Running org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.439 s - in org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits [INFO] Running org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A ``` * Same test setup as above, but with `fs.s3a.ssl.channel.mode = openssl` * All tests passed, including `ITestS3ACommitterMRJob` (which I presume is a flaky test) * Verified that setting `fs.s3a.ssl.channel.mode` to `openssl` uses WildFly OpenSSL; took a jstack while running an S3A test: ``` java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at org.wildfly.openssl.OpenSSLSocket.runHandshake(OpenSSLSocket.java:306) at org.wildfly.openssl.OpenSSLSocket.startHandshake(OpenSSLSocket.java:210) at com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy18.connect(Unknown Source) at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1297) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4920) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4866) at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1394) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1333) at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:507) ``` * Performed some tests using the CLI; ran the following commands: ``` mvn package -Pdist -DskipTests -Dmaven.javadoc.skip=true -DskipShade -Pnative tar -xvf hadoop-dist/target/hadoop-3.3.0-SNAPSHOT.tar.gz cd hadoop-3.3.0-SNAPSHOT/ export HADOOP_CLASSPATH=$(find share/hadoop/ -name '*.jar' | xargs echo | tr ' ' ':') ./bin/hadoop fs -ls s3a://[bucket-name]/ 2019-12-13 16:05:14,031 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties 2019-12-13 16:05:14,091 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 2019-12-13 16:05:14,091 INFO impl.MetricsSystemImpl: s3a-file-system metrics system started 2019-12-13 16:05:15,424 INFO s3a.S3AFileSystem: S3Guard is disabled on this bucket: ... ... 2019-12-13 16:05:15,765 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system... 2019-12-13 16:05:15,765 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped. 2019-12-13 16:05:15,765 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete. ``` * Confirmed that when setting `fs.s3a.ssl.channel.mode` to `default` and without setting `org.wildfly.openssl.path`, the CLI still works; confirmed that when DEBUG logs are enabled, the following log line is printed: ``` 2019-12-13 16:50:04,864 DEBUG ssl.DelegatingSSLSocketFactory: Failed to load OpenSSL. Falling back to the JSSE default. ``` * Confirmed that when setting `fs.s3a.ssl.channel.mode` to `openssl` and without setting `org.wildfly.openssl.path`, the CLI fails (as expected) with the following error: ``` 2019-12-13 16:55:01,146 WARN fs.FileSystem: Failed to initialize fileystem s3a://cloudera-dev-hive-on-s3/: java.io.IOException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: openssl.TLS, provider: openssl, class: org.wildfly.openssl.OpenSSLContextSPI$OpenSSLTLSContextSpi) ls: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: openssl.TLS, provider: openssl, class: org.wildfly.openssl.OpenSSLContextSPI$OpenSSLTLSContextSpi) ``` * Confirmed that when setting `fs.s3a.ssl.channel.mode` to `openssl` and setting `org.wildfly.openssl.path` (`export HADOOP_OPTS="-Dorg.wildfly.openssl.path=/usr/lib/x86_64-linux-gnu/"`), the CLI works (stdout is the same as above).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
