[
https://issues.apache.org/jira/browse/HADOOP-19336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17898615#comment-17898615
]
Steve Loughran commented on HADOOP-19336:
-----------------------------------------
text failures in detail.
This is the test which injects failures into the upload and then verifies that
retry of upload works, so address the way the v2 sdk doesn't handle this
properly.
{code}
[ERROR]
testSizeOfEncryptedObjectFromHeaderWithV1Compatibility(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)
Time elapsed: 1.701 s <<< FAILURE!
org.junit.ComparisonFailure: [Length of
s3a://stevel-london/job-00-fork-0009/test/testSizeOfEncryptedObjectFromHeaderWithV1Compatibility/file
status:
S3AFileStatus{path=s3a://stevel-london/job-00-fork-0009/test/testSizeOfEncryptedObjectFromHeaderWithV1Compatibility/file;
isDirectory=false; length=1024; replication=1; blocksize=33554432;
modification_time=1731674289000; access_time=0; owner=stevel; group=stevel;
permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true;
isErasureCoded=false} isEmptyDirectory=FALSE
eTag="0f343b0931126a20f133d67c2b018a3b"
versionId=JyA1I_OW8osQTS3zWdn_Z0qlQYqBZ_7.] expected:<10[]L> but was:<10[24]L>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryption.assertFileLength(ITestS3AClientSideEncryption.java:447)
at
org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryption.testSizeOfEncryptedObjectFromHeaderWithV1Compatibility(ITestS3AClientSideEncryption.java:345)
[ERROR]
testCommitOperations[array-commit-true](org.apache.hadoop.fs.s3a.commit.ITestUploadRecovery)
Time elapsed: 14.664 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSClientIOException: upload part #1 upload ID
9O39v3a8xF298CRy1gIbZ8pLvi9MCrJmEB3Z3u.BzsuH5b1swnDrPSW9rPQSph1nOVBF_SVAsgzCdCLGDy50irq_wyNs9ZZpGR5UzS6TuEFdzQzFHHx7I6VLm5Dq2jhF
on job-00-fork-0007/test/testCommitOperations[array-commit-true]:
software.amazon.encryption.s3.S3EncryptionClientException: Unable to execute
HTTP request: Encountered fatal error in publisher: Unable to execute HTTP
request: Encountered fatal error in publisher
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:231)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:207)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.uploadPart(WriteOperationHelper.java:593)
at
org.apache.hadoop.fs.s3a.commit.impl.CommitOperations.uploadFileData(CommitOperations.java:652)
at
org.apache.hadoop.fs.s3a.commit.impl.CommitOperations.uploadFileToPendingCommit(CommitOperations.java:583)
at
org.apache.hadoop.fs.s3a.commit.ITestUploadRecovery.testCommitOperations(ITestUploadRecovery.java:234)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to
execute HTTP request: Encountered fatal error in publisher
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at
software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
at
java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.IllegalStateException: Encountered fatal error in publisher
at
software.amazon.awssdk.utils.async.SimplePublisher.panicAndDie(SimplePublisher.java:339)
at
software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:226)
at
software.amazon.awssdk.utils.async.SimplePublisher.send(SimplePublisher.java:128)
at
software.amazon.awssdk.utils.async.InputStreamConsumingPublisher.doBlockingWrite(InputStreamConsumingPublisher.java:58)
at
software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:83)
at
software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.doBlockingWrite(InputStreamWithExecutorAsyncRequestBody.java:107)
at
software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.lambda$subscribe$0(InputStreamWithExecutorAsyncRequestBody.java:80)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.IllegalStateException: Must use either different key or iv
for GCM encryption
at com.sun.crypto.provider.CipherCore.checkReinit(CipherCore.java:1088)
at com.sun.crypto.provider.CipherCore.update(CipherCore.java:662)
at com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:380)
at javax.crypto.Cipher.update(Cipher.java:1835)
at
software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:52)
at
software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:16)
at
software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:267)
at
software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
... 10 more
[ERROR]
testMagicWriteRecovery[array-commit-true](org.apache.hadoop.fs.s3a.commit.ITestUploadRecovery)
Time elapsed: 11.485 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSClientIOException: upload part #1 upload ID
poj2y5bS3_3s.OaqLdPerW167MDOZ4BKi0MbwLK5JjQHq4dcEkxjlRnQD.zXrC2cZhjdfMMb2b0aoPYcRpwnBkFhlDFm1TL.mvb5EyU4g4LDz35yX490PwWi.Yqk7dRP
on job-00-fork-0007/test/testMagicWriteRecovery[array-commit-true]/file.txt:
software.amazon.encryption.s3.S3EncryptionClientException: Unable to execute
HTTP request: Encountered fatal error in publisher: Unable to execute HTTP
request: Encountered fatal error in publisher
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:231)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:207)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.uploadPart(WriteOperationHelper.java:593)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.lambda$uploadBlockAsync$1(S3ABlockOutputStream.java:1058)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to
execute HTTP request: Encountered fatal error in publisher
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at
software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
at
java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
... 3 more
Caused by: java.lang.IllegalStateException: Encountered fatal error in publisher
at
software.amazon.awssdk.utils.async.SimplePublisher.panicAndDie(SimplePublisher.java:339)
at
software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:226)
at
software.amazon.awssdk.utils.async.SimplePublisher.send(SimplePublisher.java:128)
at
software.amazon.awssdk.utils.async.InputStreamConsumingPublisher.doBlockingWrite(InputStreamConsumingPublisher.java:58)
at
software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:83)
at
software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.doBlockingWrite(InputStreamWithExecutorAsyncRequestBody.java:107)
at
software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.lambda$subscribe$0(InputStreamWithExecutorAsyncRequestBody.java:80)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.IllegalStateException: Must use either different key or iv
for GCM encryption
at com.sun.crypto.provider.CipherCore.checkReinit(CipherCore.java:1088)
at com.sun.crypto.provider.CipherCore.update(CipherCore.java:662)
at com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:380)
at javax.crypto.Cipher.update(Cipher.java:1835)
at
software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:52)
at
software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:16)
at
software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:267)
at
software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
... 10 more
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlocksize
[ERROR]
testMagicWriteRecovery[disk-commit-false](org.apache.hadoop.fs.s3a.commit.ITestUploadRecovery)
Time elapsed: 13.595 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSClientIOException: upload part #1 upload ID
lz.olcLNCZB8AJooWJc.aUTSVnnZqydBs9Xj_lWnbF5XSbeGzATEYAwibdBFHsXmurN025md274EYdptz0blTnUYBOh9bsyaJ4ksA7YMnLAb8OOAIxus2xvGCmN1hFhB
on job-00-fork-0007/test/testMagicWriteRecovery[disk-commit-false]/file.txt:
software.amazon.encryption.s3.S3EncryptionClientException: Unable to execute
HTTP request: Encountered fatal error in publisher: Unable to execute HTTP
request: Encountered fatal error in publisher
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:231)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:207)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.uploadPart(WriteOperationHelper.java:593)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.lambda$uploadBlockAsync$1(S3ABlockOutputStream.java:1058)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to
execute HTTP request: Encountered fatal error in publisher
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at
software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
at
java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
... 3 more
Caused by: java.lang.IllegalStateException: Encountered fatal error in publisher
at
software.amazon.awssdk.utils.async.SimplePublisher.panicAndDie(SimplePublisher.java:339)
at
software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:226)
at
software.amazon.awssdk.utils.async.SimplePublisher.send(SimplePublisher.java:128)
at
software.amazon.awssdk.utils.async.InputStreamConsumingPublisher.doBlockingWrite(InputStreamConsumingPublisher.java:58)
at
software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:83)
at
software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.doBlockingWrite(InputStreamWithExecutorAsyncRequestBody.java:107)
at
software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.lambda$subscribe$0(InputStreamWithExecutorAsyncRequestBody.java:80)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.IllegalStateException: Must use either different key or iv
for GCM encryption
at com.sun.crypto.provider.CipherCore.checkReinit(CipherCore.java:1088)
at com.sun.crypto.provider.CipherCore.update(CipherCore.java:662)
at com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:380)
at javax.crypto.Cipher.update(Cipher.java:1835)
at
software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:52)
at
software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:16)
at
software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:267)
at
software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
... 10 more
[ERROR]
testMagicWriteRecovery[bytebuffer-commit-false](org.apache.hadoop.fs.s3a.commit.ITestUploadRecovery)
Time elapsed: 13.09 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSClientIOException: upload part #1 upload ID
J_351iP7h8rmJCZ63AFvMekCewJmai.FhXjNf3IzBH0QErHp1b3AsxBzDtLK.ybHv5qb7U8L_nRqM8AWBHSpzdVGr3eJsNFr1btVUPGcchlAgKf9_qD_ADYpbeTNYNLj
on
job-00-fork-0007/test/testMagicWriteRecovery[bytebuffer-commit-false]/file.txt:
software.amazon.encryption.s3.S3EncryptionClientException: Unable to execute
HTTP request: Encountered fatal error in publisher: Unable to execute HTTP
request: Encountered fatal error in publisher
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:231)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:207)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.uploadPart(WriteOperationHelper.java:593)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.lambda$uploadBlockAsync$1(S3ABlockOutputStream.java:1058)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:225)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to
execute HTTP request: Encountered fatal error in publisher
at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at
software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
at
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
at
java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
... 3 more
Caused by: java.lang.IllegalStateException: Encountered fatal error in publisher
at
software.amazon.awssdk.utils.async.SimplePublisher.panicAndDie(SimplePublisher.java:339)
at
software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:226)
at
software.amazon.awssdk.utils.async.SimplePublisher.send(SimplePublisher.java:128)
at
software.amazon.awssdk.utils.async.InputStreamConsumingPublisher.doBlockingWrite(InputStreamConsumingPublisher.java:58)
at
software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:83)
at
software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.doBlockingWrite(InputStreamWithExecutorAsyncRequestBody.java:107)
at
software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.lambda$subscribe$0(InputStreamWithExecutorAsyncRequestBody.java:80)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.IllegalStateException: Must use either different key or iv
for GCM encryption
at com.sun.crypto.provider.CipherCore.checkReinit(CipherCore.java:1088)
at com.sun.crypto.provider.CipherCore.update(CipherCore.java:662)
at com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:380)
at javax.crypto.Cipher.update(Cipher.java:1835)
at
software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:52)
at
software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:16)
at
software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:267)
at
software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
... 10 more
{code}
> S3A: Test failures after CSE support added
> ------------------------------------------
>
> Key: HADOOP-19336
> URL: https://issues.apache.org/jira/browse/HADOOP-19336
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.5.0
> Reporter: Steve Loughran
> Assignee: Syed Shameerur Rahman
> Priority: Major
>
> Testing hadoop trunk with CSE-KMS configured I get
> * 400 error without region set (KMS providing nothing helpful). Proposed:
> move troubleshooting into encryption.md, cover 400 and this as a possible cuse
> * test failures
> {code}
> [ERROR]
> ITestS3AClientSideEncryptionKms>ITestS3AClientSideEncryption.testSizeOfEncryptedObjectFromHeaderWithV1Compatibility:345->ITestS3AClientSideEncryption.assertFileLength:447
> [Length of
> s3a://stevel-london/job-00-fork-0009/test/testSizeOfEncryptedObjectFromHeaderWithV1Compatibility/file
> status:
> S3AFileStatus{path=s3a://stevel-london/job-00-fork-0009/test/testSizeOfEncryptedObjectFromHeaderWithV1Compatibility/file;
> isDirectory=false; length=1024; replication=1; blocksize=33554432;
> modification_time=1731674289000; access_time=0; owner=stevel; group=stevel;
> permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true;
> isErasureCoded=false} isEmptyDirectory=FALSE
> eTag="0f343b0931126a20f133d67c2b018a3b"
> versionId=JyA1I_OW8osQTS3zWdn_Z0qlQYqBZ_7.] expected:<10[]L> but was:<10[24]L>
> [ERROR] ITestAwsSdkWorkarounds.testNoisyLogging:99 [LOG output does not
> contain the forbidden text. Has the SDK been fixed?]
> Expecting:
> <"">
> to contain:
> <"The provided S3AsyncClient is an instance of MultipartS3AsyncClient">
> [ERROR] Errors:
> [ERROR] ITestUploadRecovery.testCommitOperations:234 » AWSClientIO upload
> part #1 uplo...
> [ERROR] ITestUploadRecovery.testMagicWriteRecovery[array-commit-true] »
> AWSClientIO up...
> [ERROR] ITestUploadRecovery.testMagicWriteRecovery[bytebuffer-commit-false]
> » AWSClientIO
> [ERROR] ITestUploadRecovery.testMagicWriteRecovery[disk-commit-false] »
> AWSClientIO up...
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
