[
https://issues.apache.org/jira/browse/HADOOP-13231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312036#comment-15312036
]
Steve Loughran edited comment on HADOOP-13231 at 6/2/16 9:46 AM:
-----------------------------------------------------------------
Chris, here are the errors I saw on a parallel HADOOP-13171 run
{code}
Running org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 25.167 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool
testRegularMultiPartUpload(org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool)
Time elapsed: 13.218 sec <<< ERROR!
java.io.FileNotFoundException: saving output on
tests3a/97e97bdb-5ca4-46b8-a3d2-7dbb2100101b:
com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does
not exist. The upload ID may be invalid, or the upload may have been aborted or
completed. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchUpload;
Request ID: 0A0C0CA457261DB8), S3 Extended Request ID:
kLavwGlMzkH7dm1uwbCQ1N6fK8HfUOi/PkNf4lSE7rXRQVmLkkqE6Nd86n2wu8UOZy8jdXK7+TY=
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:106)
at
org.apache.hadoop.fs.s3a.S3AOutputStream.close(S3AOutputStream.java:123)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at
org.apache.hadoop.fs.contract.ContractTestUtils.generateTestFile(ContractTestUtils.java:864)
at
org.apache.hadoop.fs.contract.ContractTestUtils.createAndVerifyFile(ContractTestUtils.java:892)
at
org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool.testRegularMultiPartUpload(TestS3ABlockingThreadPool.java:68)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified
upload does not exist. The upload ID may be invalid, or the upload may have
been aborted or completed. (Service: Amazon S3; Status Code: 404; Error Code:
NoSuchUpload; Request ID: 0A0C0CA457261DB8)
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at
com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:2705)
at
com.amazonaws.services.s3.transfer.internal.CompleteMultipartUpload.call(CompleteMultipartUpload.java:91)
at
com.amazonaws.services.s3.transfer.internal.CompleteMultipartUpload.call(CompleteMultipartUpload.java:38)
at
org.apache.hadoop.fs.s3a.BlockingThreadPoolExecutorService$CallableWithPermitRelease.call(BlockingThreadPoolExecutorService.java:239)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
testFastMultiPartUpload(org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool)
Time elapsed: 11.866 sec <<< ERROR!
java.io.FileNotFoundException: Completing multi-part upload on
tests3a/77dd6617-f0ac-45d4-bd3f-3475c07bb002:
com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does
not exist. The upload ID may be invalid, or the upload may have been aborted or
completed. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchUpload;
Request ID: C8FD35777E51FF7C), S3 Extended Request ID:
MZ8VyewWz4/qUu5h/MTDv+F7q9co2fhUkOCLxSt9Z5mXkO8vimUGQW9Fizvm0hkG82IPuvaiWaI=
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at
com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:2705)
at
org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload.complete(S3AFastOutputStream.java:374)
at
org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload.access$200(S3AFastOutputStream.java:319)
at
org.apache.hadoop.fs.s3a.S3AFastOutputStream.close(S3AFastOutputStream.java:255)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at
org.apache.hadoop.fs.contract.ContractTestUtils.generateTestFile(ContractTestUtils.java:864)
at
org.apache.hadoop.fs.contract.ContractTestUtils.createAndVerifyFile(ContractTestUtils.java:892)
at
org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool.testFastMultiPartUpload(TestS3ABlockingThreadPool.java:76)
Running org.apache.hadoop.fs.s3a.TestS3ABlocksize
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.275 sec - in
org.apache.hadoop.fs.s3a
{code}
was (Author: [email protected]):
Chris, here are the errors I saw on a parallel HADOOP-13171 run
{code}
Running org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 25.167 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool
testRegularMultiPartUpload(org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool)
Time elapsed: 13.218 sec <<< ERROR!
java.io.FileNotFoundException: saving output on
tests3a/97e97bdb-5ca4-46b8-a3d2-7dbb2100101b:
com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does
not exist. The upload ID may be invalid, or the upload may have been aborted or
completed. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchUpload;
Request ID: 0A0C0CA457261DB8), S3 Extended Request ID:
kLavwGlMzkH7dm1uwbCQ1N6fK8HfUOi/PkNf4lSE7rXRQVmLkkqE6Nd86n2wu8UOZy8jdXK7+TY=
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:106)
at
org.apache.hadoop.fs.s3a.S3AOutputStream.close(S3AOutputStream.java:123)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at
org.apache.hadoop.fs.contract.ContractTestUtils.generateTestFile(ContractTestUtils.java:864)
at
org.apache.hadoop.fs.contract.ContractTestUtils.createAndVerifyFile(ContractTestUtils.java:892)
at
org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool.testRegularMultiPartUpload(TestS3ABlockingThreadPool.java:68)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified
upload does not exist. The upload ID may be invalid, or the upload may have
been aborted or completed. (Service: Amazon S3; Status Code: 404; Error Code:
NoSuchUpload; Request ID: 0A0C0CA457261DB8)
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at
com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:2705)
at
com.amazonaws.services.s3.transfer.internal.CompleteMultipartUpload.call(CompleteMultipartUpload.java:91)
at
com.amazonaws.services.s3.transfer.internal.CompleteMultipartUpload.call(CompleteMultipartUpload.java:38)
at
org.apache.hadoop.fs.s3a.BlockingThreadPoolExecutorService$CallableWithPermitRelease.call(BlockingThreadPoolExecutorService.java:239)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
testFastMultiPartUpload(org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool)
Time elapsed: 11.866 sec <<< ERROR!
java.io.FileNotFoundException: Completing multi-part upload on
tests3a/77dd6617-f0ac-45d4-bd3f-3475c07bb002:
com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does
not exist. The upload ID may be invalid, or the upload may have been aborted or
completed. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchUpload;
Request ID: C8FD35777E51FF7C), S3 Extended Request ID:
MZ8VyewWz4/qUu5h/MTDv+F7q9co2fhUkOCLxSt9Z5mXkO8vimUGQW9Fizvm0hkG82IPuvaiWaI=
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at
com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:2705)
at
org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload.complete(S3AFastOutputStream.java:374)
at
org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload.access$200(S3AFastOutputStream.java:319)
at
org.apache.hadoop.fs.s3a.S3AFastOutputStream.close(S3AFastOutputStream.java:255)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at
org.apache.hadoop.fs.contract.ContractTestUtils.generateTestFile(ContractTestUtils.java:864)
at
org.apache.hadoop.fs.contract.ContractTestUtils.createAndVerifyFile(ContractTestUtils.java:892)
at
org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool.testFastMultiPartUpload(TestS3ABlockingThreadPool.java:76)
Running org.apache.hadoop.fs.s3a.TestS3ABlocksize
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.275 sec - in
org.apache.hadoop.fs.s3a
{cpde}
> Isolate test path used by a few S3A tests for more reliable parallel
> execution.
> -------------------------------------------------------------------------------
>
> Key: HADOOP-13231
> URL: https://issues.apache.org/jira/browse/HADOOP-13231
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Priority: Minor
>
> I have noticed a few more spots in S3A tests that do not make use of the
> isolated test directory path when running in parallel mode. While I don't
> have any evidence that this is really causing problems for parallel test runs
> right now, it would still be good practice to clean these up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
