[
https://issues.apache.org/jira/browse/HADOOP-16767?focusedWorklogId=549397&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-549397
]
ASF GitHub Bot logged work on HADOOP-16767:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 08/Feb/21 05:30
Start Date: 08/Feb/21 05:30
Worklog Time Spent: 10m
Work Description: yzhangal commented on pull request #2685:
URL: https://github.com/apache/hadoop/pull/2685#issuecomment-774878069
Hi @steveloughran , sorry I'm behind getting HADOOP-17338 to 2.10 branch. I
have a version ready, but before that, I'd like to get HADOOP-16767 to 2.10.
Here are the test results I have with both together (largely some timeout
failures). Would you please help getting this one in and I will put
HADOOP-17338 on top of it.
Thanks a lot.
[WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
64.07 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
171.593 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[INFO] Tests run: 48, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
168.231 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
[ERROR] Tests run: 9, Failures: 0, Errors: 3, Skipped: 0, Time elapsed:
558.659 s <<< FAILURE! - in
org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
76.078 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
457.171 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
110.398 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
65.483 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
27.233 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
6.871 s - in org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.107 s - in org.apache.hadoop.fs.s3a.impl.TestDirectoryMarkerPolicy
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
20.985 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
[WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
242.879 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.15
s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
25.962 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
87.202 s - in
org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
[WARNING] Tests run: 63, Failures: 0, Errors: 0, Skipped: 3, Time elapsed:
643.64 s - in
org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.531
s - in org.apache.hadoop.fs.s3a.TestListing
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.121 s - in org.apache.hadoop.fs.s3a.TestS3AExceptionTranslation
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
4.151 s - in org.apache.hadoop.fs.s3a.ITestS3GuardCreate
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.917 s - in org.apache.hadoop.fs.s3a.TestSSEConfiguration
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
39.734 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSES3
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.955
s - in org.apache.hadoop.fs.s3a.ITestS3ABlocksize
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.49
s - in org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed:
15.067 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
4.267 s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed:
14.164 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
7.357 s - in org.apache.hadoop.fs.s3a.scale.ITestDynamoDBMetadataStoreScale
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed:
14.207 s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
4.388 s - in org.apache.hadoop.fs.s3a.scale.ITestS3ACreatePerformance
[WARNING] Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed:
22.418 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
4.524 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed:
14.804 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesSSECDiskBlocks
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
4.647 s - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
6.706 s - in org.apache.hadoop.fs.s3a.scale.ITestLocalMetadataStoreScale
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed:
14.39 s - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.855
s - in org.apache.hadoop.fs.s3a.ITestS3ADelayedFNF
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
42.084 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSDefaultKey
[INFO] Tests run: 30, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
11.345 s - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.855
s - in org.apache.hadoop.fs.s3a.ITestS3AInconsistency
[ERROR] Tests run: 10, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
103.302 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEC
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
31.618 s - in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.258
s - in org.apache.hadoop.fs.s3a.yarn.ITestS3A
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.81
s - in org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
4.178 s - in org.apache.hadoop.fs.s3a.ITestS3GuardWriteBack
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
23.037 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
43.654 s - in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
[WARNING] Tests run: 9, Failures: 0, Errors: 0, Skipped: 8, Time elapsed:
49.644 s - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
0.189 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionSSEKMSUserDefinedKey
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
21.4 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
21.999 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.596
s - in org.apache.hadoop.fs.s3a.TestS3AGetFileStatus
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
15.316 s - in org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
[INFO] Tests run: 31, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
221.542 s - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.298
s - in org.apache.hadoop.fs.s3a.s3guard.TestPathMetadataDynamoDBTranslation
[WARNING] Tests run: 21, Failures: 0, Errors: 0, Skipped: 21, Time elapsed:
53.878 s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
[WARNING] Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed:
28.464 s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
4.58 s - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.127 s - in org.apache.hadoop.fs.s3a.s3guard.TestS3GuardCLI
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.1
s - in org.apache.hadoop.fs.s3a.s3guard.TestDirListingMetadata
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.084
s - in org.apache.hadoop.fs.s3a.s3guard.TestS3Guard
[INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.595 s - in org.apache.hadoop.fs.s3a.s3guard.TestNullMetadataStore
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.544 s - in org.apache.hadoop.fs.s3a.s3guard.TestLocalMetadataStore
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.103
s - in org.apache.hadoop.fs.s3a.ITestS3ATestUtils
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.084
s - in org.apache.hadoop.fs.s3a.TestDataBlocks
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
4.197 s - in org.apache.hadoop.fs.s3a.ITestS3GuardEmptyDirs
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.287
s - in org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.229
s - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.091 s - in org.apache.hadoop.fs.s3a.TestS3AInputPolicies
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
150.313 s - in org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
16.538 s - in org.apache.hadoop.fs.s3a.ITestS3AEmptyDirectory
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
0.017 s - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation
[ERROR] Tests run: 603, Failures: 3, Errors: 3, Skipped: 95
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 549397)
Time Spent: 20m (was: 10m)
> S3AInputStream reopening does not handle non IO exceptions properly
> -------------------------------------------------------------------
>
> Key: HADOOP-16767
> URL: https://issues.apache.org/jira/browse/HADOOP-16767
> Project: Hadoop Common
> Issue Type: Sub-task
> Affects Versions: 3.1.1
> Reporter: Sergei Poganshev
> Assignee: Sergei Poganshev
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since only {{IOException}} is getting caught in
> [closeStream|https://github.com/apache/hadoop/blob/24080666e5e2214d4a362c889cd9aa617be5de81/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L579],
> {{reopen()}} fails in case {{SdkClientException}} occurs on a drain attempt.
> This leads to multiple failing retries of {{invoker.retry("read",...)}} and
> then the whole {{read()}} operation failing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
