[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295117#comment-15295117 ] Chris Nauroth commented on HADOOP-13130: Excellent, thank you for this patch Steve. > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 2.8.0 > > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-016.patch, > HADOOP-13130-branch-2-006.patch, HADOOP-13130-branch-2-007.patch, > HADOOP-13130-branch-2-008.patch, HADOOP-13130-branch-2-009.patch, > HADOOP-13130-branch-2-010.patch, HADOOP-13130-branch-2-011.patch, > HADOOP-13130-branch-2-012.patch, HADOOP-13130-branch-2-013.patch, > HADOOP-13130-branch-2-014.patch, HADOOP-13130-branch-2-015.patch, > HADOOP-13130-branch-2-016.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295092#comment-15295092 ] Hudson commented on HADOOP-13130: - SUCCESS: Integrated in Hadoop-trunk-Commit #9836 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9836/]) HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve (stevel: rev 39ec1515a205952eda7e171408a8b83eceb4abde) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/InvalidRequestException.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathNotFoundException.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSClientIOException.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathPermissionException.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/package-info.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathAccessDeniedException.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AFailureHandling.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSS3IOException.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSServiceIOException.java > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 2.8.0 > > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-016.patch, > HADOOP-13130-branch-2-006.patch, HADOOP-13130-branch-2-007.patch, > HADOOP-13130-branch-2-008.patch, HADOOP-13130-branch-2-009.patch, > HADOOP-13130-branch-2-010.patch, HADOOP-13130-branch-2-011.patch, > HADOOP-13130-branch-2-012.patch, HADOOP-13130-branch-2-013.patch, > HADOOP-13130-branch-2-014.patch, HADOOP-13130-branch-2-015.patch, > HADOOP-13130-branch-2-016.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295055#comment-15295055 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 4 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 38s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 31s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 36s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 33s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 29s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 19s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 8s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 35s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 35s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 33s {color} | {color:red} root: The patch generated 2 new + 35 unchanged - 7 fixed = 37 total (was 42) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 25s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 16s {color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s {color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 53m 4s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.metrics2.impl.TestMetricsSystemImpl | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:2c91fd8 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12805465/HADOOP-13130-016.patch | | JIRA Issue | HADOOP-13130 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 7e5b2c534949 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 500e946 | | Default Java | 1.8.0_91 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/9547/artifact/patchprocess/diff-checkstyle-root.txt | | unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/9547/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt | | unit test logs | https://builds.apache.org/job/PreCommit-HADOOP-Build/9547/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt | | Test
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295044#comment-15295044 ] Steve Loughran commented on HADOOP-13130: - Trunk tests against S3 ireland all happy {code} Results : Tests run: 237, Failures: 0, Errors: 0, Skipped: 7 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 17:19 min [INFO] Finished at: 2016-05-21T15:11:39+01:00 [INFO] Final Memory: 21M/457M {code} > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-016.patch, > HADOOP-13130-branch-2-006.patch, HADOOP-13130-branch-2-007.patch, > HADOOP-13130-branch-2-008.patch, HADOOP-13130-branch-2-009.patch, > HADOOP-13130-branch-2-010.patch, HADOOP-13130-branch-2-011.patch, > HADOOP-13130-branch-2-012.patch, HADOOP-13130-branch-2-013.patch, > HADOOP-13130-branch-2-014.patch, HADOOP-13130-branch-2-015.patch, > HADOOP-13130-branch-2-016.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295031#comment-15295031 ] Steve Loughran commented on HADOOP-13130: - OK, applying to 2.8+. The difference in the constructor in trunk means that I'm going to have to do a slightly different patch there, I'll stick it up here and test locally before committing > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch, > HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch, > HADOOP-13130-branch-2-009.patch, HADOOP-13130-branch-2-010.patch, > HADOOP-13130-branch-2-011.patch, HADOOP-13130-branch-2-012.patch, > HADOOP-13130-branch-2-013.patch, HADOOP-13130-branch-2-014.patch, > HADOOP-13130-branch-2-015.patch, HADOOP-13130-branch-2-016.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295025#comment-15295025 ] Steve Loughran commented on HADOOP-13130: - well caught. I'll cut the fail() line from the commit > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch, > HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch, > HADOOP-13130-branch-2-009.patch, HADOOP-13130-branch-2-010.patch, > HADOOP-13130-branch-2-011.patch, HADOOP-13130-branch-2-012.patch, > HADOOP-13130-branch-2-013.patch, HADOOP-13130-branch-2-014.patch, > HADOOP-13130-branch-2-015.patch, HADOOP-13130-branch-2-016.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294041#comment-15294041 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 4 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 48s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 22s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 26s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 15s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 23s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 13s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 30s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 55s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 56s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 56s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 58s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 58s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 28s {color} | {color:red} root: The patch generated 2 new + 36 unchanged - 7 fixed = 38 total (was 43) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 28s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 52s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 56s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 29s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_101. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 23s {color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} |
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293980#comment-15293980 ] Steven Wong commented on HADOOP-13130: -- Commenting on patch 014... {{TestS3AAWSCredentialsProvider.testBadConfiguration}} has: {code} if (!(e.getCause() instanceof ClassNotFoundException)) { LOG.error("Unexpected nested cause: {} in {}", e.getCause(), e, e); fail("no exception"); throw e; } {code} Choose one to keep: either the {{fail}} method call or the {{throw}} statement, because the first hides the second. If keeping the {{fail}} call, its message is inaccurate and should change. > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch, > HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch, > HADOOP-13130-branch-2-009.patch, HADOOP-13130-branch-2-010.patch, > HADOOP-13130-branch-2-011.patch, HADOOP-13130-branch-2-012.patch, > HADOOP-13130-branch-2-013.patch, HADOOP-13130-branch-2-014.patch, > HADOOP-13130-branch-2-015.patch, HADOOP-13130-branch-2-016.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293702#comment-15293702 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 4 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 58s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 58s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 13s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 22s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 28s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 55s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 31s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 12s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 12s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 22s {color} | {color:red} root: The patch generated 3 new + 37 unchanged - 7 fixed = 40 total (was 44) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 26s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 2s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 47s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_101. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 20s {color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 75m 3s
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292847#comment-15292847 ] Chris Nauroth commented on HADOOP-13130: Patch 012 looks good. Thanks for digging out more details about those errors. We'll need a minor rebase now that HADOOP-13183 has been committed, and the Findbugs warning is worth fixing. > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch, > HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch, > HADOOP-13130-branch-2-009.patch, HADOOP-13130-branch-2-010.patch, > HADOOP-13130-branch-2-011.patch, HADOOP-13130-branch-2-012.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292006#comment-15292006 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 3 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 29s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 15s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 14s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 5s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 28s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 6s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 15s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 10s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 10s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 6s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 6s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 21s {color} | {color:green} root: The patch generated 0 new + 36 unchanged - 6 fixed = 36 total (was 42) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 43s {color} | {color:red} hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 45s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 12s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 34s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_101. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 25s {color} | {color:red} The patch generated 1 ASF License warnings. {color}
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292000#comment-15292000 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 3 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 40s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 59s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 36s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 10s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 26s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 7s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 14s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 55s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 41s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 14s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 21s {color} | {color:green} root: The patch generated 0 new + 37 unchanged - 6 fixed = 37 total (was 43) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 43s {color} | {color:red} hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 8s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 31s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_101. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 23s {color} | {color:red} The patch generated 1 ASF License
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291704#comment-15291704 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 7m 33s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 3 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 47s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 6s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 38s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 19s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 27s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 18s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 12s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 25s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 33s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 22s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 22s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 23s {color} | {color:green} root: The patch generated 0 new + 37 unchanged - 6 fixed = 37 total (was 43) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 28s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 31s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 12s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 28s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_101. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s {color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} |
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291553#comment-15291553 ] Chris Nauroth commented on HADOOP-13130: Thanks, Steve. I'll wait for another patch revision. > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch, > HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch, > HADOOP-13130-branch-2-009.patch, HADOOP-13130-branch-2-010.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291534#comment-15291534 ] Steve Loughran commented on HADOOP-13130: - Here are the errors which can come back. I'm thinking we could do more than translate all 400 responses into "bad request", as there is more detail there http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch, > HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch, > HADOOP-13130-branch-2-009.patch, HADOOP-13130-branch-2-010.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291359#comment-15291359 ] Chris Nauroth commented on HADOOP-13130: Thanks, Steve. Patch 009 looks good. I don't have any further comments. I'm putting it through one more full test run before I sign off. I tried combing through S3 docs to get an explanation of that 400 response, but I didn't have any luck. It seems like it ought to be a 403, although there is some wiggle room in the phrasing of the docs. Note use of the word "might" here: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEAD.html bq. Otherwise, the operation might return responses such as 404 Not Found and 403 Forbidden. > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch, > HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch, > HADOOP-13130-branch-2-009.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291318#comment-15291318 ] Steve Loughran commented on HADOOP-13130: - Bad request here. The bucket exists, but isn't giving me access. This is frankfurt {code} testDeleteEmptyDirNonRecursive(org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete) Time elapsed: 0.208 sec <<< ERROR! org.apache.hadoop.fs.InvalidRequestException: doesBucketExist on stevel-frankfurt-3: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 49355E46D8DFCA6B), S3 Extended Request ID: KRFtrdEbdZdA4Z6ve2exgBmQArLniiq85f/yUf0NC+btW58ExNxoo3Omhe5Cup0QE7ub5lTes5U= at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:96) at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:293) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:272) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2786) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2823) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2805) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:382) at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:165) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) {code} > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch, > HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch, > HADOOP-13130-branch-2-009.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291311#comment-15291311 ] Steve Loughran commented on HADOOP-13130: - Note that with this setup, asking for a bucket which isn't there returns an FNFE which lists the bucket name {code} testDeleteEmptyDirNonRecursive(org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete) Time elapsed: 0.549 sec <<< ERROR! java.io.FileNotFoundException: Bucket no-bucket-here-at-all does not exist at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:290) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:272) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2786) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2823) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2805) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:382) at org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:165) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) {code} Asking for a bucket which exists but which you lack the rights to appears to trigger BadRequestExceptions in circumstances I don't fully understand > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch, > HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch, > HADOOP-13130-branch-2-009.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290772#comment-15290772 ] Steve Loughran commented on HADOOP-13130: - got it —and yes, it makes sense. I'll factor the method out. > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch, > HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290592#comment-15290592 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 10m 4s {color} | {color:red} Docker failed to build yetus/hadoop:babe025. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12804745/HADOOP-13130-branch-2-008.patch | | JIRA Issue | HADOOP-13130 | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/9503/console | | Powered by | Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch, > HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290283#comment-15290283 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 3 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 34s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 23s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 54s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 44s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 25s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 59s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 16s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 48s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 30s {color} | {color:green} root: The patch generated 0 new + 36 unchanged - 7 fixed = 36 total (was 43) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 28s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 46s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 0s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color}
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289883#comment-15289883 ] Chris Nauroth commented on HADOOP-13130: Thank you for patch 008. This is looking good. I completed a clean parallel test run in ~26 minutes against S3 buckets in US-west-2. Thank you for the clarification on {{eventually}} and {{Callable}}. That makes sense. There is one thing left unaddressed from my last comment. There was an EOF assertion that used a descriptive message, and I was wondering if you wanted to do the same for other EOF assertions. I'm not sure if this slipped through the cracks in 008, or if it was a conscious choice not to change it. To make sure it's clear, here is the descriptive assertion: {code} assertEquals("Expected EOF got char " + (char) c, -1, c); {code} ...and here are the other ones that might benefit from the same message... {code} assertEquals(-1, instream.read(buf)); {code} {code} assertEquals(-1, instream.read(instream.getPos(), buf, 0, buf.length)); {code} {code} assertEquals(-1, instream.read(shortLen + 510, buf, 0, buf.length)); {code} > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch, > HADOOP-13130-branch-2-007.patch, HADOOP-13130-branch-2-008.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289130#comment-15289130 ] Steve Loughran commented on HADOOP-13130: - h3. guarded log debug statements I'd left them all alone, just trying to keep the diff down, but may as well cull them altogether. h3. append() throws UnsupportedOperationException I went through and had a look, and no, most implementations just throw an IOE with random messages. I think we could tighten the FS API there. Created HADOOP-13172 to cover. h3. {{String header = operation}} Moved to {{String.format}} for clarity, fixed. h3. {{this exception is non-null if the service exception is an s3 on}} s/on/r/one/ h3. {{eventually}}. {{Callable}} can throw exceptions, {{Runnable can't}}. I also actually thought of playing template games and extracting the result, but gave up as it was hard. That could be a future improvement which the current code sets up. > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289129#comment-15289129 ] Steve Loughran commented on HADOOP-13130: - h3. guarded log debug statements I'd left them all alone, just trying to keep the diff down, but may as well cull them altogether. h3. append() throws UnsupportedOperationException I went through and had a look, and no, most implementations just throw an IOE with random messages. I think we could tighten the FS API there. Created HADOOP-13172 to cover. h3. {{String header = operation}} Moved to {{String.format}} for clarity, fixed. h3. {{this exception is non-null if the service exception is an s3 on}} s/on/r/one/ h3. {{eventually}}. {{Callable}} can throw exceptions, {{Runnable can't}}. I also actually thought of playing template games and extracting the result, but gave up as it was hard. That could be a future improvement which the current code sets up. > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288966#comment-15288966 ] Steve Loughran commented on HADOOP-13130: - yes, seen them too. Didn't test enough. More specifically, I'd seen the tests failing but thought it was related to something else I'd done. thank you for the stack traces > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch, HADOOP-13130-002.patch, > HADOOP-13130-002.patch, HADOOP-13130-003.patch, HADOOP-13130-004.patch, > HADOOP-13130-005.patch, HADOOP-13130-branch-2-006.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287946#comment-15287946 ] Chris Nauroth commented on HADOOP-13130: [~ste...@apache.org], thank you for the patch. Here are a few comments. {code} if (LOG.isDebugEnabled()) { LOG.debug("Completing multi-part upload for key '{}', id '{}'", key, uploadId); } {code} The log level guard is unnecessary. {code} public FSDataOutputStream append(Path f, int bufferSize, Progressable progress) throws IOException { -throw new IOException("Not supported"); +throw new UnsupportedOperationException("Not supported"); } {code} Possibly backwards-incompatible? Maybe someone coded error handling that catches {{IOException}} and falls back to a non-append strategy for non-HDFS? {code} String header = operation + (path != null ? ("on " + path) : "") + ": "; String message = header + exception; {code} The message will have no space between {{operation}} and {{"on "}}. {code} // this exception is non-null if the service exception is an s3 on {code} Typo at end of comment? {code} public static void eventually(int timeout, Callable callback) throws Exception { Exception lastException; long endtime = System.currentTimeMillis() + timeout; do { try { callback.call(); return; } catch (FailFastException e) { throw e; } catch (Exception e) { lastException = e; } Thread.sleep(500); } while (endtime > System.currentTimeMillis()); throw lastException; } {code} {{eventually}} doesn't appear to be interested in the results returned from the callback, so would {{Runnable}} be a better fit than {{Callable}}? {code} assertEquals("Expected EOF got char " + (char) c, -1, c); byte[] buf = new byte[256]; assertEquals(-1, instream.read(buf)); assertEquals(-1, instream.read(instream.getPos(), buf, 0, buf.length)); // now do a block read fully, again, backwards from the current pos try { instream.readFully(shortLen + 512, buf); fail("Expected readFully to fail"); } catch (EOFException expected) { LOG.debug("Expected: ", expected); } assertEquals(-1, instream.read(shortLen + 510, buf, 0, buf.length)); {code} Do you want to use the descriptive "Expected EOF" message on all of these EOF assertions? See below for several test failures I got after applying the patch to branch-2. I see these failures consistently, running both with and without the parallel-tests profile. If these failures don't repro for you, let me know, and I'll dig deeper on my side. {code} testProxyConnection(org.apache.hadoop.fs.s3a.TestS3AConfiguration) Time elapsed: 1.635 sec <<< ERROR! java.io.IOException: doesBucketExiston cnauroth-test-aws-s3a: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection to http://127.0.0.1:1 refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:728) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785) at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1107) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1070) at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:289) at
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286864#comment-15286864 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 17s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 34s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 48s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 38s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 29s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 35s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 22s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 15m 13s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 23s {color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s {color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s {color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 56s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 52s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 52s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 37s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 37s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 20s {color} | {color:red} root: The patch generated 1 new + 26 unchanged - 6 fixed = 27 total (was 32) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 40s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 9s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 6s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} |
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285182#comment-15285182 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 49s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 11s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 41s {color} | {color:green} trunk passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 44s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 26s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 32s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s {color} | {color:green} trunk passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 43s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 43s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 40s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 40s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 27s {color} | {color:green} root: The patch generated 0 new + 24 unchanged - 5 fixed = 24 total (was 29) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 57s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 44s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 70m 35s {color} | {color:black} {color} | \\ \\ ||
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284733#comment-15284733 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 34s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 5s {color} | {color:green} trunk passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 28s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s {color} | {color:green} trunk passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 56s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 1s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 1s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 45s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 45s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 25s {color} | {color:red} root: The patch generated 1 new + 24 unchanged - 6 fixed = 25 total (was 30) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 8s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 11s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 68m 5s {color} | {color:black} {color} | \\ \\
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284407#comment-15284407 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m 37s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 36s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 22s {color} | {color:green} trunk passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 47s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s {color} | {color:green} trunk passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 56s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 3s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 3s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 46s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 46s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 25s {color} | {color:red} root: The patch generated 1 new + 24 unchanged - 6 fixed = 25 total (was 30) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 7s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 16s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 73m 53s {color} | {color:black} {color} | \\
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15283892#comment-15283892 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 11m 3s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 34s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 3s {color} | {color:green} trunk passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 24s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s {color} | {color:green} trunk passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 1s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 1s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 47s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 47s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 27s {color} | {color:red} root: The patch generated 3 new + 29 unchanged - 1 fixed = 32 total (was 30) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 27s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 31s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s {color} | {color:red} hadoop-tools_hadoop-aws-jdk1.8.0_91 with JDK v1.8.0_91 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 12s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 15s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black}
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15283874#comment-15283874 ] Steve Loughran commented on HADOOP-13130: - comment/jIRA liveness probe tests > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15283094#comment-15283094 ] Hadoop QA commented on HADOOP-13130: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 53s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 1s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 10s {color} | {color:green} trunk passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 51s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 51s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 41s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 38s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 54s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 40s {color} | {color:green} trunk passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 53s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 32s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 32s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 49s {color} | {color:red} root: The patch generated 31 new + 29 unchanged - 0 fixed = 60 total (was 29) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 6s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 16s {color} | {color:red} hadoop-tools_hadoop-aws-jdk1.8.0_91 with JDK v1.8.0_91 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 35s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 26s {color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 57s {color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s {color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} |
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282954#comment-15282954 ] Steve Loughran commented on HADOOP-13130: - I forgot to mention: patch 001 does make a couple of changes where handling of the file-open-in-lazy seek logic was raising EOF exceptions in non-positioned read() calls, calls where a -1 was expected to be raised instead. > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13130-001.patch > > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282766#comment-15282766 ] Steve Loughran commented on HADOOP-13130: - One aspect of this is that I've managed to create a test which triggers an EOF exception in read()... the {{lazySeek()}} logic is outside the try/catch clause here. If someone overwrites a file with a shorter one, and then you seek to somewhere which you think is in range —but which isn't, a 419 comes back, which I'm not mapping to EOF. Fix: move the {{lazySeek()}} > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281891#comment-15281891 ] Chris Nauroth commented on HADOOP-13130: +1 for the proposal. > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281565#comment-15281565 ] Steve Loughran commented on HADOOP-13130: - Here's what I'm thinking # All methods which work with the AWS libs are wrapped by something that catches all the Amazon exceptions # we have some well defined translations for some error codes (400 -> {{org.apache.hadoop.fs.InvalidRequestException}}, 401 -> {{PathAccessDeniedException}} # we have a subclass of {{PathIOException}}, {{PathHttpIOException}} which adds a status code field. this goes {{org.apache.hadoop.fs}} for use elsewhere (or we keep in s3a and maybe uprate later?) # amazon exceptions are caught and translated; URL of endpoint, verb and status code are all included in the exception. This would be very similar to the {{org.apache.hadoop.fs.swift.exceptions.SwiftInvalidResponseException}} exceptions > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281559#comment-15281559 ] Steve Loughran commented on HADOOP-13130: - Another example, forbidden access to a bucket. {code} testEncrypt256(org.apache.hadoop.fs.s3a.TestS3AEncryptionAlgorithmPropagation) Time elapsed: 0.843 sec <<< ERROR! com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 409F9DDD8763ADE7) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1050) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1027) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1025) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:82) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1432) at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:514) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:921) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:902) at org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset(ContractTestUtils.java:141) at org.apache.hadoop.fs.s3a.TestS3AEncryptionAlgorithmPropagation.writeThenReadFile(TestS3AEncryptionAlgorithmPropagation.java:87) at org.apache.hadoop.fs.s3a.TestS3AEncryptionAlgorithmPropagation.testEncrypt256(TestS3AEncryptionAlgorithmPropagation.java:69) {code} > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13130) s3a failures can surface as RTEs, not IOEs
[ https://issues.apache.org/jira/browse/HADOOP-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279931#comment-15279931 ] Steve Loughran commented on HADOOP-13130: - Here's an interesting example. A mkdir() operation is failing because the caller is (deliberately) requesting an unsupported encryption. algorithm. {code} testEncrypt256(org.apache.hadoop.fs.s3a.TestS3AEncryptionAlgorithmPropagation) Time elapsed: 3.555 sec <<< ERROR! com.amazonaws.services.s3.model.AmazonS3Exception: The encryption method specified is not supported (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: A7FEE89E7EB4FC6D) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1472) at org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:1307) at org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:1284) at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:981) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1894) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:323) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:181) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) {code} > s3a failures can surface as RTEs, not IOEs > -- > > Key: HADOOP-13130 > URL: https://issues.apache.org/jira/browse/HADOOP-13130 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran > > S3A failures happening in the AWS library surface as > {{AmazonClientException}} derivatives, rather than IOEs. As the amazon > exceptions are runtime exceptions, any code which catches IOEs for error > handling breaks. > The fix will be to catch and wrap. The hard thing will be to wrap it with > meaningful exceptions rather than a generic IOE. Furthermore, if anyone has > been catching AWS exceptions, they are going to be disappointed. That means > that fixing this situation could be considered "incompatible" —but only for > code which contains assumptions about the underlying FS and the exceptions > they raise. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org