[jira] [Commented] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[ https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380306#comment-16380306 ] Hudson commented on HADOOP-15264: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13735 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13735/]) HADOOP-15264. AWS "shaded" SDK 1.271 is pulling in netty 4.1.17. (stevel: rev e015e009897e481edc79f4ba72e2c53610b178a3) * (edit) hadoop-project/pom.xml > AWS "shaded" SDK 1.271 is pulling in netty 4.1.17 > - > > Key: HADOOP-15264 > URL: https://issues.apache.org/jira/browse/HADOOP-15264 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Fix For: 3.1.0 > > Attachments: HADOOP-15264-001.patch > > > The latest versions of the AWS Shaded SDK are declaring a dependency on netty > 4.1.17 > {code} > [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile > [INFO] | \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile > [INFO] | +- io.netty:netty-codec-http:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-codec:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-handler:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-buffer:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-common:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-transport:jar:4.1.17.Final:compile > [INFO] | \- io.netty:netty-resolver:jar:4.1.17.Final:compile > {code} > We either exclude these or roll back HADOOP-15040. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[ https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379319#comment-16379319 ] Aaron Fabbri commented on HADOOP-15264: --- +1 LGTM. No testing but I pulled trunk of the aws-java-sdk source and searched for netty. Looks like it is only used for {{kenesisvideo}}: {noformat} $ find . -name \*\.java -exec grep -Hni netty "{}" \; | grep -v kinesisvideo | wc -l 0 {noformat} > AWS "shaded" SDK 1.271 is pulling in netty 4.1.17 > - > > Key: HADOOP-15264 > URL: https://issues.apache.org/jira/browse/HADOOP-15264 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15264-001.patch > > > The latest versions of the AWS Shaded SDK are declaring a dependency on netty > 4.1.17 > {code} > [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile > [INFO] | \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile > [INFO] | +- io.netty:netty-codec-http:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-codec:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-handler:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-buffer:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-common:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-transport:jar:4.1.17.Final:compile > [INFO] | \- io.netty:netty-resolver:jar:4.1.17.Final:compile > {code} > We either exclude these or roll back HADOOP-15040. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[ https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379304#comment-16379304 ] Wangda Tan commented on HADOOP-15264: - Thanks [~ste...@apache.org], the patch LGTM. Do you want to make any more changes / verification within this JIRA's scope? > AWS "shaded" SDK 1.271 is pulling in netty 4.1.17 > - > > Key: HADOOP-15264 > URL: https://issues.apache.org/jira/browse/HADOOP-15264 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15264-001.patch > > > The latest versions of the AWS Shaded SDK are declaring a dependency on netty > 4.1.17 > {code} > [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile > [INFO] | \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile > [INFO] | +- io.netty:netty-codec-http:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-codec:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-handler:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-buffer:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-common:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-transport:jar:4.1.17.Final:compile > [INFO] | \- io.netty:netty-resolver:jar:4.1.17.Final:compile > {code} > We either exclude these or roll back HADOOP-15040. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[ https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378809#comment-16378809 ] genericqa commented on HADOOP-15264: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 29s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 24m 14s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 47s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 9s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 36m 25s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HADOOP-15264 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12912085/HADOOP-15264-001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml | | uname | Linux 10ba4f2dd454 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 1e85a99 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14224/testReport/ | | Max. process+thread count | 408 (vs. ulimit of 1) | | modules | C: hadoop-project U: hadoop-project | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14224/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > AWS "shaded" SDK 1.271 is pulling in netty 4.1.17 > - > > Key: HADOOP-15264 > URL: https://issues.apache.org/jira/browse/HADOOP-15264 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15264-001.patch >
[jira] [Commented] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[ https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378749#comment-16378749 ] Steve Loughran commented on HADOOP-15264: - Think we need some explicit policy w.r.t JAR updates in our s3a testing docs Something like * JAR updates potentially brittle, esp the 1.XX to 1.YY. Even the minor ones have surprises (+point to links related to abort(), NPEs) * regression test everything with/without s3guard, scale including assumed roles * + against > 1 endpoint, especially a v4 * Maybe even try v2 signing * Seek tests from anyone with internal endpoints. e.g [~ehiggs] * Don't panic about new deprecation warnings, be selective about whether to upgrade, as its not always clear the new versions are better in terms of maintenance * do a mvn dep of the hadoop-aws and hadoop-cloudstore modules, review, reject any new additions * any downstream tests you can pull off too, including my little [cloudstore diags|https://github.com/steveloughran/cloudstore] * *review the logs*. Those extra warnings and stack traces didn't cause problems in the tests, but showed up when we looked at things > AWS "shaded" SDK 1.271 is pulling in netty 4.1.17 > - > > Key: HADOOP-15264 > URL: https://issues.apache.org/jira/browse/HADOOP-15264 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15264-001.patch > > > The latest versions of the AWS Shaded SDK are declaring a dependency on netty > 4.1.17 > {code} > [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile > [INFO] | \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile > [INFO] | +- io.netty:netty-codec-http:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-codec:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-handler:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-buffer:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-common:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-transport:jar:4.1.17.Final:compile > [INFO] | \- io.netty:netty-resolver:jar:4.1.17.Final:compile > {code} > We either exclude these or roll back HADOOP-15040. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[ https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377371#comment-16377371 ] Steve Loughran commented on HADOOP-15264: - I've got a patch which excludes the JARs; they're only needed I'd rather do that than upgrade the AWS SDK to a new one, in case there are other surprises. We've been using the 1.11.271 for a few weeks, no NPE stack traces, no complaints that we are closing streams in an abort() call, etc. Happy. Patch attached, tested against AWS S3 London; one failure in ITestS3GuardToolLocal; trying s3 ireland ann everything is rejected at 400. I think something is up with my S3 binding today, as I've been seeing failures elsewhere. Assume unrelated. > AWS "shaded" SDK 1.271 is pulling in netty 4.1.17 > - > > Key: HADOOP-15264 > URL: https://issues.apache.org/jira/browse/HADOOP-15264 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15264-001.patch > > > The latest versions of the AWS Shaded SDK are declaring a dependency on netty > 4.1.17 > {code} > [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile > [INFO] | \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile > [INFO] | +- io.netty:netty-codec-http:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-codec:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-handler:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-buffer:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-common:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-transport:jar:4.1.17.Final:compile > [INFO] | \- io.netty:netty-resolver:jar:4.1.17.Final:compile > {code} > We either exclude these or roll back HADOOP-15040. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[ https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377331#comment-16377331 ] Wangda Tan commented on HADOOP-15264: - [~ste...@apache.org], from the latest comment on https://github.com/aws/aws-sdk-java/issues/1488, Andrew Shore mentioned he will add the mapping. So is there anything we need to do from our side? Should we still mark this to be blocker of 3.1.0? > AWS "shaded" SDK 1.271 is pulling in netty 4.1.17 > - > > Key: HADOOP-15264 > URL: https://issues.apache.org/jira/browse/HADOOP-15264 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Priority: Blocker > > The latest versions of the AWS Shaded SDK are declaring a dependency on netty > 4.1.17 > {code} > [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile > [INFO] | \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile > [INFO] | +- io.netty:netty-codec-http:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-codec:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-handler:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-buffer:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-common:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-transport:jar:4.1.17.Final:compile > [INFO] | \- io.netty:netty-resolver:jar:4.1.17.Final:compile > {code} > We either exclude these or roll back HADOOP-15040. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17
[ https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377262#comment-16377262 ] Steve Loughran commented on HADOOP-15264: - filed AWS SDK issue [1488|https://github.com/aws/aws-sdk-java/issues/1488] > AWS "shaded" SDK 1.271 is pulling in netty 4.1.17 > - > > Key: HADOOP-15264 > URL: https://issues.apache.org/jira/browse/HADOOP-15264 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Priority: Blocker > > The latest versions of the AWS Shaded SDK are declaring a dependency on netty > 4.1.17 > {code} > [INFO] +- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile > [INFO] | \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile > [INFO] | +- io.netty:netty-codec-http:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-codec:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-handler:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-buffer:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-common:jar:4.1.17.Final:compile > [INFO] | +- io.netty:netty-transport:jar:4.1.17.Final:compile > [INFO] | \- io.netty:netty-resolver:jar:4.1.17.Final:compile > {code} > We either exclude these or roll back HADOOP-15040. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org