Re: Set minimum version of Hadoop 3 to JDK 8
You're absolutely right Steve. We should get the jobs under control. At the very least, before we move to JDK 8 in Jenkins, we should get the Java8 nightly builds working. I didn't realize we had these, and I had thought all compiling and most test issues with JDK8 were already fixed based on what HADOOP-11090 looks like; but I guess there's more. https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-common-trunk-Java8/ https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/ https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Java8/ https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk-Java8/ I'll take a deeper look at these and file JIRAs as necessary later today. On Wed, Apr 22, 2015 at 3:13 AM, Steve Loughran ste...@hortonworks.com wrote: On 21 Apr 2015, at 23:31, Andrew Wang andrew.w...@cloudera.com wrote: Hey Robert, As a first step, could we try switching all our precommit and nightly builds over to use 1.8? This is a prerequisite for HADOOP-11858, and safe to do in any case since it'll still target 1.7. +1 we have jenkin builds running on Java 8 -its just they have enough longstanding and intermittent test failures that they get ignored I've been trying to keep hadoop-common somewhat under control, but -YARN is failing due to to hard coded ports in tests - this needs to be fixed urgently: YARN-3528 -HDFS build process itself is failing with some config problem checkstyle I'll note that HADOOP-10530 details the pain Steve went through switching us to JDK7. Might be some lessons learned about how to do this transition more smoothly. The key thing is to have Jenkins happy before trying to change anything. And make sure that happy means that jenkins really is building and testing against the version of Java that you think it is. I spent about 15 minutes/hour of my weekend kicking off builds and tuning them. -I volunteered Colin Patrick McCabe to to that bit of the switch, though of course he doesn't have to accept that opportunuty ( https://issues.apache.org/jira/browse/HADOOP-10530?focusedCommentId=14239250page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14239250 Someone needs to take ownership of the Jenkins builds are broken problem -ideally someone in each subproject. That's a prerequisite to any major changes in the build process, but should also be best practice anyway. Ask your QE teams would you like the ASF projects to care whether their jenkins test runs succeed and see what they say. as a reminder, all the jenkins builds are here https://builds.apache.org/view/H-L/view/Hadoop/ please look at them and try to fix them —without even waiting for a switch to Java 8
Re: Set minimum version of Hadoop 3 to JDK 8
On 21 Apr 2015, at 23:31, Andrew Wang andrew.w...@cloudera.com wrote: Hey Robert, As a first step, could we try switching all our precommit and nightly builds over to use 1.8? This is a prerequisite for HADOOP-11858, and safe to do in any case since it'll still target 1.7. +1 we have jenkin builds running on Java 8 -its just they have enough longstanding and intermittent test failures that they get ignored I've been trying to keep hadoop-common somewhat under control, but -YARN is failing due to to hard coded ports in tests - this needs to be fixed urgently: YARN-3528 -HDFS build process itself is failing with some config problem checkstyle I'll note that HADOOP-10530 details the pain Steve went through switching us to JDK7. Might be some lessons learned about how to do this transition more smoothly. The key thing is to have Jenkins happy before trying to change anything. And make sure that happy means that jenkins really is building and testing against the version of Java that you think it is. I spent about 15 minutes/hour of my weekend kicking off builds and tuning them. -I volunteered Colin Patrick McCabe to to that bit of the switch, though of course he doesn't have to accept that opportunuty ( https://issues.apache.org/jira/browse/HADOOP-10530?focusedCommentId=14239250page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14239250 Someone needs to take ownership of the Jenkins builds are broken problem -ideally someone in each subproject. That's a prerequisite to any major changes in the build process, but should also be best practice anyway. Ask your QE teams would you like the ASF projects to care whether their jenkins test runs succeed and see what they say. as a reminder, all the jenkins builds are here https://builds.apache.org/view/H-L/view/Hadoop/ please look at them and try to fix them —without even waiting for a switch to Java 8
Re: Set minimum version of Hadoop 3 to JDK 8
Hey Robert, As a first step, could we try switching all our precommit and nightly builds over to use 1.8? This is a prerequisite for HADOOP-11858, and safe to do in any case since it'll still target 1.7. I'll note that HADOOP-10530 details the pain Steve went through switching us to JDK7. Might be some lessons learned about how to do this transition more smoothly. Thanks, Andrew On Tue, Apr 21, 2015 at 3:15 PM, Robert Kanter rkan...@cloudera.com wrote: + yarn-dev, hdfs-dev, mapred-dev On Tue, Apr 21, 2015 at 3:14 PM, Robert Kanter rkan...@cloudera.com wrote: Hi all, Moving forward on some of the discussions on Hadoop 3, I've created HADOOP-11858 to set the minimum version of Hadoop 3 to JDK 8. I just wanted to let everyone know in case there's some reason we shouldn't go ahead with this. thanks - Robert
Re: Set minimum version of Hadoop 3 to JDK 8
+ yarn-dev, hdfs-dev, mapred-dev On Tue, Apr 21, 2015 at 3:14 PM, Robert Kanter rkan...@cloudera.com wrote: Hi all, Moving forward on some of the discussions on Hadoop 3, I've created HADOOP-11858 to set the minimum version of Hadoop 3 to JDK 8. I just wanted to let everyone know in case there's some reason we shouldn't go ahead with this. thanks - Robert
Re: Set minimum version of Hadoop 3 to JDK 8
We don't want JDK 8 only code going into branch-2 line. Moving Jenkins to 1.8 right-away will shield such code, how do we address that? Thanks, +Vinod On Apr 21, 2015, at 5:54 PM, Robert Kanter rkan...@cloudera.com wrote: Sure, I'll try to change the Jenkins builds to 1.8 first. On Tue, Apr 21, 2015 at 3:31 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Robert, As a first step, could we try switching all our precommit and nightly builds over to use 1.8? This is a prerequisite for HADOOP-11858, and safe to do in any case since it'll still target 1.7. I'll note that HADOOP-10530 details the pain Steve went through switching us to JDK7. Might be some lessons learned about how to do this transition more smoothly. Thanks, Andrew On Tue, Apr 21, 2015 at 3:15 PM, Robert Kanter rkan...@cloudera.com wrote: + yarn-dev, hdfs-dev, mapred-dev On Tue, Apr 21, 2015 at 3:14 PM, Robert Kanter rkan...@cloudera.com wrote: Hi all, Moving forward on some of the discussions on Hadoop 3, I've created HADOOP-11858 to set the minimum version of Hadoop 3 to JDK 8. I just wanted to let everyone know in case there's some reason we shouldn't go ahead with this. thanks - Robert
Re: Set minimum version of Hadoop 3 to JDK 8
A few options: * Only change the builds for master to use jdk8 * build with both jdk7 and jdk8 by copying jobs * build with both jdk7 and jdk8 using a jenkins matrix build Robert, if you'd like help with any of these please send me a ping off-list. On Tue, Apr 21, 2015 at 8:19 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: We don't want JDK 8 only code going into branch-2 line. Moving Jenkins to 1.8 right-away will shield such code, how do we address that? Thanks, +Vinod On Apr 21, 2015, at 5:54 PM, Robert Kanter rkan...@cloudera.com wrote: Sure, I'll try to change the Jenkins builds to 1.8 first. On Tue, Apr 21, 2015 at 3:31 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Robert, As a first step, could we try switching all our precommit and nightly builds over to use 1.8? This is a prerequisite for HADOOP-11858, and safe to do in any case since it'll still target 1.7. I'll note that HADOOP-10530 details the pain Steve went through switching us to JDK7. Might be some lessons learned about how to do this transition more smoothly. Thanks, Andrew On Tue, Apr 21, 2015 at 3:15 PM, Robert Kanter rkan...@cloudera.com wrote: + yarn-dev, hdfs-dev, mapred-dev On Tue, Apr 21, 2015 at 3:14 PM, Robert Kanter rkan...@cloudera.com wrote: Hi all, Moving forward on some of the discussions on Hadoop 3, I've created HADOOP-11858 to set the minimum version of Hadoop 3 to JDK 8. I just wanted to let everyone know in case there's some reason we shouldn't go ahead with this. thanks - Robert -- Sean
Re: Set minimum version of Hadoop 3 to JDK 8
Suppose we configure maven-compiler-plugin with source set to 1.7 but target set to 1.8 in trunk. I believe this would have the effect of generating JDK 8 bytecode, but enforcing that our code sticks to JDK 7 compatibility at compile time. Does that still satisfy requirements for HADOOP-11858? I'd prefer to avoid executing duplicate builds for different JDK versions. Pre-commit already takes a long time, and I suspect doubling the amount of builds will make us starved for executors in Jenkins. Chris Nauroth Hortonworks http://hortonworks.com/ On 4/21/15, 8:38 PM, Sean Busbey bus...@cloudera.com wrote: A few options: * Only change the builds for master to use jdk8 * build with both jdk7 and jdk8 by copying jobs * build with both jdk7 and jdk8 using a jenkins matrix build Robert, if you'd like help with any of these please send me a ping off-list. On Tue, Apr 21, 2015 at 8:19 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: We don't want JDK 8 only code going into branch-2 line. Moving Jenkins to 1.8 right-away will shield such code, how do we address that? Thanks, +Vinod On Apr 21, 2015, at 5:54 PM, Robert Kanter rkan...@cloudera.com wrote: Sure, I'll try to change the Jenkins builds to 1.8 first. On Tue, Apr 21, 2015 at 3:31 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Robert, As a first step, could we try switching all our precommit and nightly builds over to use 1.8? This is a prerequisite for HADOOP-11858, and safe to do in any case since it'll still target 1.7. I'll note that HADOOP-10530 details the pain Steve went through switching us to JDK7. Might be some lessons learned about how to do this transition more smoothly. Thanks, Andrew On Tue, Apr 21, 2015 at 3:15 PM, Robert Kanter rkan...@cloudera.com wrote: + yarn-dev, hdfs-dev, mapred-dev On Tue, Apr 21, 2015 at 3:14 PM, Robert Kanter rkan...@cloudera.com wrote: Hi all, Moving forward on some of the discussions on Hadoop 3, I've created HADOOP-11858 to set the minimum version of Hadoop 3 to JDK 8. I just wanted to let everyone know in case there's some reason we shouldn't go ahead with this. thanks - Robert -- Sean
Re: Set minimum version of Hadoop 3 to JDK 8
Sure, I'll try to change the Jenkins builds to 1.8 first. On Tue, Apr 21, 2015 at 3:31 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Robert, As a first step, could we try switching all our precommit and nightly builds over to use 1.8? This is a prerequisite for HADOOP-11858, and safe to do in any case since it'll still target 1.7. I'll note that HADOOP-10530 details the pain Steve went through switching us to JDK7. Might be some lessons learned about how to do this transition more smoothly. Thanks, Andrew On Tue, Apr 21, 2015 at 3:15 PM, Robert Kanter rkan...@cloudera.com wrote: + yarn-dev, hdfs-dev, mapred-dev On Tue, Apr 21, 2015 at 3:14 PM, Robert Kanter rkan...@cloudera.com wrote: Hi all, Moving forward on some of the discussions on Hadoop 3, I've created HADOOP-11858 to set the minimum version of Hadoop 3 to JDK 8. I just wanted to let everyone know in case there's some reason we shouldn't go ahead with this. thanks - Robert
