Re: [VOTE] Release Apache Hadoop 0.23.11
+1 (binding) - Verified signatures and digests - Deployed binary tarball to a single-node cluster and ran some MR example jobs - Built from source, deployed to a single-node cluster and ran some MR example jobs Jason On 06/19/2014 10:14 AM, Thomas Graves wrote: Hey Everyone, There have been various bug fixes that have went into branch-0.23 since the 0.23.10 release. We think its time to do a 0.23.11. This is also the last planned release off of branch-0.23 we plan on doing. The RC is available at: http://people.apache.org/~tgraves/hadoop-0.23.11-candidate-0/ The RC Tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.11-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days til June 26th. I am +1 (binding). thanks, Tom Graves
Hadoop-Mapreduce-trunk - Build # 1812 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1812/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 35162 lines...] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [3.316s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [55.447s] [INFO] hadoop-mapreduce-client-common SUCCESS [26.174s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [4.438s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [7:10.890s] [INFO] hadoop-mapreduce-client-hs SUCCESS [4:45.027s] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [1:38:52.153s] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1:52:19.846s [INFO] Finished at: Wed Jun 25 16:16:59 UTC 2014 [INFO] Final Memory: 28M/80M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project hadoop-mapreduce-client-jobclient: There was a timeout or other error in the fork - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Sending artifact delta relative to Hadoop-Mapreduce-trunk #1765 Archived 2 artifacts Archive block size is 32768 Received 0 blocks and 16954836 bytes Compression is 0.0% Took 12 sec Updating HADOOP-10747 Updating HADOOP-10746 Updating HADOOP-10728 Updating YARN-2195 Updating HDFS-6587 Updating YARN-2192 Updating HDFS-6475 Updating YARN-2152 Updating HADOOP-10717 Updating YARN-2109 Updating YARN-1365 Updating HADOOP-9629 Updating YARN-2111 Updating YARN-2074 Updating HADOOP-10652 Updating HDFS-6593 Updating YARN-2072 Updating HDFS-6430 Updating HADOOP-10674 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
Re: Moving to JDK7, JDK8 and new major releases
On Tue, Jun 24, 2014 at 4:44 PM, Alejandro Abdelnur t...@cloudera.com wrote: After reading this thread and thinking a bit about it, I think it should be OK such move up to JDK7 in Hadoop I agree with Alejandro. Changing minimum JDKs is not an incompatible change and is fine in the 2 branch. (Although I think it is would *not* be appropriate for a patch release.) Of course we need to do it with forethought and testing, but moving off of JDK 6, which is EOL'ed is a good thing. Moving to Java 8 as a minimum seems much too aggressive and I would push back on that. I'm also think that we need to let the dust settle on the Hadoop 2 line for a while before we talk about Hadoop 3. It seems that it has only been in the last 6 months that Hadoop 2 adoption has reached the main stream users. Our user community needs time to digest the changes in Hadoop 2.x before we fracture the community by starting to discuss Hadoop 3 releases. .. Owen
Re: [VOTE] Change by-laws on release votes: 5 days instead of 7
+1 Arpit On Tue, Jun 24, 2014 at 1:53 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, As discussed, I'd like to call a vote on changing our by-laws to change release votes from 7 days to 5. I've attached the change to by-laws I'm proposing. Please vote, the vote will the usual period of 7 days. thanks, Arun [main]$ svn diff Index: author/src/documentation/content/xdocs/bylaws.xml === --- author/src/documentation/content/xdocs/bylaws.xml (revision 1605015) +++ author/src/documentation/content/xdocs/bylaws.xml (working copy) @@ -344,7 +344,16 @@ pVotes are open for a period of 7 days to allow all active voters time to consider the vote. Votes relating to code changes are not subject to a strict timetable but should be -made as timely as possible./p/li +made as timely as possible./p + + ul + li strongProduct Release - Vote Timeframe/strong + pRelease votes, alone, run for a period of 5 days. All other + votes are subject to the above timeframe of 7 days./p + /li + /ul + /li + /ul /section /body -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Change by-laws on release votes: 5 days instead of 7
+1 -giri On Wed, Jun 25, 2014 at 12:02 PM, Arpit Agarwal aagar...@hortonworks.com wrote: +1 Arpit On Tue, Jun 24, 2014 at 1:53 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, As discussed, I'd like to call a vote on changing our by-laws to change release votes from 7 days to 5. I've attached the change to by-laws I'm proposing. Please vote, the vote will the usual period of 7 days. thanks, Arun [main]$ svn diff Index: author/src/documentation/content/xdocs/bylaws.xml === --- author/src/documentation/content/xdocs/bylaws.xml (revision 1605015) +++ author/src/documentation/content/xdocs/bylaws.xml (working copy) @@ -344,7 +344,16 @@ pVotes are open for a period of 7 days to allow all active voters time to consider the vote. Votes relating to code changes are not subject to a strict timetable but should be -made as timely as possible./p/li +made as timely as possible./p + + ul + li strongProduct Release - Vote Timeframe/strong + pRelease votes, alone, run for a period of 5 days. All other + votes are subject to the above timeframe of 7 days./p + /li + /ul + /li + /ul /section /body -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 0.23.11
+1 (non-binding) Stood up a pseudo-dist cluster and ran a few MR jobs. On Wed, Jun 25, 2014 at 9:05 AM, Jason Lowe jl...@yahoo-inc.com.invalid wrote: +1 (binding) - Verified signatures and digests - Deployed binary tarball to a single-node cluster and ran some MR example jobs - Built from source, deployed to a single-node cluster and ran some MR example jobs Jason On 06/19/2014 10:14 AM, Thomas Graves wrote: Hey Everyone, There have been various bug fixes that have went into branch-0.23 since the 0.23.10 release. We think its time to do a 0.23.11. This is also the last planned release off of branch-0.23 we plan on doing. The RC is available at: http://people.apache.org/~tgraves/hadoop-0.23.11-candidate-0/ The RC Tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.11-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days til June 26th. I am +1 (binding). thanks, Tom Graves
Re: Moving to JDK7, JDK8 and new major releases
I'm also +1 for getting us to JDK7 within the 2.x line after reading the proposals and catching up on the discussion in this thread. Has anyone yet considered how to coordinate this change with downstream projects? Would we request downstream projects to upgrade to JDK7 first before we make the move? Would we switch to JDK7, but run javac -target 1.6 to maintain compatibility for downstream projects during an interim period? Chris Nauroth Hortonworks http://hortonworks.com/ On Wed, Jun 25, 2014 at 9:48 AM, Owen O'Malley omal...@apache.org wrote: On Tue, Jun 24, 2014 at 4:44 PM, Alejandro Abdelnur t...@cloudera.com wrote: After reading this thread and thinking a bit about it, I think it should be OK such move up to JDK7 in Hadoop I agree with Alejandro. Changing minimum JDKs is not an incompatible change and is fine in the 2 branch. (Although I think it is would *not* be appropriate for a patch release.) Of course we need to do it with forethought and testing, but moving off of JDK 6, which is EOL'ed is a good thing. Moving to Java 8 as a minimum seems much too aggressive and I would push back on that. I'm also think that we need to let the dust settle on the Hadoop 2 line for a while before we talk about Hadoop 3. It seems that it has only been in the last 6 months that Hadoop 2 adoption has reached the main stream users. Our user community needs time to digest the changes in Hadoop 2.x before we fracture the community by starting to discuss Hadoop 3 releases. .. Owen -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Change by-laws on release votes: 5 days instead of 7
+1 (binding) Kihwal On 6/24/14, 3:53 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, As discussed, I'd like to call a vote on changing our by-laws to change release votes from 7 days to 5. I've attached the change to by-laws I'm proposing. Please vote, the vote will the usual period of 7 days. thanks, Arun [main]$ svn diff Index: author/src/documentation/content/xdocs/bylaws.xml === --- author/src/documentation/content/xdocs/bylaws.xml (revision 1605015) +++ author/src/documentation/content/xdocs/bylaws.xml (working copy) @@ -344,7 +344,16 @@ pVotes are open for a period of 7 days to allow all active voters time to consider the vote. Votes relating to code changes are not subject to a strict timetable but should be -made as timely as possible./p/li +made as timely as possible./p + + ul + li strongProduct Release - Vote Timeframe/strong + pRelease votes, alone, run for a period of 5 days. All other + votes are subject to the above timeframe of 7 days./p + /li + /ul + /li + /ul /section /body -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Moving to JDK7, JDK8 and new major releases
Chris, Compiling with jdk7 and doing javac -target 1.6 is not sufficient, you are still using jdk7 libraries and you could use new APIs, thus breaking jdk6 both at compile and runtime. you need to compile with jdk6 to ensure you are not running into that scenario. that is why i was suggesting the nightly jdk6 build/test jenkins job. On Wed, Jun 25, 2014 at 2:04 PM, Chris Nauroth cnaur...@hortonworks.com wrote: I'm also +1 for getting us to JDK7 within the 2.x line after reading the proposals and catching up on the discussion in this thread. Has anyone yet considered how to coordinate this change with downstream projects? Would we request downstream projects to upgrade to JDK7 first before we make the move? Would we switch to JDK7, but run javac -target 1.6 to maintain compatibility for downstream projects during an interim period? Chris Nauroth Hortonworks http://hortonworks.com/ On Wed, Jun 25, 2014 at 9:48 AM, Owen O'Malley omal...@apache.org wrote: On Tue, Jun 24, 2014 at 4:44 PM, Alejandro Abdelnur t...@cloudera.com wrote: After reading this thread and thinking a bit about it, I think it should be OK such move up to JDK7 in Hadoop I agree with Alejandro. Changing minimum JDKs is not an incompatible change and is fine in the 2 branch. (Although I think it is would *not* be appropriate for a patch release.) Of course we need to do it with forethought and testing, but moving off of JDK 6, which is EOL'ed is a good thing. Moving to Java 8 as a minimum seems much too aggressive and I would push back on that. I'm also think that we need to let the dust settle on the Hadoop 2 line for a while before we talk about Hadoop 3. It seems that it has only been in the last 6 months that Hadoop 2 adoption has reached the main stream users. Our user community needs time to digest the changes in Hadoop 2.x before we fracture the community by starting to discuss Hadoop 3 releases. .. Owen -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro
Re: Moving to JDK7, JDK8 and new major releases
+1 (non-binding) for 2.5 to be the last release to ensure JDK6. My higher-level goal though is to avoid going through this same pain again when JDK7 goes EOL. I'd like to do a JDK8-based release before then for this reason. This is why I suggested skipping an intermediate 2.x+JDK7 release and leapfrogging to 3.0+JDK8. I'm thinking skipping an intermediate release and leapfrogging to 3.0 makes it difficult to maintain branch-2. It's only about a half year from 2.2 GA, so we should maintain branch-2 and create bug-fix releases for long-term even if 3.0+JDK8 is released. Thanks, Akira (2014/06/24 17:56), Steve Loughran wrote: +1, though I think 2.5 may be premature if we want to send a warning note last ever. That's an issue for followon when in branch 2. Guava and protobuf.jar are two things we have to leave alone, with the first being unfortunate, but their attitude to updates is pretty dramatic. The latter? We all know how traumatic that can be. -Steve On 24 June 2014 16:44, Alejandro Abdelnur t...@cloudera.com wrote: After reading this thread and thinking a bit about it, I think it should be OK such move up to JDK7 in Hadoop 2 for the following reasons: * Existing Hadoop 2 releases and related projects are running on JDK7 in production. * Commercial vendors of Hadoop have already done lot of work to ensure Hadoop on JDK7 works while keeping Hadoop on JDK6 working. * Different from many of the 3rd party libraries used by Hadoop, JDK is much stricter on backwards compatibility. IMPORTANT: I take this as an exception and not as a carte blanche for 3rd party dependencies and for moving from JDK7 to JDK8 (though it could OK for the later if we end up in the same state of affairs) Even for Hadoop 2.5, I think we could do the move: * Create the Hadoop 2.5 release branch. * Have one nightly Jenkins job that builds Hadoop 2.5 branch with JDK6 to ensure not JDK7 language/API feature creeps out in Hadoop 2.5. Keep this for all Hadoop 2.5.x releases. * Sanity tests for the Hadoop 2.5.x releases should be done with JDK7. * Apply Steve’s patch to require JDK7 on trunk and branch-2. * Move all Apache Jenkins jobs to build/test using JDK7. * Starting from Hadoop 2.6 we support JDK7 language/API features. Effectively what we are ensuring that Hadoop 2.5.x builds and test with JDK6 JDK7 and that all tests towards the release are done with JDK7. Users can proactively upgrade to JDK7 before upgrading to Hadoop 2.5.x, or if upgrade to Hadoop 2.5.x and they run into any issue because of JDK6 (which it would be quite unlikely) they can reactively upgrade to JDK7. Thoughts? On Tue, Jun 24, 2014 at 4:22 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hi all, On dependencies, we've bumped library versions when we think it's safe and the APIs in the new version are compatible. Or, it's not leaked to the app classpath (e.g the JUnit version bump). I think the JIRAs Arun mentioned fall into one of those categories. Steve can do a better job explaining this to me, but we haven't bumped things like Jetty or Guava because they are on the classpath and are not compatible. There is this line in the compat guidelines: - Existing MapReduce, YARN HDFS applications and frameworks should work unmodified within a major release i.e. Apache Hadoop ABI is supported. Since Hadoop apps can and do depend on the Hadoop classpath, the classpath is effectively part of our API. I'm sure there are user apps out there that will break if we make incompatible changes to the classpath. I haven't read up on the MR JIRA Arun mentioned, but there MR isn't the only YARN app out there. Sticking to the theme of work unmodified, let's think about the user effort required to upgrade their JDK. This can be a very expensive task. It might need approval up and down the org, meaning lots of certification, testing, and signoff. Considering the amount of user effort involved here, it really seems like dropping a JDK is something that should only happen in a major release. Else, there's the potential for nasty surprises in a supposedly minor release. That said, we are in an unhappy place right now regarding JDK6, and it's true that almost everyone's moved off of JDK6 at this point. So, I'd be okay with an intermediate 2.x release that drops JDK6 support (but no incompatible changes to the classpath like Guava). This is basically free, and we could start using JDK7 idioms like multi-catch and new NIO stuff in Hadoop code (a minor draw I guess). My higher-level goal though is to avoid going through this same pain again when JDK7 goes EOL. I'd like to do a JDK8-based release before then for this reason. This is why I suggested skipping an intermediate 2.x+JDK7 release and leapfrogging to 3.0+JDK8. 10 months is really not that far in the future, and it seems like a better place to focus our efforts. I was also hoping it'd be realistic to fix our classpath leakage by then, since then
Re: [VOTE] Release Apache Hadoop 2.4.1
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 +1 Built and deployed clusters on Amazon. Ran a basic test suite. Thanks Arun On 06/25/14 17:11, Akira AJISAKA wrote: Thanks Arun for another RC! I'm +1 (non-binding) for RC2. HDFS-6527 should be reverted because the issue is only in 2.5 and trunk. In addition, I hope HDFS-6591 to be merged. Other than that, RC1 is good to me. I tested RC1 with distributed cluster on CentOS 6.3: - Successful build from src (including native library) - Successful RM automatic fail-over and running MapReduce job also succeeded - Successful rolling upgrade HDFS from 2.4.0 to 2.4.1 - Successful downgrade HDFS from 2.4.1 to 2.4.0 - Documentation looks good Thanks, Akira (2014/06/23 9:59), Steve Loughran wrote: someone's filed a JIRA on loops in Hedged Read, with tests ... https://issues.apache.org/jira/browse/HDFS-6591 On 23 June 2014 08:58, Mit Desai mitde...@yahoo-inc.com.invalid wrote: +1 (non-binding) Tested on: Fedora17 -Successful build from src (including native) -Verified Signature -Deployed source to my single node cluster and ran couple of sample MR jobs - M!T On 6/21/14, 1:51 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created another release candidate (rc1) for hadoop-2.4.1 based on the feedback that I would like to push out. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJTq3XoAAoJEGunL/HJl4XexG0QAJnmxxfljAB2QWbkK5EfQNq3 DW8PkA2tZAyLrdCeMaBMOybnrrtHRUJpPloh34pm6aeFINIcdwjYlx/42Zfe5Wk8 24nLsDYad4Zai0N9hwOs1zk8z2Tj0cZbA3VoOqrexTnGQb6Z7O12yY/vRJ1iWanT Qa2qCgrfleUoCxBwTBrtO68Z98EmJHOlWd8W6QyZpMZiKC/EsGpjkCTLZzXvirkF 5/u29HXo0yJT1xA8iCleOdp7MCTmnfCF7sLvCV2rLN2ERJPaPE19AXn8pFAqyqeH 9JVIO2SCXJbmTxIlKN/UFGgP/v0KaLleYupUltQ4CIM+omkbsBxLPN2vIHavoxup /1w7JBfmq67RIX7AHkUJgS4Dzs+GOK81dpt2niEfu1dx7h7qq4eeAvLKImjIRlRi EqKqxqWBoDAb6FGBPRHsJVXb2zxn1NAYVIYYD4AW27+S0OyrTvwQWwmurhjG+h45 XC5Z+jFG+FGc96On9DtNxSUTYB9a0GpBBnjU+u1enT99n3j0X5YGmi2B/ca4Cp9J WQ4CDeQfp4+87LijF1ZH8ObQn7L0vWudehhcMjAC3qo9NK0oZ8eSMd48WaFCS0AI /xdfJT069RN4U9633KoGT/HXIXf6pcULEc7kNCgqULjXZO7hGl2H6Q3hxCBh0Xs5 zA8LmbrvDThJYIwZRXRR =rTDe -END PGP SIGNATURE-
