I had to slap Hadoop-yarn 2.4.1 libraries on top of slider 0.40 to get the AM restart feature working. It's kind of a hack, so +1 for the plan to move slider to hadoop 2.5.
-Gour On Tue, Aug 5, 2014 at 6:41 AM, Jon Maron <[email protected]> wrote: > Sounds liike a viable plan. Do we know the hadoop version that is planned > for Champlain? > > — Jon > > On Aug 5, 2014, at 5:31 AM, Steve Loughran <[email protected]> wrote: > > > A quick update on Hadoop versions > > > > Slider 0.40: works on Hadoop 2.4.0+; but AM restart doesn't work properly > > on Hadoop 2.4 > > > > Hadoop 2.5 is about to RC; I want to do a couple of tests there as part > of > > the Hadoop release process -Slider is the defacto AM failure/restart > > functional test, so we get to verify it works. > > > > Hadoop 2.5 is also in sync with artifacts we need. (especially jackson, > > https://issues.apache.org/jira/browse/HADOOP-10104 ) -this syncs it up > with > > Curator. FWIW that update broke Tez as they were uploading the jackson > JARs > > themselves ). Tez worked because their Jars were the Hadoop 2.4 versions, > > and now they are unhappy about the change ( > > https://issues.apache.org/jira/browse/YARN-2092 ). My stance is that we > > can't lock down Hadoop dependencies for all YARn apps, so unless we do > > better isolation/dcache all of us YARN-app authors who want to use the > > classpath of the target machine will have to deal with that. > > > > Hadoop 2.6 is the next Hadoop release after. This has some things we'd > like > > -Credential Provider: https://issues.apache.org/jira/browse/HADOOP-10607 > > -hopefully, YARN service registry ( > > https://issues.apache.org/jira/browse/YARN-913). I have a proto registry > > design/imple there which I want to stick up for comments by the end of > the > > week. There'a a branch in features/ which uses this...I'm using slider > as a > > way of seeing if the API is usable > > -Java 7 (once I push in that patch) > > > > Once Hadoop 2.5 is out I plan to switch to Hadoop 2.6-SNAPSHOT as the > base > > version for slider dev.. good for Slider and Hadoop itself (we test the > > features), but it would mean that releases after that point won't even > work > > on Hadoop 2.5 > > > > I'm going to propose, therefore: > > > > 1. We cut a Slider 0.50 release to immediately follow the Hadoop 2.5 > > release, as the one-and-only Hadoop 2.5+ version; Slider 0.40 will be the > > last Hadoop 2.4 release. There's no actual source difference between > them, > > we'd just build, test and release against the 2.5 JARs. I already test > > against branch-2 on Java 8 on one VM, so don't expect surprises. > > > > 2. After that release, we switch develop/ to build against Hadoop > branch-2 > > only. Either developers build locally, or they do builds against the > apache > > snapshot repository > > > > Thoughts? > > > > -steve > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
