That classpath policy was explicitly added because we can't lock down our dependencies for security/bug fix reasons, and also because if we do update something explicitly, their transitive dependencies can change -beyond our control.
https://issues.apache.org/jira/browse/HADOOP-9555 is an example of this: an update of ZK explicitly to fix an HA problem. Are there changes in its dependencies? I don't know. But we didn't have a choice to update if we wanted NN & RM failover to work reliably, so we have to take any other changes that went in. JDK upgrades can be viewed as an extension of this -we are changing the base platform that Hadoop runs on. More precisely, for the Java 6- >Java 7 update, we are reflecting the fact that nobody is running in production on Java 6 Do you realise we actually moved to Java 6 in 2008? https://issues.apache.org/jira/browse/HADOOP-2325 . That was six years ago -half the names on that list are not active on the project any more. What we did there was issue a warning in 0.18 that it would be the last Java 5 version; 0.19 moved up -we can do the same for a Hadoop 2.x release at some point this year. On 24 June 2014 11:43, Arun C Murthy <a...@hortonworks.com> wrote: > Andrew, > > Thanks for starting this thread. I'll edit the wiki to provide more > context around rolling-upgrades etc. which, as I pointed out in the > original thread, are key IMHO. > > On Jun 24, 2014, at 11:17 AM, Andrew Wang <andrew.w...@cloudera.com> > wrote: > > https://wiki.apache.org/hadoop/MovingToJdk7and8 > > > > I think based on our current compatibility guidelines, Proposal A is the > > most attractive. We're pretty hamstrung by the requirement to keep the > > classpath the same, which would be solved by either OSGI or shading our > > deps (but that's a different discussion). > > I don't see that anywhere in our current compatibility guidelines. > > As you can see from > http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Compatibility.html > we do not have such a policy (pasted here for convenience): > > Java Classpath > > User applications built against Hadoop might add all Hadoop jars > (including Hadoop's library dependencies) to the application's classpath. > Adding new dependencies or updating the version of existing dependencies > may interfere with those in applications' classpaths. > > Policy > > Currently, there is NO policy on when Hadoop's dependencies can change. > > Furthermore, we have *already* changed our classpath in hadoop-2.x. Again, > as I pointed out in the previous thread, here is the precedent: > > On Jun 21, 2014, at 5:59 PM, Arun C Murthy <a...@hortonworks.com> wrote: > > > Also, this is something we already have done i.e. we updated some of our > software deps in hadoop-2.4 v/s hadoop-2.2 - clearly not something as > dramatic as JDK. Here are some examples: > > https://issues.apache.org/jira/browse/HADOOP-9991 > > https://issues.apache.org/jira/browse/HADOOP-10102 > > https://issues.apache.org/jira/browse/HADOOP-10103 > > https://issues.apache.org/jira/browse/HADOOP-10104 > > https://issues.apache.org/jira/browse/HADOOP-10503 > > thanks, > Arun > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
