Thanks Tsuyoshi for opening the discussion. One benefit of the dependency/classpath isolation work is that it can open up a possibility of having diverging dependencies in a safe manner so that upgrading libraries may have less impact. I'll spend some more time on HADOOP-13070 to make some progress. Help is welcome there! :)
That said, upgrading libraries can still go on in parallel. If the past experience is any guide, the hadoop dependencies badly trail current user dependencies. If anything, we would be reducing the occurrences of problems or workarounds people put in by upgrading our dependencies. On Thu, Jul 21, 2016 at 2:30 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote: > Hi developers, > > I'd like to discuss how to make an advance towards dependency > management in Apache Hadoop trunk code since there has been lots work > about updating dependencies in parallel. Summarizing recent works and > activities as follows: > > 0) Currently, we have merged minimum update dependencies for making > Hadoop JDK-8 compatible(compilable and runnable on JDK-8). > 1) After that, some people suggest that we should update the other > dependencies on trunk(e.g. protobuf, netty, jackthon etc.). > 2) In parallel, Sangjin and Sean are working on classpath isolation: > HADOOP-13070, HADOOP-11804 and HADOOP-11656. > > Main problems we try to solve in the activities above is as follows: > > * 1) tries to solve dependency hell between user-level jar and > system(Hadoop)-level jar. > * 2) tries to solve updating old libraries. > > IIUC, 1) and 2) looks not related, but it's related in fact. 2) tries > to separate class loader between client-side dependencies and > server-side dependencies in Hadoop, so we can the change policy of > updating libraries after doing 2). We can also decide which libraries > can be shaded after 2). > > Hence, IMHO, a straight way we should go to is doing 2 at first. > After that, we can update both client-side and server-side > dependencies based on new policy(maybe we should discuss what kind of > incompatibility is acceptable, and the others are not). > > Thoughts? > > Thanks, > - Tsuyoshi > > --------------------------------------------------------------------- > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org > >
