On Wed, Oct 7, 2015 at 9:35 AM, Alan Burlison <alan.burli...@oracle.com> wrote: > On 06/10/2015 10:52, Steve Loughran wrote: > >> HADOOP-11127, "Improve versioning and compatibility support in native >> library for downstream hadoop-common users." says "we need to do >> better here", which is probably some way of packaging native libs. > > > From that JIRA: > >> Colin Patrick McCabe added a comment - 18/Apr/15 00:48 >> >> I was thinking we: >> 1. Add the Hadoop release version to libhadoop.so. It's very, very >> simple and solves a lot of problems here. >> 2. Remove libhadoop.so and libhdfs.so from the release tarball, since >> they are CPU and OS-specific and the tarballs are not >> 3. Schedule some follow-on work to include the native libraries >> inside jars, as Chris suggested. This will take longer but ultimately >> be the best solution. > > > And: > >> I just spotted one: HADOOP-10027. A field was removed from the Java >> layer, which still could get referenced by an older version of the native >> layer. A backwards-compatible version of that patch would preserve the >> old fields in the Java layer. > > > I've been thinking about this and I really don't think the strategy of > trying to shim old methods and fields back in to Hadoop is the correct one. > The current Java-JNI interactions have been developed in an ad-hoc manner > with no formal API definition and are explicitly Not-An-Interface and as a > result no consideration has been given to cross-version stability. A > compatibility shim approach is neither sustainable nor maintainable even on > a single platform, and will severely compromise efforts to get Hadoop native > components working on other platforms.
I agree. > > The approach suggested in HADOOP-11127 seems a much better way forward, in > particular #2 (versioned libhadoop). As pointed out in the JIRA, #1 (freeze > libahdoop forever) is an obvious non-starter, and #3 (distribute libahadoop > inside the JAR) is also a non-starter as it will not work cross-platform. > > I'm happy to work on HADOOP-10027 and make that a prerequisite for fixing > the Solaris DomainSocket issues discussed in this thread. I believe it's not > practical to provide a fix for DomainSocket on Solaris with a 'No JNI > signature changes' restriction. I think you could come up with a select/poll solution while using the old function signatures. A 4-byte int is more than enough information to pass in, given that you can use it as an index into a table in the C code. There are also a lot of other solution to this problem, like I pointed out earlier. For example, you dismissed the timer wheel suggestion because of a detail of a unit test, but we could easily change the test. Anyway, changing the function signatures in the way you described is certainly reasonable and I wouldn't object to it. It is probably the most natural solution. > > Does that sound acceptable? If so I can draft up a proposal for native > library version and platform naming, library search locations etc. Yes, I think it would be good to make some progress on HADOOP-11127. We have been putting off the issue for too long. best, Colin > > > Thanks, > > -- > Alan Burlison > --
