Hi Each external library has an separately associated custom class loader (URLClassLoader) that loads the contained functions and any dependencies (jar files) packaged inside the library. The custom loader fits into the natural hierarchy with system (classpath based) classloader as the immediate parent and bootstrap classloader as the root. As the delegation model works, a class loader attempts to load a class only after parent(s) in the lineage have failed.
In your specific case, the system class loader is able to find the jackson library from the configured classpath and so the library classloader is not involved at all for this jar. The problem arising due to different versions of a jar present across the hierarchy of classloaders is similar to one faced in web containers such as Tomcat or application servers such as Websphere where system loaded classes can override any application specific classes due to name conflict. The way you are defining a library is correct. The fix needs to be in the way the classes are loaded by AsterixDB (here acting as an application server). I see two ways:- a) Skipping the system class loader: AsterixDB uses a URLClassloader to load libraries. Setting the parent of the this classloader as null implies the bootrstrap classloader (root) as the immediate parent. The bootstrap classloader loads rt.jar (java runtime classes) and i18n.jar and these definitely do not have conflicts with your library dependencies (if they do, not sure why one would override the classes in rt.jar and so we need to fix the library). Note that the system classloader is out of the hierarchy here and so any other jars loaded by AsterixDB from its classpath (pom.xml) would not be visible. b) Defining a Custom Class loader: Inside AsterixDB, we provide a custom implementation of classloader that attempts to load a class prior to delegating to the parent. This way, the library classes and packaged dependencies override any system level classes from the class path and even the classes contained in rt.jar and i18n.jar. I am opening the discussion here to suggest further alternatives or provide preferences. I have a preference for (a) (skipping the system class loader) for two reasons: a) it is simpler b) the other option allows a custom class loader to override classes in rt.jar., which is ok but not how classloaders are supposed to work in principle. Regards, Raman On Jun 29, 2016 11:07 PM, "Mike Carey" <[email protected]> wrote: > Any classloader experts have suggestions...? > On Jun 29, 2016 10:26 PM, "Xikui Wang" <[email protected]> wrote: > > > Hi Devs, > > > > We found a problem when trying to build external functions for the > > cloudberry demo. > > > > When the external function is depend on certain library, the library that > > comes with the external function will be blocked by same library in > > AsterixDB. In our case, our external function 'geoTag' uses jackson > v2.7.1, > > and we packed all dependencies into one single jar. When running 'geoTag' > > on Asterix, it will call jackson v2.0.0 in AsterixDB which causes > > NullPointerException. We have to manually change pom.xml in AsterixDB to > > fix that. > > > > We are wondering is that because we load the external function in a wrong > > way, or this could be one possible interesting problem which is worth > > noticing. Thanks. > > > > Best, > > Xikui > > >
