Operations related to setting up an external library are contained in ExternalLibraryUtil <https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java>
At line 382, we have // create and return the class loader ClassLoader classLoader = new URLClassLoader(urls, parentClassLoader); return classLoader; Above, we have the parentClassLoader set to the classloader for ExternalLibraryUtil which is the application class loader (AsterixDB's classloader that loads the dependencies from pom.xml). The proposed solution (a) in earlier thread - skipping application classloader would translate to replacing the above code with ClassLoader classLoader = new URLClassLoader(urls, null); Regards, Raman On Thu, Jun 30, 2016 at 4:41 PM, Xikui Wang <[email protected]> wrote: > > > > Hi Abdullah, > > I reverted my code to reproduce the problem. Noticing this external > function has a couple of other bugs but the dependency one is blocking > others, so this should be enough to reproduce the problem. > > > > The external function package is geoTag.zip. > > > > > > > Test scripts are in tweetGeoTag.zip > > External function is loading data from data/, i.e.: data/state.json . So > all json files in data.zip need to be placed under > ../asterixdb/asterix-app/data/ > > The real_tweets_adm.adm used in ddl is also attached. > > This setting will cause > > java.lang.NoSuchMethodError: > > com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z > > at > > com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:541) > > at > > com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:452) > > at org.wololo.geojson.GeoJSONFactory.<clinit>(GeoJSONFactory.java:17) > > at > > > edu.uci.ics.cloudberry.gnosis.USGeoJSONIndex.loadShape(IGeoIndex.scala:29) > > at > > > edu.uci.ics.cloudberry.gnosis.USGeoGnosis$.loadShape(USGeoGnosis.scala:101) > > at > > > edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:20) > > at > > > edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:18) > > at > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) > > at > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) > > at scala.collection.immutable.List.foreach(List.scala:381) > > at scala.collection.TraversableLike$class.map(TraversableLike.scala:245) > > at scala.collection.immutable.List.map(List.scala:285) > > at edu.uci.ics.cloudberry.gnosis.USGeoGnosis.load(USGeoGnosis.scala:18) > > at edu.uci.ics.cloudberry.gnosis.USGeoGnosis.<init>(USGeoGnosis.scala:15) > > at > > > edu.uci.ics.cloudberry.noah.feed.GeoTagFunction.initialize(GeoTagFunction.java:34) > > at > > > org.apache.asterix.external.library.ExternalFunction.initialize(ExternalFunction.java:113) > > > > The attachments exceed the size limits. I moved them under this google > drive directory: > > > https://drive.google.com/folderview?id=0B_6Dzy3OTjaNRUVJWTRvWEtKSU0&usp=sharing > > Let me know if you need anything else. Thanks. > > Best, > Xikui > > On Thu, Jun 30, 2016 at 12:06 PM, abdullah alamoudi <[email protected]> > wrote: > > > I will look into that. > >> > >> @Xikui, > >> Do you have a reproducable use case where such a behavior can be > observed? > >> > >> Cheers, > >> Abdullah. > >> > >> On Thu, Jun 30, 2016 at 7:53 PM, Raman Grover <[email protected]> > >> wrote: > >> > >> > Hi > >> > > >> > Each external library has an separately associated custom class loader > >> > (URLClassLoader) that loads the contained functions and any > dependencies > >> > (jar files) packaged inside the library. The custom loader fits into > the > >> > natural hierarchy with system (classpath based) classloader as the > >> > immediate parent and bootstrap classloader as the root. As the > >> delegation > >> > model works, a class loader attempts to load a class only after > >> parent(s) > >> > in the lineage have failed. > >> > > >> > In your specific case, the system class loader is able to find the > >> jackson > >> > library from the configured classpath and so the library classloader > is > >> not > >> > involved at all for this jar. The problem arising due to different > >> versions > >> > of a jar present across the hierarchy of classloaders is similar to > one > >> > faced in web containers such as Tomcat or application servers such as > >> > Websphere where system loaded classes can override any application > >> specific > >> > classes due to name conflict. > >> > > >> > The way you are defining a library is correct. The fix needs to be in > >> the > >> > way the classes are loaded by AsterixDB (here acting as an application > >> > server). > >> > > >> > I see two ways:- > >> > > >> > a) Skipping the system class loader: AsterixDB uses a URLClassloader > to > >> > load libraries. Setting the parent of the this classloader as null > >> implies > >> > the bootrstrap classloader (root) as the immediate parent. The > bootstrap > >> > classloader loads rt.jar (java runtime classes) and i18n.jar and these > >> > definitely do not have conflicts with your library dependencies (if > they > >> > do, not sure why one would override the classes in rt.jar and so we > >> need to > >> > fix the library). Note that the system classloader is out of the > >> hierarchy > >> > here and so any other jars loaded by AsterixDB from its classpath > >> (pom.xml) > >> > would not be visible. > >> > > >> > b) Defining a Custom Class loader: Inside AsterixDB, we provide a > custom > >> > implementation of classloader that attempts to load a class prior to > >> > delegating to the parent. This way, the library classes and packaged > >> > dependencies override any system level classes from the class path and > >> even > >> > the classes contained in rt.jar and i18n.jar. > >> > > >> > I am opening the discussion here to suggest further alternatives or > >> provide > >> > preferences. > >> > > >> > I have a preference for (a) (skipping the system class loader) for > two > >> > reasons: > >> > > >> > a) it is simpler > >> > > >> > b) the other option allows a custom class loader to override classes > in > >> > rt.jar., which is ok but not how classloaders are supposed to work in > >> > principle. > >> > > >> > Regards, > >> > > >> > Raman > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > On Jun 29, 2016 11:07 PM, "Mike Carey" <[email protected]> wrote: > >> > > >> > > Any classloader experts have suggestions...? > >> > > On Jun 29, 2016 10:26 PM, "Xikui Wang" <[email protected]> wrote: > >> > > > >> > > > Hi Devs, > >> > > > > >> > > > We found a problem when trying to build external functions for the > >> > > > cloudberry demo. > >> > > > > >> > > > When the external function is depend on certain library, the > library > >> > that > >> > > > comes with the external function will be blocked by same library > in > >> > > > AsterixDB. In our case, our external function 'geoTag' uses > jackson > >> > > v2.7.1, > >> > > > and we packed all dependencies into one single jar. When running > >> > 'geoTag' > >> > > > on Asterix, it will call jackson v2.0.0 in AsterixDB which causes > >> > > > NullPointerException. We have to manually change pom.xml in > >> AsterixDB > >> > to > >> > > > fix that. > >> > > > > >> > > > We are wondering is that because we load the external function in > a > >> > wrong > >> > > > way, or this could be one possible interesting problem which is > >> worth > >> > > > noticing. Thanks. > >> > > > > >> > > > Best, > >> > > > Xikui > >> > > > > >> > > > >> > > >> > > > > > -- Raman
