Hi Raman, Thanks for your help. I tried this quick fix on my branch, but it introduces some new exceptions. I think this causes Asterix fails at entering the external function. The error message is attached.
Best, Xikui On Fri, Jul 1, 2016 at 10:11 AM, Raman Grover <[email protected]> wrote: > Operations related to setting up an external library are contained in > ExternalLibraryUtil > < > https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java > > > > At line 382, we have > // create and return the class loader > > ClassLoader classLoader = new URLClassLoader(urls, parentClassLoader); > return classLoader; > > Above, we have the parentClassLoader set to the classloader for > ExternalLibraryUtil which is the application class loader (AsterixDB's > classloader that loads the dependencies from pom.xml). The proposed > solution (a) in earlier thread - skipping application classloader would > translate to replacing the above code with > > ClassLoader classLoader = new URLClassLoader(urls, null); > > Regards, > Raman > > On Thu, Jun 30, 2016 at 4:41 PM, Xikui Wang <[email protected]> wrote: > > > > > > > Hi Abdullah, > > > > I reverted my code to reproduce the problem. Noticing this external > > function has a couple of other bugs but the dependency one is blocking > > others, so this should be enough to reproduce the problem. > > > > > > > The external function package is geoTag.zip. > > > > > > > > > > > Test scripts are in tweetGeoTag.zip > > > > External function is loading data from data/, i.e.: data/state.json . So > > all json files in data.zip need to be placed under > > ../asterixdb/asterix-app/data/ > > > > The real_tweets_adm.adm used in ddl is also attached. > > > > This setting will cause > > > > java.lang.NoSuchMethodError: > > > com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z > > > at > > > > com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:541) > > > at > > > > com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:452) > > > at org.wololo.geojson.GeoJSONFactory.<clinit>(GeoJSONFactory.java:17) > > > at > > > > > > edu.uci.ics.cloudberry.gnosis.USGeoJSONIndex.loadShape(IGeoIndex.scala:29) > > > at > > > > > > edu.uci.ics.cloudberry.gnosis.USGeoGnosis$.loadShape(USGeoGnosis.scala:101) > > > at > > > > > > edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:20) > > > at > > > > > > edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:18) > > > at > > > > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) > > > at > > > > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) > > > at scala.collection.immutable.List.foreach(List.scala:381) > > > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:245) > > > at scala.collection.immutable.List.map(List.scala:285) > > > at edu.uci.ics.cloudberry.gnosis.USGeoGnosis.load(USGeoGnosis.scala:18) > > > at > edu.uci.ics.cloudberry.gnosis.USGeoGnosis.<init>(USGeoGnosis.scala:15) > > > at > > > > > > edu.uci.ics.cloudberry.noah.feed.GeoTagFunction.initialize(GeoTagFunction.java:34) > > > at > > > > > > org.apache.asterix.external.library.ExternalFunction.initialize(ExternalFunction.java:113) > > > > > > > The attachments exceed the size limits. I moved them under this google > > drive directory: > > > > > > > https://drive.google.com/folderview?id=0B_6Dzy3OTjaNRUVJWTRvWEtKSU0&usp=sharing > > > > Let me know if you need anything else. Thanks. > > > > Best, > > Xikui > > > > On Thu, Jun 30, 2016 at 12:06 PM, abdullah alamoudi <[email protected]> > > wrote: > > > > > I will look into that. > > >> > > >> @Xikui, > > >> Do you have a reproducable use case where such a behavior can be > > observed? > > >> > > >> Cheers, > > >> Abdullah. > > >> > > >> On Thu, Jun 30, 2016 at 7:53 PM, Raman Grover < > [email protected]> > > >> wrote: > > >> > > >> > Hi > > >> > > > >> > Each external library has an separately associated custom class > loader > > >> > (URLClassLoader) that loads the contained functions and any > > dependencies > > >> > (jar files) packaged inside the library. The custom loader fits into > > the > > >> > natural hierarchy with system (classpath based) classloader as the > > >> > immediate parent and bootstrap classloader as the root. As the > > >> delegation > > >> > model works, a class loader attempts to load a class only after > > >> parent(s) > > >> > in the lineage have failed. > > >> > > > >> > In your specific case, the system class loader is able to find the > > >> jackson > > >> > library from the configured classpath and so the library classloader > > is > > >> not > > >> > involved at all for this jar. The problem arising due to different > > >> versions > > >> > of a jar present across the hierarchy of classloaders is similar to > > one > > >> > faced in web containers such as Tomcat or application servers such > as > > >> > Websphere where system loaded classes can override any application > > >> specific > > >> > classes due to name conflict. > > >> > > > >> > The way you are defining a library is correct. The fix needs to be > in > > >> the > > >> > way the classes are loaded by AsterixDB (here acting as an > application > > >> > server). > > >> > > > >> > I see two ways:- > > >> > > > >> > a) Skipping the system class loader: AsterixDB uses a URLClassloader > > to > > >> > load libraries. Setting the parent of the this classloader as null > > >> implies > > >> > the bootrstrap classloader (root) as the immediate parent. The > > bootstrap > > >> > classloader loads rt.jar (java runtime classes) and i18n.jar and > these > > >> > definitely do not have conflicts with your library dependencies (if > > they > > >> > do, not sure why one would override the classes in rt.jar and so we > > >> need to > > >> > fix the library). Note that the system classloader is out of the > > >> hierarchy > > >> > here and so any other jars loaded by AsterixDB from its classpath > > >> (pom.xml) > > >> > would not be visible. > > >> > > > >> > b) Defining a Custom Class loader: Inside AsterixDB, we provide a > > custom > > >> > implementation of classloader that attempts to load a class prior to > > >> > delegating to the parent. This way, the library classes and packaged > > >> > dependencies override any system level classes from the class path > and > > >> even > > >> > the classes contained in rt.jar and i18n.jar. > > >> > > > >> > I am opening the discussion here to suggest further alternatives or > > >> provide > > >> > preferences. > > >> > > > >> > I have a preference for (a) (skipping the system class loader) for > > two > > >> > reasons: > > >> > > > >> > a) it is simpler > > >> > > > >> > b) the other option allows a custom class loader to override classes > > in > > >> > rt.jar., which is ok but not how classloaders are supposed to work > in > > >> > principle. > > >> > > > >> > Regards, > > >> > > > >> > Raman > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > On Jun 29, 2016 11:07 PM, "Mike Carey" <[email protected]> wrote: > > >> > > > >> > > Any classloader experts have suggestions...? > > >> > > On Jun 29, 2016 10:26 PM, "Xikui Wang" <[email protected]> wrote: > > >> > > > > >> > > > Hi Devs, > > >> > > > > > >> > > > We found a problem when trying to build external functions for > the > > >> > > > cloudberry demo. > > >> > > > > > >> > > > When the external function is depend on certain library, the > > library > > >> > that > > >> > > > comes with the external function will be blocked by same library > > in > > >> > > > AsterixDB. In our case, our external function 'geoTag' uses > > jackson > > >> > > v2.7.1, > > >> > > > and we packed all dependencies into one single jar. When running > > >> > 'geoTag' > > >> > > > on Asterix, it will call jackson v2.0.0 in AsterixDB which > causes > > >> > > > NullPointerException. We have to manually change pom.xml in > > >> AsterixDB > > >> > to > > >> > > > fix that. > > >> > > > > > >> > > > We are wondering is that because we load the external function > in > > a > > >> > wrong > > >> > > > way, or this could be one possible interesting problem which is > > >> worth > > >> > > > noticing. Thanks. > > >> > > > > > >> > > > Best, > > >> > > > Xikui > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > > > > > -- > Raman >
