> > Hi Abdullah, I reverted my code to reproduce the problem. Noticing this external function has a couple of other bugs but the dependency one is blocking others, so this should be enough to reproduce the problem.
> The external function package is geoTag.zip. > > > Test scripts are in tweetGeoTag.zip External function is loading data from data/, i.e.: data/state.json . So all json files in data.zip need to be placed under ../asterixdb/asterix-app/data/ The real_tweets_adm.adm used in ddl is also attached. This setting will cause java.lang.NoSuchMethodError: > com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z > at > com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:541) > at > com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:452) > at org.wololo.geojson.GeoJSONFactory.<clinit>(GeoJSONFactory.java:17) > at > edu.uci.ics.cloudberry.gnosis.USGeoJSONIndex.loadShape(IGeoIndex.scala:29) > at > edu.uci.ics.cloudberry.gnosis.USGeoGnosis$.loadShape(USGeoGnosis.scala:101) > at > edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:20) > at > edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:18) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) > at scala.collection.immutable.List.foreach(List.scala:381) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:245) > at scala.collection.immutable.List.map(List.scala:285) > at edu.uci.ics.cloudberry.gnosis.USGeoGnosis.load(USGeoGnosis.scala:18) > at edu.uci.ics.cloudberry.gnosis.USGeoGnosis.<init>(USGeoGnosis.scala:15) > at > edu.uci.ics.cloudberry.noah.feed.GeoTagFunction.initialize(GeoTagFunction.java:34) > at > org.apache.asterix.external.library.ExternalFunction.initialize(ExternalFunction.java:113) > The attachments exceed the size limits. I moved them under this google drive directory: https://drive.google.com/folderview?id=0B_6Dzy3OTjaNRUVJWTRvWEtKSU0&usp=sharing Let me know if you need anything else. Thanks. Best, Xikui On Thu, Jun 30, 2016 at 12:06 PM, abdullah alamoudi <[email protected]> wrote: > I will look into that. >> >> @Xikui, >> Do you have a reproducable use case where such a behavior can be observed? >> >> Cheers, >> Abdullah. >> >> On Thu, Jun 30, 2016 at 7:53 PM, Raman Grover <[email protected]> >> wrote: >> >> > Hi >> > >> > Each external library has an separately associated custom class loader >> > (URLClassLoader) that loads the contained functions and any dependencies >> > (jar files) packaged inside the library. The custom loader fits into the >> > natural hierarchy with system (classpath based) classloader as the >> > immediate parent and bootstrap classloader as the root. As the >> delegation >> > model works, a class loader attempts to load a class only after >> parent(s) >> > in the lineage have failed. >> > >> > In your specific case, the system class loader is able to find the >> jackson >> > library from the configured classpath and so the library classloader is >> not >> > involved at all for this jar. The problem arising due to different >> versions >> > of a jar present across the hierarchy of classloaders is similar to one >> > faced in web containers such as Tomcat or application servers such as >> > Websphere where system loaded classes can override any application >> specific >> > classes due to name conflict. >> > >> > The way you are defining a library is correct. The fix needs to be in >> the >> > way the classes are loaded by AsterixDB (here acting as an application >> > server). >> > >> > I see two ways:- >> > >> > a) Skipping the system class loader: AsterixDB uses a URLClassloader to >> > load libraries. Setting the parent of the this classloader as null >> implies >> > the bootrstrap classloader (root) as the immediate parent. The bootstrap >> > classloader loads rt.jar (java runtime classes) and i18n.jar and these >> > definitely do not have conflicts with your library dependencies (if they >> > do, not sure why one would override the classes in rt.jar and so we >> need to >> > fix the library). Note that the system classloader is out of the >> hierarchy >> > here and so any other jars loaded by AsterixDB from its classpath >> (pom.xml) >> > would not be visible. >> > >> > b) Defining a Custom Class loader: Inside AsterixDB, we provide a custom >> > implementation of classloader that attempts to load a class prior to >> > delegating to the parent. This way, the library classes and packaged >> > dependencies override any system level classes from the class path and >> even >> > the classes contained in rt.jar and i18n.jar. >> > >> > I am opening the discussion here to suggest further alternatives or >> provide >> > preferences. >> > >> > I have a preference for (a) (skipping the system class loader) for two >> > reasons: >> > >> > a) it is simpler >> > >> > b) the other option allows a custom class loader to override classes in >> > rt.jar., which is ok but not how classloaders are supposed to work in >> > principle. >> > >> > Regards, >> > >> > Raman >> > >> > >> > >> > >> > >> > >> > >> > On Jun 29, 2016 11:07 PM, "Mike Carey" <[email protected]> wrote: >> > >> > > Any classloader experts have suggestions...? >> > > On Jun 29, 2016 10:26 PM, "Xikui Wang" <[email protected]> wrote: >> > > >> > > > Hi Devs, >> > > > >> > > > We found a problem when trying to build external functions for the >> > > > cloudberry demo. >> > > > >> > > > When the external function is depend on certain library, the library >> > that >> > > > comes with the external function will be blocked by same library in >> > > > AsterixDB. In our case, our external function 'geoTag' uses jackson >> > > v2.7.1, >> > > > and we packed all dependencies into one single jar. When running >> > 'geoTag' >> > > > on Asterix, it will call jackson v2.0.0 in AsterixDB which causes >> > > > NullPointerException. We have to manually change pom.xml in >> AsterixDB >> > to >> > > > fix that. >> > > > >> > > > We are wondering is that because we load the external function in a >> > wrong >> > > > way, or this could be one possible interesting problem which is >> worth >> > > > noticing. Thanks. >> > > > >> > > > Best, >> > > > Xikui >> > > > >> > > >> > >> > >
