i am missing the attachment On Jul 2, 2016 6:39 PM, "Xikui Wang" <[email protected]> wrote:
> Hi Raman, > > Thanks for your help. I tried this quick fix on my branch, but it > introduces some new exceptions. I think this causes Asterix fails at > entering the external function. The error message is attached. > > Best, > Xikui > > On Fri, Jul 1, 2016 at 10:11 AM, Raman Grover <[email protected]> > wrote: > >> Operations related to setting up an external library are contained in >> ExternalLibraryUtil >> < >> https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java >> > >> >> At line 382, we have >> // create and return the class loader >> >> ClassLoader classLoader = new URLClassLoader(urls, parentClassLoader); >> return classLoader; >> >> Above, we have the parentClassLoader set to the classloader for >> ExternalLibraryUtil which is the application class loader (AsterixDB's >> classloader that loads the dependencies from pom.xml). The proposed >> solution (a) in earlier thread - skipping application classloader would >> translate to replacing the above code with >> >> ClassLoader classLoader = new URLClassLoader(urls, null); >> >> Regards, >> Raman >> >> On Thu, Jun 30, 2016 at 4:41 PM, Xikui Wang <[email protected]> wrote: >> >> > > >> > > Hi Abdullah, >> > >> > I reverted my code to reproduce the problem. Noticing this external >> > function has a couple of other bugs but the dependency one is blocking >> > others, so this should be enough to reproduce the problem. >> > >> > > >> > The external function package is geoTag.zip. >> > >> > > >> > > >> > > Test scripts are in tweetGeoTag.zip >> > >> > External function is loading data from data/, i.e.: data/state.json . So >> > all json files in data.zip need to be placed under >> > ../asterixdb/asterix-app/data/ >> > >> > The real_tweets_adm.adm used in ddl is also attached. >> > >> > This setting will cause >> > >> > java.lang.NoSuchMethodError: >> > > com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z >> > > at >> > > >> com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:541) >> > > at >> > > >> com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:452) >> > > at org.wololo.geojson.GeoJSONFactory.<clinit>(GeoJSONFactory.java:17) >> > > at >> > > >> > >> edu.uci.ics.cloudberry.gnosis.USGeoJSONIndex.loadShape(IGeoIndex.scala:29) >> > > at >> > > >> > >> edu.uci.ics.cloudberry.gnosis.USGeoGnosis$.loadShape(USGeoGnosis.scala:101) >> > > at >> > > >> > >> edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:20) >> > > at >> > > >> > >> edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:18) >> > > at >> > > >> > >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) >> > > at >> > > >> > >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245) >> > > at scala.collection.immutable.List.foreach(List.scala:381) >> > > at >> scala.collection.TraversableLike$class.map(TraversableLike.scala:245) >> > > at scala.collection.immutable.List.map(List.scala:285) >> > > at >> edu.uci.ics.cloudberry.gnosis.USGeoGnosis.load(USGeoGnosis.scala:18) >> > > at >> edu.uci.ics.cloudberry.gnosis.USGeoGnosis.<init>(USGeoGnosis.scala:15) >> > > at >> > > >> > >> edu.uci.ics.cloudberry.noah.feed.GeoTagFunction.initialize(GeoTagFunction.java:34) >> > > at >> > > >> > >> org.apache.asterix.external.library.ExternalFunction.initialize(ExternalFunction.java:113) >> > > >> > >> > The attachments exceed the size limits. I moved them under this google >> > drive directory: >> > >> > >> > >> https://drive.google.com/folderview?id=0B_6Dzy3OTjaNRUVJWTRvWEtKSU0&usp=sharing >> > >> > Let me know if you need anything else. Thanks. >> > >> > Best, >> > Xikui >> > >> > On Thu, Jun 30, 2016 at 12:06 PM, abdullah alamoudi <[email protected] >> > >> > wrote: >> > >> > > I will look into that. >> > >> >> > >> @Xikui, >> > >> Do you have a reproducable use case where such a behavior can be >> > observed? >> > >> >> > >> Cheers, >> > >> Abdullah. >> > >> >> > >> On Thu, Jun 30, 2016 at 7:53 PM, Raman Grover < >> [email protected]> >> > >> wrote: >> > >> >> > >> > Hi >> > >> > >> > >> > Each external library has an separately associated custom class >> loader >> > >> > (URLClassLoader) that loads the contained functions and any >> > dependencies >> > >> > (jar files) packaged inside the library. The custom loader fits >> into >> > the >> > >> > natural hierarchy with system (classpath based) classloader as the >> > >> > immediate parent and bootstrap classloader as the root. As the >> > >> delegation >> > >> > model works, a class loader attempts to load a class only after >> > >> parent(s) >> > >> > in the lineage have failed. >> > >> > >> > >> > In your specific case, the system class loader is able to find the >> > >> jackson >> > >> > library from the configured classpath and so the library >> classloader >> > is >> > >> not >> > >> > involved at all for this jar. The problem arising due to different >> > >> versions >> > >> > of a jar present across the hierarchy of classloaders is similar to >> > one >> > >> > faced in web containers such as Tomcat or application servers such >> as >> > >> > Websphere where system loaded classes can override any application >> > >> specific >> > >> > classes due to name conflict. >> > >> > >> > >> > The way you are defining a library is correct. The fix needs to be >> in >> > >> the >> > >> > way the classes are loaded by AsterixDB (here acting as an >> application >> > >> > server). >> > >> > >> > >> > I see two ways:- >> > >> > >> > >> > a) Skipping the system class loader: AsterixDB uses a >> URLClassloader >> > to >> > >> > load libraries. Setting the parent of the this classloader as null >> > >> implies >> > >> > the bootrstrap classloader (root) as the immediate parent. The >> > bootstrap >> > >> > classloader loads rt.jar (java runtime classes) and i18n.jar and >> these >> > >> > definitely do not have conflicts with your library dependencies (if >> > they >> > >> > do, not sure why one would override the classes in rt.jar and so we >> > >> need to >> > >> > fix the library). Note that the system classloader is out of the >> > >> hierarchy >> > >> > here and so any other jars loaded by AsterixDB from its classpath >> > >> (pom.xml) >> > >> > would not be visible. >> > >> > >> > >> > b) Defining a Custom Class loader: Inside AsterixDB, we provide a >> > custom >> > >> > implementation of classloader that attempts to load a class prior >> to >> > >> > delegating to the parent. This way, the library classes and >> packaged >> > >> > dependencies override any system level classes from the class path >> and >> > >> even >> > >> > the classes contained in rt.jar and i18n.jar. >> > >> > >> > >> > I am opening the discussion here to suggest further alternatives or >> > >> provide >> > >> > preferences. >> > >> > >> > >> > I have a preference for (a) (skipping the system class loader) for >> > two >> > >> > reasons: >> > >> > >> > >> > a) it is simpler >> > >> > >> > >> > b) the other option allows a custom class loader to override >> classes >> > in >> > >> > rt.jar., which is ok but not how classloaders are supposed to work >> in >> > >> > principle. >> > >> > >> > >> > Regards, >> > >> > >> > >> > Raman >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > On Jun 29, 2016 11:07 PM, "Mike Carey" <[email protected]> wrote: >> > >> > >> > >> > > Any classloader experts have suggestions...? >> > >> > > On Jun 29, 2016 10:26 PM, "Xikui Wang" <[email protected]> wrote: >> > >> > > >> > >> > > > Hi Devs, >> > >> > > > >> > >> > > > We found a problem when trying to build external functions for >> the >> > >> > > > cloudberry demo. >> > >> > > > >> > >> > > > When the external function is depend on certain library, the >> > library >> > >> > that >> > >> > > > comes with the external function will be blocked by same >> library >> > in >> > >> > > > AsterixDB. In our case, our external function 'geoTag' uses >> > jackson >> > >> > > v2.7.1, >> > >> > > > and we packed all dependencies into one single jar. When >> running >> > >> > 'geoTag' >> > >> > > > on Asterix, it will call jackson v2.0.0 in AsterixDB which >> causes >> > >> > > > NullPointerException. We have to manually change pom.xml in >> > >> AsterixDB >> > >> > to >> > >> > > > fix that. >> > >> > > > >> > >> > > > We are wondering is that because we load the external function >> in >> > a >> > >> > wrong >> > >> > > > way, or this could be one possible interesting problem which is >> > >> worth >> > >> > > > noticing. Thanks. >> > >> > > > >> > >> > > > Best, >> > >> > > > Xikui >> > >> > > > >> > >> > > >> > >> > >> > >> >> > > >> > > >> > >> >> >> >> -- >> Raman >> > >
