Re: External function dependency problem

Sun, 03 Jul 2016 23:38:23 -0700 

Sounds good to me!

Cheers,
Till

On 3 Jul 2016, at 23:34, Michael Blow wrote:


I think a better solution might be to invert the normal class loader
resolution order with an extension of URLClassLoader that overrides
loadClass(), which tries resolving classes internally before delegating to the parent. Boot classpath classes can still be forced to load correctly. I think this will solve the use case without the regression due to inability 
to resolve *db classes.

Thanks,

-MDB

On Sat, Jul 2, 2016 at 11:44 PM Xikui Wang <[email protected]> wrote:


The error log is as follow:

org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.algebricks.common.exceptions.AlgebricksException:
 Unable to load/instantiate class
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory
at

org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.open(AssignRuntimeFactory.java:111)
at

org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$2.open(AlgebricksMetaOperatorDescriptor.java:143)
at

org.apache.asterix.external.feed.dataflow.SyncFeedRuntimeInputHandler.open(SyncFeedRuntimeInputHandler.java:39)
at

org.apache.asterix.external.operators.FeedMetaComputeNodePushable.open(FeedMetaComputeNodePushable.java:127)
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:341)
at org.apache.hyracks.control.nc.Task.run(Task.java:297)
at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.hyracks.algebricks.common.exceptions.AlgebricksException:
 Unable to load/instantiate class
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory
at

org.apache.asterix.external.library.ExternalFunction.<init>(ExternalFunction.java:75)
at

org.apache.asterix.external.library.ExternalScalarFunction.<init>(ExternalFunctionProvider.java:53)
at

org.apache.asterix.external.library.ExternalFunctionProvider.getExternalFunctionEvaluator(ExternalFunctionProvider.java:39)
at

org.apache.asterix.external.library.ExternalScalarFunctionEvaluatorFactory.createScalarEvaluator(ExternalScalarFunctionEvaluatorFactory.java:41)
at

org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.open(AssignRuntimeFactory.java:109)
... 8 more
Caused by: java.lang.ClassCastException:
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory cannot be cast to
org.apache.asterix.external.api.IFunctionFactory
at

org.apache.asterix.external.library.ExternalFunction.<init>(ExternalFunction.java:72)
... 12 more
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.algebricks.common.exceptions.AlgebricksException:
 Unable to load/instantiate class
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory
at

org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.open(AssignRuntimeFactory.java:111)
at

org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$2.open(AlgebricksMetaOperatorDescriptor.java:143)
at

org.apache.asterix.external.feed.dataflow.SyncFeedRuntimeInputHandler.open(SyncFeedRuntimeInputHandler.java:39)
at

org.apache.asterix.external.operators.FeedMetaComputeNodePushable.open(FeedMetaComputeNodePushable.java:127)
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:341)
at org.apache.hyracks.control.nc.Task.run(Task.java:297)
at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.hyracks.algebricks.common.exceptions.AlgebricksException:
 Unable to load/instantiate class
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory
at

org.apache.asterix.external.library.ExternalFunction.<init>(ExternalFunction.java:75)
at

org.apache.asterix.external.library.ExternalScalarFunction.<init>(ExternalFunctionProvider.java:53)
at

org.apache.asterix.external.library.ExternalFunctionProvider.getExternalFunctionEvaluator(ExternalFunctionProvider.java:39)
at

org.apache.asterix.external.library.ExternalScalarFunctionEvaluatorFactory.createScalarEvaluator(ExternalScalarFunctionEvaluatorFactory.java:41)
at

org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.open(AssignRuntimeFactory.java:109)
... 8 more
Caused by: java.lang.ClassCastException:
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory cannot be cast to
org.apache.asterix.external.api.IFunctionFactory
at

org.apache.asterix.external.library.ExternalFunction.<init>(ExternalFunction.java:72)
... 12 more
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.algebricks.common.exceptions.AlgebricksException:
 Unable to load/instantiate class
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:365)
at org.apache.hyracks.control.nc.Task.run(Task.java:297)
at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.algebricks.common.exceptions.AlgebricksException:
 Unable to load/instantiate class
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory
at

org.apache.asterix.external.operators.FeedMetaComputeNodePushable.open(FeedMetaComputeNodePushable.java:130)
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:341)
... 4 more
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.algebricks.common.exceptions.AlgebricksException:
 Unable to load/instantiate class
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory
at

org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.open(AssignRuntimeFactory.java:111)
at

org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$2.open(AlgebricksMetaOperatorDescriptor.java:143)
at

org.apache.asterix.external.feed.dataflow.SyncFeedRuntimeInputHandler.open(SyncFeedRuntimeInputHandler.java:39)
at

org.apache.asterix.external.operators.FeedMetaComputeNodePushable.open(FeedMetaComputeNodePushable.java:127)
... 5 more
Caused by:
org.apache.hyracks.algebricks.common.exceptions.AlgebricksException:
 Unable to load/instantiate class
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory
at

org.apache.asterix.external.library.ExternalFunction.<init>(ExternalFunction.java:75)
at

org.apache.asterix.external.library.ExternalScalarFunction.<init>(ExternalFunctionProvider.java:53)
at

org.apache.asterix.external.library.ExternalFunctionProvider.getExternalFunctionEvaluator(ExternalFunctionProvider.java:39)
at

org.apache.asterix.external.library.ExternalScalarFunctionEvaluatorFactory.createScalarEvaluator(ExternalScalarFunctionEvaluatorFactory.java:41)
at

org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.open(AssignRuntimeFactory.java:109)
... 8 more
Caused by: java.lang.ClassCastException:
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory cannot be cast to
org.apache.asterix.external.api.IFunctionFactory
at

org.apache.asterix.external.library.ExternalFunction.<init>(ExternalFunction.java:72)
... 12 more
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.algebricks.common.exceptions.AlgebricksException:
 Unable to load/instantiate class
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:365)
at org.apache.hyracks.control.nc.Task.run(Task.java:297)
at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.algebricks.common.exceptions.AlgebricksException:
 Unable to load/instantiate class
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory
at

org.apache.asterix.external.operators.FeedMetaComputeNodePushable.open(FeedMetaComputeNodePushable.java:130)
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:341)
... 4 more
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.algebricks.common.exceptions.AlgebricksException:
 Unable to load/instantiate class
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory
at

org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.open(AssignRuntimeFactory.java:111)
at

org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$2.open(AlgebricksMetaOperatorDescriptor.java:143)
at

org.apache.asterix.external.feed.dataflow.SyncFeedRuntimeInputHandler.open(SyncFeedRuntimeInputHandler.java:39)
at

org.apache.asterix.external.operators.FeedMetaComputeNodePushable.open(FeedMetaComputeNodePushable.java:127)
... 5 more
Caused by:
org.apache.hyracks.algebricks.common.exceptions.AlgebricksException:
 Unable to load/instantiate class
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory
at

org.apache.asterix.external.library.ExternalFunction.<init>(ExternalFunction.java:75)
at

org.apache.asterix.external.library.ExternalScalarFunction.<init>(ExternalFunctionProvider.java:53)
at

org.apache.asterix.external.library.ExternalFunctionProvider.getExternalFunctionEvaluator(ExternalFunctionProvider.java:39)
at

org.apache.asterix.external.library.ExternalScalarFunctionEvaluatorFactory.createScalarEvaluator(ExternalScalarFunctionEvaluatorFactory.java:41)
at

org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.open(AssignRuntimeFactory.java:109)
... 8 more
Caused by: java.lang.ClassCastException:
edu.uci.ics.cloudberry.noah.feed.GeoTagFactory cannot be cast to
org.apache.asterix.external.api.IFunctionFactory
at

org.apache.asterix.external.library.ExternalFunction.<init>(ExternalFunction.java:72)
... 12 more
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.InterruptedException
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:365)
at org.apache.hyracks.control.nc.Task.run(Task.java:297)
at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.InterruptedException
at

org.apache.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender(NonDeterministicChannelReader.java:126)
at

org.apache.hyracks.dataflow.std.collectors.NonDeterministicFrameReader.nextFrame(NonDeterministicFrameReader.java:43)
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:344)
... 4 more
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at

org.apache.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender(NonDeterministicChannelReader.java:124)
... 6 more
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.InterruptedException
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:365)
at org.apache.hyracks.control.nc.Task.run(Task.java:297)
at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.InterruptedException
at

org.apache.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender(NonDeterministicChannelReader.java:126)
at

org.apache.hyracks.dataflow.std.collectors.NonDeterministicFrameReader.nextFrame(NonDeterministicFrameReader.java:43)
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:344)
... 4 more
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at

org.apache.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender(NonDeterministicChannelReader.java:124)
... 6 more
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.InterruptedException
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:365)
at org.apache.hyracks.control.nc.Task.run(Task.java:297)
at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.InterruptedException
at

org.apache.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender(NonDeterministicChannelReader.java:126)
at

org.apache.hyracks.dataflow.std.collectors.NonDeterministicFrameReader.nextFrame(NonDeterministicFrameReader.java:43)
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:344)
... 4 more
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at

org.apache.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender(NonDeterministicChannelReader.java:124)
... 6 more
org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.InterruptedException
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:365)
at org.apache.hyracks.control.nc.Task.run(Task.java:297)
at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.InterruptedException
at

org.apache.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender(NonDeterministicChannelReader.java:126)
at

org.apache.hyracks.dataflow.std.collectors.NonDeterministicFrameReader.nextFrame(NonDeterministicFrameReader.java:43)
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:344)
... 4 more
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at

org.apache.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender(NonDeterministicChannelReader.java:124)
... 6 more
testFile

src/test/resources/runtimets/queries/external-library/tweetGeoTag/tweetGeoTag.3.update.aql
raised an exception:
java.lang.Exception: HTTP operation failed: 2
STATUS LINE: HTTP/1.1 500 Server Error
SUMMARY: Error: Failure in feed
STACKTRACE: org.apache.asterix.common.exceptions.AsterixException: Failure 
in feed
at

org.apache.asterix.external.feed.management.FeedLifecycleEventSubscriber.assertNoFailure(FeedLifecycleEventSubscriber.java:62)
at

org.apache.asterix.external.feed.management.FeedLifecycleEventSubscriber.assertEvent(FeedLifecycleEventSubscriber.java:55)
at

org.apache.asterix.aql.translator.QueryTranslator.handleConnectFeedStatement(QueryTranslator.java:2238)
at

org.apache.asterix.aql.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:366)
at

org.apache.asterix.aql.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:253)
at

org.apache.asterix.api.http.servlet.RESTAPIServlet.handleRequest(RESTAPIServlet.java:195)
at

org.apache.asterix.api.http.servlet.UpdateAPIServlet.handleRequest(UpdateAPIServlet.java:30)
at

org.apache.asterix.api.http.servlet.RESTAPIServlet.doPost(RESTAPIServlet.java:162)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:546) 
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:483)
at

org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:228)
at

org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:956)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:411)
at

org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:188)
at

org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:891)
at

org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at

org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114)
at org.eclipse.jetty.server.Server.handle(Server.java:353)
at

org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:598)
at

org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1076)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:805)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:427) 
at

org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:510)
at

org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34)
at

org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40)
at

org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:450)
at java.lang.Thread.run(Thread.java:745)

at

org.apache.asterix.test.aql.TestExecutor.executeHttpMethod(TestExecutor.java:275)
at

org.apache.asterix.test.aql.TestExecutor.executeUpdate(TestExecutor.java:331)
at
org.apache.asterix.test.aql.TestExecutor.executeTest(TestExecutor.java:496)
at
org.apache.asterix.test.aql.TestExecutor.executeTest(TestExecutor.java:762)
at
org.apache.asterix.test.runtime.ExecutionTest.test(ExecutionTest.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at

org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at

org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at

org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at

org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at

org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at

org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at

org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at

com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
at

com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at

com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) 
...Unexpected!
On Sat, Jul 2, 2016 at 7:23 PM, Raman Grover <[email protected]> 
wrote:


i am missing the attachment
On Jul 2, 2016 6:39 PM, "Xikui Wang" <[email protected]> wrote:


Hi Raman,

Thanks for your help. I tried this quick fix on my branch, but it
introduces some new exceptions. I think this causes Asterix fails at 
entering the external function. The error message is attached.

Best,
Xikui
On Fri, Jul 1, 2016 at 10:11 AM, Raman Grover <[email protected] 


wrote:
Operations related to setting up an external library are contained in 
ExternalLibraryUtil
<




https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java




At line 382, we have
// create and return the class loader
ClassLoader classLoader = new URLClassLoader(urls, parentClassLoader); 
return classLoader;

Above, we have the parentClassLoader set to the classloader for
ExternalLibraryUtil which is the application class loader (AsterixDB's classloader that loads the dependencies from pom.xml). The proposed 
solution (a) in earlier thread - skipping application classloader

would

translate to replacing the above code with

ClassLoader classLoader = new URLClassLoader(urls, null);

Regards,
Raman
On Thu, Jun 30, 2016 at 4:41 PM, Xikui Wang <[email protected]> wrote: 



Hi Abdullah,
I reverted my code to reproduce the problem. Noticing this external 
function has a couple of other bugs but the dependency one is

blocking

others, so this should be enough to reproduce the problem.




The external function package is geoTag.zip.


​

​Test scripts are in tweetGeoTag.zip
External function is loading data from data/, i.e.: data/state.json 
.

So

all json files in data.zip need to be placed under
../asterixdb/asterix-app/data/

The real_tweets_adm.adm used in ddl is also attached.

This setting will cause

java.lang.NoSuchMethodError:

com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
at




com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:541)

at




com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:452)

at

org.wololo.geojson.GeoJSONFactory.<clinit>(GeoJSONFactory.java:17)

at








edu.uci.ics.cloudberry.gnosis.USGeoJSONIndex.loadShape(IGeoIndex.scala:29)

at








edu.uci.ics.cloudberry.gnosis.USGeoGnosis$.loadShape(USGeoGnosis.scala:101)

at








edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:20)

at








edu.uci.ics.cloudberry.gnosis.USGeoGnosis$$anonfun$load$1.apply(USGeoGnosis.scala:18)

at








scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at








scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

at scala.collection.immutable.List.foreach(List.scala:381)
at

scala.collection.TraversableLike$class.map(TraversableLike.scala:245)

at scala.collection.immutable.List.map(List.scala:285)
at

edu.uci.ics.cloudberry.gnosis.USGeoGnosis.load(USGeoGnosis.scala:18)

at

edu.uci.ics.cloudberry.gnosis.USGeoGnosis.<init>(USGeoGnosis.scala:15)

at








edu.uci.ics.cloudberry.noah.feed.GeoTagFunction.initialize(GeoTagFunction.java:34)

at








org.apache.asterix.external.library.ExternalFunction.initialize(ExternalFunction.java:113)




The attachments exceed the size limits. I moved them under this

google

drive directory:








https://drive.google.com/folderview?id=0B_6Dzy3OTjaNRUVJWTRvWEtKSU0&usp=sharing


Let me know if you need anything else. Thanks.

Best,
Xikui

On Thu, Jun 30, 2016 at 12:06 PM, abdullah alamoudi <

[email protected]


wrote:


I will look into that.


@Xikui,
Do you have a reproducable use case where such a behavior can be 
observed?


Cheers,
Abdullah.

On Thu, Jun 30, 2016 at 7:53 PM, Raman Grover <

[email protected]>

wrote:


Hi
Each external library has an separately associated custom class 
loader

(URLClassLoader) that loads the contained functions and any

dependencies
(jar files) packaged inside the library. The custom loader fits 
into

the

natural hierarchy with system (classpath based) classloader as

the

immediate parent and bootstrap classloader as the root. As the

delegation
model works, a class loader attempts to load a class only after 
parent(s)

in the lineage have failed.

In your specific case, the system class loader is able to find

the

jackson

library from the configured classpath and so the library

classloader

is

not

involved at all for this jar. The problem arising due to

different

versions

of a jar present across the hierarchy of classloaders is

similar

to

one

faced in web containers such as Tomcat or application servers

such

as

Websphere where system loaded classes can override any

application

specific

classes due to name conflict.
The way you are defining a library is correct. The fix needs to 
be

in

the

way the classes are loaded by AsterixDB (here acting as an

application

server).

I see two ways:-

a) Skipping the system class loader: AsterixDB uses a

URLClassloader

to

load libraries. Setting the  parent of the this classloader as

null

implies

the bootrstrap classloader (root) as the immediate parent. The

bootstrap

classloader loads rt.jar (java runtime classes) and i18n.jar

and

these
definitely do not have conflicts with your library dependencies 
(if

they

do, not sure why one would override the classes in rt.jar and

so

we

need to

fix the library). Note that the system classloader is out of

the

hierarchy

here and so any other jars loaded by AsterixDB from its

classpath

(pom.xml)

would not be visible.

b) Defining a Custom Class loader: Inside AsterixDB, we

provide a

custom

implementation of classloader that attempts to load a class

prior

to

delegating to the parent. This way, the library classes and

packaged

dependencies override any system level classes from the class

path

and

even

the classes contained in rt.jar and i18n.jar.

I am opening the discussion here to suggest further

alternatives

or

provide

preferences.
I have a preference for (a) (skipping the system class loader) 
for

two

reasons:

a) it is simpler

b) the other option allows a custom class loader to override

classes

in

rt.jar., which is ok but not how classloaders are supposed to

work

in

principle.

Regards,

Raman







On Jun 29, 2016 11:07 PM, "Mike Carey" <[email protected]>

wrote:



Any classloader experts have suggestions...?
On Jun 29, 2016 10:26 PM, "Xikui Wang" <[email protected]>

wrote:



Hi Devs,

We found a problem when trying to build external functions

for

the

cloudberry demo.

When the external function is depend on certain library,

the

library

that

comes with the external function will be blocked by same

library

in

AsterixDB. In our case, our external function 'geoTag' uses

jackson

v2.7.1,

and we packed all dependencies into one single jar. When

running

'geoTag'

on Asterix, it will call jackson v2.0.0 in AsterixDB which

causes

NullPointerException. We have to manually change pom.xml in

AsterixDB

to

fix that.

We are wondering is that because we load the external

function

in

a

wrong

way, or this could be one possible interesting problem

which

is

worth

noticing. Thanks.

Best,
Xikui
















--
Raman

Reply via email to