[
https://issues.apache.org/jira/browse/TOREE-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966372#comment-15966372
]
Josh trier commented on TOREE-355:
----------------------------------
I'm still hitting this problem. I pulled the toree master branch & built then
installed the pip package in dist/ folder.
I'm connecting to an yarn cluster in the pyspark kernel.
Using cloudbreak/ambari which is deploying:
Amazon Linux AMI release 2016.09
HDP 2.6
openJDK 1.8
{quote}
17/04/12 18:22:13 INFO EmptyLocalActorRef: Message
[org.apache.toree.kernel.protocol.v5.KernelMessage] from
Actor[akka://spark-kernel-actor-system/user/kernel_message_relay#-1377949836]
to Actor[akka://spark-kernel-actor-system/user/status] was not delivered. [1]
dead letters encountered. This logging can be turned off or adjusted with
configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.
17/04/12 18:22:14 WARN Main$$anon$1: No external magics provided to
PluginManager!
17/04/12 18:22:14 ERROR ApplicationMaster: User class threw exception:
java.lang.IncompatibleClassChangeError: class
org.clapper.classutil.asm.ASMEmptyVisitor has interface
org.objectweb.asm.ClassVisitor as super class
java.lang.IncompatibleClassChangeError: class
org.clapper.classutil.asm.ASMEmptyVisitor has interface
org.objectweb.asm.ClassVisitor as super class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:250)
at
org.clapper.classutil.ClassFinder.org$clapper$classutil$ClassFinder$$classData(ClassFinder.scala:427)
at
org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
at
org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
at scala.collection.immutable.Stream.map(Stream.scala:418)
at
org.clapper.classutil.ClassFinder.processOpenZip(ClassFinder.scala:385)
at org.clapper.classutil.ClassFinder.processJar(ClassFinder.scala:340)
at
org.clapper.classutil.ClassFinder.findClassesIn(ClassFinder.scala:329)
at org.clapper.classutil.ClassFinder.find(ClassFinder.scala:320)
at org.clapper.classutil.ClassFinder.getClasses(ClassFinder.scala:311)
at
org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
at
org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
at scala.util.Try$.apply(Try.scala:192)
at
org.apache.toree.plugins.PluginSearcher.loadClassMap(PluginSearcher.scala:73)
at
org.apache.toree.plugins.PluginSearcher.internalClassInfo$lzycompute(PluginSearcher.scala:35)
at
org.apache.toree.plugins.PluginSearcher.internalClassInfo(PluginSearcher.scala:34)
at
org.apache.toree.plugins.PluginSearcher.internal$lzycompute(PluginSearcher.scala:38)
at
org.apache.toree.plugins.PluginSearcher.internal(PluginSearcher.scala:38)
at
org.apache.toree.plugins.PluginManager.internalPlugins$lzycompute(PluginManager.scala:45)
at
org.apache.toree.plugins.PluginManager.internalPlugins(PluginManager.scala:44)
at
org.apache.toree.plugins.PluginManager.initialize(PluginManager.scala:80)
at
org.apache.toree.boot.layer.StandardComponentInitialization$class.initializePlugins(ComponentInitialization.scala:221)
at
org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeComponents(ComponentInitialization.scala:86)
at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35)
at
org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:101)
at
org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:40)
at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:381)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at org.apache.toree.Main$.main(Main.scala:24)
at org.apache.toree.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
17/04/12 18:22:14 INFO ApplicationMaster: Final app status: FAILED, exitCode:
15, (reason: User class threw exception:
java.lang.IncompatibleClassChangeError: class
org.clapper.classutil.asm.ASMEmptyVisitor has interface
org.objectweb.asm.ClassVisitor as super class)
17/04/12 18:22:14 ERROR ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:194)
at
org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:401)
at
org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:254)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:766)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66)
at
org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:764)
at
org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.util.concurrent.ExecutionException: Boxed Error
at scala.concurrent.impl.Promise$.resolver(Promise.scala:55)
at
scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47)
at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244)
at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
at
scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:655)
Caused by: java.lang.IncompatibleClassChangeError: class
org.clapper.classutil.asm.ASMEmptyVisitor has interface
org.objectweb.asm.ClassVisitor as super class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:250)
at
org.clapper.classutil.ClassFinder.org$clapper$classutil$ClassFinder$$classData(ClassFinder.scala:427)
at
org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
at
org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
at scala.collection.immutable.Stream.map(Stream.scala:418)
at
org.clapper.classutil.ClassFinder.processOpenZip(ClassFinder.scala:385)
at org.clapper.classutil.ClassFinder.processJar(ClassFinder.scala:340)
at
org.clapper.classutil.ClassFinder.findClassesIn(ClassFinder.scala:329)
at org.clapper.classutil.ClassFinder.find(ClassFinder.scala:320)
at org.clapper.classutil.ClassFinder.getClasses(ClassFinder.scala:311)
at
org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
at
org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
at scala.util.Try$.apply(Try.scala:192)
at
org.apache.toree.plugins.PluginSearcher.loadClassMap(PluginSearcher.scala:73)
at
org.apache.toree.plugins.PluginSearcher.internalClassInfo$lzycompute(PluginSearcher.scala:35)
at
org.apache.toree.plugins.PluginSearcher.internalClassInfo(PluginSearcher.scala:34)
at
org.apache.toree.plugins.PluginSearcher.internal$lzycompute(PluginSearcher.scala:38)
at
org.apache.toree.plugins.PluginSearcher.internal(PluginSearcher.scala:38)
at
org.apache.toree.plugins.PluginManager.internalPlugins$lzycompute(PluginManager.scala:45)
at
org.apache.toree.plugins.PluginManager.internalPlugins(PluginManager.scala:44)
at
org.apache.toree.plugins.PluginManager.initialize(PluginManager.scala:80)
at
org.apache.toree.boot.layer.StandardComponentInitialization$class.initializePlugins(ComponentInitialization.scala:221)
at
org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeComponents(ComponentInitialization.scala:86)
at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35)
at
org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:101)
at
org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:40)
at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:381)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at org.apache.toree.Main$.main(Main.scala:24)
at org.apache.toree.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
17/04/12 18:22:14 INFO Main$$anon$1: Shutting down kernel
17/04/12 18:22:14 INFO Main$$anon$1: Shutting down interpreters
17/04/12 18:22:14 INFO Main$$anon$1: Shutting down actor system
17/04/12 18:22:14 INFO ApplicationMaster: Unregistering ApplicationMaster with
FAILED (diag message: User class threw exception:
java.lang.IncompatibleClassChangeError: class
org.clapper.classutil.asm.ASMEmptyVisitor has interface
org.objectweb.asm.ClassVisitor as super class)
17/04/12 18:22:14 INFO ApplicationMaster: Deleting staging directory
hdfs://ip-10-0-153-71.ec2.internal:8020/user/testuser2/.sparkStaging/application_1492016613726_0213
17/04/12 18:22:14 INFO ShutdownHookManager: Shutdown hook called
{quote}
> java.lang.IncompatibleClassChangeError: class
> org.clapper.classutil.asm.ASMEmptyVisitor has interface
> org.objectweb.asm.ClassVisitor as super class
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: TOREE-355
> URL: https://issues.apache.org/jira/browse/TOREE-355
> Project: TOREE
> Issue Type: Bug
> Environment: Toree 0.2.0.dev1
> CDH 5.9's embedded spark-2.0.0-beta2
> Reporter: Adrien Lavoillotte
> Assignee: Luciano Resende
> Labels: easyfix
> Fix For: 0.2.0
>
>
> Using Toree 0.2.0.dev1 with CDH's embedded spark 2 did not cause TOREE-327
> for me. Instead, I got this error:
> {code}
> Exception in thread "main" java.lang.IncompatibleClassChangeError: class
> org.clapper.classutil.asm.ASMEmptyVisitor has interface
> org.objectweb.asm.ClassVisitor as super class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:250)
> at
> org.clapper.classutil.ClassFinder.org$clapper$classutil$ClassFinder$$classData(ClassFinder.scala:427)
> at
> org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
> at
> org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
> at scala.collection.immutable.Stream.map(Stream.scala:418)
> at
> org.clapper.classutil.ClassFinder.processOpenZip(ClassFinder.scala:385)
> at org.clapper.classutil.ClassFinder.processJar(ClassFinder.scala:340)
> at
> org.clapper.classutil.ClassFinder.findClassesIn(ClassFinder.scala:329)
> at org.clapper.classutil.ClassFinder.find(ClassFinder.scala:320)
> at org.clapper.classutil.ClassFinder.getClasses(ClassFinder.scala:311)
> at
> org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
> at
> org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
> at scala.util.Try$.apply(Try.scala:192)
> at
> org.apache.toree.plugins.PluginSearcher.loadClassMap(PluginSearcher.scala:73)
> at
> org.apache.toree.plugins.PluginSearcher.internalClassInfo$lzycompute(PluginSearcher.scala:35)
> at
> org.apache.toree.plugins.PluginSearcher.internalClassInfo(PluginSearcher.scala:34)
> at
> org.apache.toree.plugins.PluginSearcher.internal$lzycompute(PluginSearcher.scala:38)
> at
> org.apache.toree.plugins.PluginSearcher.internal(PluginSearcher.scala:38)
> at
> org.apache.toree.plugins.PluginManager.internalPlugins$lzycompute(PluginManager.scala:45)
> at
> org.apache.toree.plugins.PluginManager.internalPlugins(PluginManager.scala:44)
> at
> org.apache.toree.plugins.PluginManager.initialize(PluginManager.scala:80)
> at
> org.apache.toree.boot.layer.StandardComponentInitialization$class.initializePlugins(ComponentInitialization.scala:221)
> at
> org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeComponents(ComponentInitialization.scala:86)
> at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35)
> at
> org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:101)
> at
> org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:40)
> at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24)
> {code}
> Apparently in some much older version of ASM (3), {{ClassVisitor}} was an
> interface and not an abstract class. Said much older version can be found in
> some Hadoop libs or older Spark libs, and ends up in the classpath with CDH's
> spark 2.
> The simplest solution is to shadow ASM, e.g. using JarJAr Links. The
> following produces an assembly jar that does not have this issue:
> {code}
> echo 'rule org.objectweb.asm.** org.apache.toree.shadow.@0' >relocate.rules
> java -jar jarjar-1.4.jar process relocate.rules toree.jar toree-relocated.jar
> {code}
> ASM is a dependency of ClassUtil, for which [an issue on this is currently
> open|https://github.com/bmc/classutil/issues/19], but the author says he may
> not want to shadow ASM, so I thought maybe you might want to.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
