[
https://issues.apache.org/jira/browse/TOREE-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16312294#comment-16312294
]
Sven Lerner commented on TOREE-459:
-----------------------------------
We see the following error (which may be related when trying to add a custom
jar from s3):
{code}
%AddJar s3://bucket/CustomJar.jar
{code}
{code}
8/01/05 00:52:24 WARN MagicManager: Magic AddJar failed to execute with error:
object Predef does not have a member classOf
scala.reflect.internal.FatalError: object Predef does not have a member classOf
at
scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)
at
scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)
at
scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)
at
scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)
at
scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)
at
scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)
at
scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)
at
scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)
at
scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)
at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)
at
scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)
at
scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)
at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)
at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)
at
scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)
at
scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)
at
scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)
at
scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)
at
scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)
at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)
at
scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)
at
scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)
at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)
at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)
at
scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)
at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5448)
at
scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.apply(Analyzer.scala:102)
at
scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:440)
at scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:431)
at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:440)
at
scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:94)
at
scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:93)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at
scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.run(Analyzer.scala:93)
at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1501)
at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1486)
at scala.tools.nsc.Global$Run.compileSources(Global.scala:1481)
at
scala.tools.nsc.interpreter.IMain.compileSourcesKeepingRun(IMain.scala:435)
at
scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compileAndSaveRun(IMain.scala:855)
at
scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compile(IMain.scala:813)
at
scala.tools.nsc.interpreter.IMain$Request.compile$lzycompute(IMain.scala:1002)
at scala.tools.nsc.interpreter.IMain$Request.compile(IMain.scala:997)
at scala.tools.nsc.interpreter.IMain.compile(IMain.scala:579)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:567)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific$class.bind(ScalaInterpreterSpecific.scala:136)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter.bind(ScalaInterpreter.scala:43)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$bindKernelVariable$1.apply$mcV$sp(ScalaInterpreter.scala:154)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$bindKernelVariable$1.apply(ScalaInterpreter.scala:154)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$bindKernelVariable$1.apply(ScalaInterpreter.scala:154)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific$class.doQuietly(ScalaInterpreterSpecific.scala:167)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter.doQuietly(ScalaInterpreter.scala:43)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter.bindKernelVariable(ScalaInterpreter.scala:152)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter.bindVariables(ScalaInterpreter.scala:103)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific$class.addJars(ScalaInterpreterSpecific.scala:115)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter.addJars(ScalaInterpreter.scala:43)
at org.apache.toree.kernel.api.Kernel.addJars(Kernel.scala:79)
at org.apache.toree.magic.builtin.AddJar.execute(AddJar.scala:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.toree.plugins.PluginMethod$$anonfun$invoke$2.apply(PluginMethod.scala:116)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.toree.plugins.PluginMethod.invoke(PluginMethod.scala:84)
at
org.apache.toree.plugins.PluginManager$$anonfun$8.apply(PluginManager.scala:334)
at
org.apache.toree.plugins.PluginManager$$anonfun$8.apply(PluginManager.scala:333)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.immutable.List.map(List.scala:285)
at
org.apache.toree.plugins.PluginManager.invokePluginMethods(PluginManager.scala:333)
at
org.apache.toree.plugins.PluginManager.fireEvent(PluginManager.scala:300)
at
org.apache.toree.plugins.PluginManager.fireEvent(PluginManager.scala:283)
at
org.apache.toree.plugins.PluginManager.fireEventFirstResult(PluginManager.scala:250)
at
org.apache.toree.magic.MagicManager.applyDynamic(MagicManager.scala:82)
at $line27.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:33)
at $line27.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:38)
at $line27.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:40)
at $line27.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:42)
at $line27.$read$$iw$$iw$$iw$$iw.<init>(<console>:44)
at $line27.$read$$iw$$iw$$iw.<init>(<console>:46)
at $line27.$read$$iw$$iw.<init>(<console>:48)
at $line27.$read$$iw.<init>(<console>:50)
at $line27.$read.<init>(<console>:52)
at $line27.$read$.<init>(<console>:56)
at $line27.$read$.<clinit>(<console>)
at $line27.$eval$.$print$lzycompute(<console>:7)
at $line27.$eval$.$print(<console>:6)
at $line27.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
at
scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
at
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
at
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
at
scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at
scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at
scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific$$anonfun$interpretAddTask$1$$anonfun$apply$3.apply(ScalaInterpreterSpecific.scala:376)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific$$anonfun$interpretAddTask$1$$anonfun$apply$3.apply(ScalaInterpreterSpecific.scala:371)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at scala.Console$.withErr(Console.scala:80)
at
org.apache.toree.global.StreamState$$anonfun$1$$anonfun$apply$1.apply(StreamState.scala:73)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at scala.Console$.withOut(Console.scala:53)
at
org.apache.toree.global.StreamState$$anonfun$1.apply(StreamState.scala:72)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at scala.Console$.withIn(Console.scala:124)
at
org.apache.toree.global.StreamState$.withStreams(StreamState.scala:71)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific$$anonfun$interpretAddTask$1.apply(ScalaInterpreterSpecific.scala:370)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific$$anonfun$interpretAddTask$1.apply(ScalaInterpreterSpecific.scala:370)
at
org.apache.toree.utils.TaskManager$$anonfun$add$2$$anon$2.run(TaskManager.scala:134)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
we were able to reproduce the error introduced by our custom jar by attempting
to add a dependency on org.scala-lang:scala-reflect:2.11.8
{code}
%AddDeps org.scala-lang scala-reflect 2.11.8
{code}
{code}
18/01/05 00:53:37 ERROR OneForOneStrategy: object Predef does not have a member
classOf
scala.reflect.internal.FatalError: object Predef does not have a member classOf
at
scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)
at
scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)
at
scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)
at
scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)
at
scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)
at
scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)
at
scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)
at
scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)
at
scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)
at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)
at
scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)
at
scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)
at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)
at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)
at
scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)
at
scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)
at
scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)
at
scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)
at
scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)
at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)
at
scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)
at
scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)
at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)
at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)
at
scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)
at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5448)
at
scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.apply(Analyzer.scala:102)
at
scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:440)
at scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:431)
at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:440)
at
scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:94)
at
scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:93)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at
scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.run(Analyzer.scala:93)
at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1501)
at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1486)
at scala.tools.nsc.Global$Run.compileSources(Global.scala:1481)
at
scala.tools.nsc.interpreter.IMain.compileSourcesKeepingRun(IMain.scala:435)
at
scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compileAndSaveRun(IMain.scala:855)
at
scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compile(IMain.scala:813)
at scala.tools.nsc.interpreter.IMain.bind(IMain.scala:675)
at scala.tools.nsc.interpreter.IMain.bind(IMain.scala:712)
at
scala.tools.nsc.interpreter.IMain$$anonfun$quietBind$1.apply(IMain.scala:711)
at
scala.tools.nsc.interpreter.IMain$$anonfun$quietBind$1.apply(IMain.scala:711)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
at scala.tools.nsc.interpreter.IMain.quietBind(IMain.scala:711)
at scala.tools.nsc.interpreter.IMain.compiled(IMain.scala:587)
at scala.tools.nsc.interpreter.IMain.eval(IMain.scala:1074)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific$class.read(ScalaInterpreterSpecific.scala:250)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter.read(ScalaInterpreter.scala:43)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$prepareResult$1.apply(ScalaInterpreter.scala:206)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$prepareResult$1.apply(ScalaInterpreter.scala:205)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter.prepareResult(ScalaInterpreter.scala:205)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$interpretMapToResultAndOutput$1.apply(ScalaInterpreter.scala:285)
at
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$interpretMapToResultAndOutput$1.apply(ScalaInterpreter.scala:280)
at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Success.map(Try.scala:237)
at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{code}
interestingly we are able to load our custom jar successfully if we specify it
in spark-opts of the toree installation like so
{code}
sudo jupyter toree install --interpreters="Scala,SQL,PySpark,SparkR"
--spark_home="/usr/lib/spark/" --python_exec="python"
--spark_opts="--jars=/usr/tmp/jars/CustomJar.jar"
{code}
we are running spark 2.2.0 and scala 2.11.8, and we confirmed that all
org.scala-lang dependencies in our jar are version 2.11.8
> Failure when adding a Jar.
> --------------------------
>
> Key: TOREE-459
> URL: https://issues.apache.org/jira/browse/TOREE-459
> Project: TOREE
> Issue Type: Bug
> Reporter: Naveen Swamy
> Assignee: Luciano Resende
> Fix For: 0.2.0
>
>
> I am trying to use toree with Spark 2.2.7(latest) and Scala 2.11 however when
> I add a jar it fails with the error below
> {{
> spark-2.2.7/jars/scala-library-2.11.8.jar(scala/reflect/package.class)' is
> broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala
> annotation)
> error: error while loading package, class file
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/package.class)' is broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala
> annotation)
> error: error while loading Serializable, class file
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/Serializable.class)' is
> broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala
> annotation)
> res2:
> Either[org.apache.toree.magic.CellMagicOutput,org.apache.toree.magic.LineMagicOutput]
> = Right(())
> error: error while loading Predef, class file
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/Predef.class)' is broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala
> annotation)
> }}
> The jar I am trying to add is:
> %AddJar
> https://search.maven.org/remotecontent?filepath=ml/dmlc/mxnet/mxnet-full_2.11-osx-x86_64-cpu/0.9.3a/mxnet-full_2.11-osx-x86_64-cpu-0.9.3a.jar
> Spark version 2.2.1
> Scala version 2.11.8
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
