Jakob Odersky created SPARK-11832:
-------------------------------------
Summary: Spark shell does not work from sbt with scala 2.11
Key: SPARK-11832
URL: https://issues.apache.org/jira/browse/SPARK-11832
Project: Spark
Issue Type: Bug
Components: Spark Shell
Reporter: Jakob Odersky
Priority: Minor
Using Scala 2.11, running the spark shell task from within sbt fails, however
running it from a distribution works.
h3. Steps to reproduce
# change scala version {{dev/change-scala-version.sh 2.11}}
# start sbt {{build/sbt -Dscala-2.11}}
# run shell task {{sparkShell}}
h3. Stacktrace
{code}
Failed to initialize compiler: object scala in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programmatically, settings.usejavacp.value = true.
Exception in thread "main" java.lang.NullPointerException
at
scala.reflect.internal.SymbolTable.exitingPhase(SymbolTable.scala:256)
at
scala.tools.nsc.interpreter.IMain$Request.x$20$lzycompute(IMain.scala:894)
at scala.tools.nsc.interpreter.IMain$Request.x$20(IMain.scala:893)
at
scala.tools.nsc.interpreter.IMain$Request.importsPreamble$lzycompute(IMain.scala:893)
at
scala.tools.nsc.interpreter.IMain$Request.importsPreamble(IMain.scala:893)
at
scala.tools.nsc.interpreter.IMain$Request$Wrapper.preamble(IMain.scala:915)
at
scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$apply$23.apply(IMain.scala:1325)
at
scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$apply$23.apply(IMain.scala:1324)
at scala.tools.nsc.util.package$.stringFromWriter(package.scala:64)
at
scala.tools.nsc.interpreter.IMain$CodeAssembler$class.apply(IMain.scala:1324)
at
scala.tools.nsc.interpreter.IMain$Request$Wrapper.apply(IMain.scala:906)
at
scala.tools.nsc.interpreter.IMain$Request.compile$lzycompute(IMain.scala:995)
at scala.tools.nsc.interpreter.IMain$Request.compile(IMain.scala:990)
at scala.tools.nsc.interpreter.IMain.compile(IMain.scala:577)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:563)
at scala.tools.nsc.interpreter.ILoop.reallyInterpret$1(ILoop.scala:802)
at
scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:836)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:694)
at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:404)
at
org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcZ$sp(SparkILoop.scala:39)
at
org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:38)
at
org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:38)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:213)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:38)
at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:94)
at
scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:922)
at
scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:911)
at
scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:911)
at
scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:911)
at org.apache.spark.repl.Main$.main(Main.scala:49)
at org.apache.spark.repl.Main.main(Main.scala)
{code}
h3. Workaround
In {{repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala}}, append
to the repl settings {{s.usejavacp.value = true}}
I haven't looked into the details of {{scala.tools.nsc.Settings}}, maybe
someone has an idea of what's going on.
Also, to be clear, this bug only affects scala 2.11 from within sbt; calling
spark-shell from a distribution or from anywhere using scala 2.10 works.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
