In IntelliJ, nothing changed. In SBT console I got this error:
$sbt
> console
[info] Running org.apache.spark.repl.Main -usejavacp
14/04/27 08:29:44 INFO spark.HttpServer: Starting HTTP Server
14/04/27 08:29:44 INFO server.Server: jetty-7.6.8.v20121106
14/04/27 08:29:44 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:50966
Failed to initialize compiler: object scala.annotation.Annotation in
compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programatically, settings.usejavacp.value = true.
14/04/27 08:29:44 WARN repl.SparkILoop$SparkILoopInterpreter: Warning:
compiler accessed before init set up. Assuming no postInit code.
Failed to initialize compiler: object scala.annotation.Annotation in
compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programatically, settings.usejavacp.value = true.
[error] (run-main-0) java.lang.AssertionError: assertion failed: null
java.lang.AssertionError: assertion failed: null
at scala.Predef$.assert(Predef.scala:179)
at
org.apache.spark.repl.SparkIMain.initializeSynchronous(SparkIMain.scala:197)
at
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:924)
at
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:881)
at
org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:881)
at
scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:881)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:973)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
[trace] Stack trace suppressed: run last compile:runMain for the full
output.
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:runMain for the full
output.
[error] (compile:runMain) Nonzero exit code: 1
[error] Total time: 1 s, completed Apr 27, 2014 8:29:44 AM
On Sun, Apr 27, 2014 at 1:42 AM, Michael Armbrust <mich...@databricks.com>wrote:
> You'll also need:
>
> libraryDependencies += "org.apache.spark" %% "spark-repl" % <spark version>
>
>
>
> On Sat, Apr 26, 2014 at 3:32 PM, Michael Armbrust
> <mich...@databricks.com>wrote:
>
>> This is a little bit of a hack, but might work for you. You'll need to
>> be on sbt 0.13.2.
>>
>> connectInput in run := true
>>
>> outputStrategy in run := Some (StdoutOutput)
>>
>> console := {
>> (runMain in Compile).toTask(" org.apache.spark.repl.Main
>> -usejavacp").value
>> }
>>
>>
>> On Sat, Apr 26, 2014 at 1:05 PM, Jonathan Chayat <
>> jonatha...@supersonicads.com> wrote:
>>
>>> Hi Michael, thanks for your prompt reply.
>>>
>>> It seems like IntelliJ Scala Console actually runs the Scala REPL (they
>>> print the same stuff when starting up).
>>> It is probably the SBT console.
>>>
>>> When I tried the same code in the Scala REPL of my project using "sbt
>>> console" it didn't work either.
>>> It only worked in spark project's bin/spark-shell
>>>
>>> Is there a way to customize the SBT console of a project listing spark
>>> as a dependency?
>>>
>>> Thx,
>>> Jon
>>>
>>>
>>> On Sat, Apr 26, 2014 at 9:42 PM, Michael Armbrust <
>>> mich...@databricks.com> wrote:
>>>
>>>> The spark-shell is a special version of the Scala REPL that serves the
>>>> classes created for each line over HTTP. Do you know if the InteliJ Spark
>>>> console is just the normal Scala repl in a GUI wrapper, or if it is
>>>> something else entirely? If its the former, perhaps it might be possible
>>>> to tell InteliJ to bring up the spark version instead.
>>>>
>>>>
>>>> On Sat, Apr 26, 2014 at 10:47 AM, Jonathan Chayat <
>>>> jonatha...@supersonicads.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> TLDR: running spark locally through IntelliJ IDEA Scala Console
>>>>> results in java.lang.ClassNotFoundException
>>>>>
>>>>> Long version:
>>>>>
>>>>> I'm an algorithms developer in SupersonicAds - an ad network. We are
>>>>> building a major new big data project and we are now in the process of
>>>>> selecting our tech stack & tools.
>>>>>
>>>>> I'm new to Spark, but I'm very excited about it. It is my opinion that
>>>>> Spark can be a great tool for us, and that we might be able to build most
>>>>> of our toolchain on top of it.
>>>>>
>>>>> We currently develop in Scala and we are using IntelliJ IDEA as our
>>>>> IDE (we love it). One of the features I love about IDEA is the Scala
>>>>> Console which lets me work interactively with all of my project's code
>>>>> available and all of the IDE's features & convenience. That is as opposed
>>>>> to the Scala Shell & Spark Shell which I dislike because it is based on
>>>>> JLine and doesn't behave like a good shell would (I cant even Ctrl-c to
>>>>> abort a line without crashing the whole thing). Of course, as an algo guy,
>>>>> having a good REPL is crucial to me.
>>>>>
>>>>> To get started, I added the following line to build.sbt:
>>>>>
>>>>>> "org.apache.spark" %% "spark-core" % "0.9.1"
>>>>>
>>>>>
>>>>> Then, added the following main class:
>>>>>
>>>>> import org.apache.spark.SparkContext
>>>>>> import org.apache.spark.SparkContext._
>>>>>>
>>>>>
>>>>>
>>>>> object Main extends App {
>>>>>> val sc = new SparkContext("local", "myApp")
>>>>>> val r = sc.parallelize(1 to 1000)
>>>>>> println("r.filter(_ % 2 == 1).first() = " + r.filter(_ % 2 ==
>>>>>> 1).first())
>>>>>> println("r.filter(_ % 2 == 1).count() = " + r.filter(_ % 2 ==
>>>>>> 1).count())
>>>>>> }
>>>>>
>>>>>
>>>>> Make, Run, Works perfectly.
>>>>>
>>>>> Next, I try running the same in the scala console.
>>>>> Bad news - the last line throws an exception:
>>>>>
>>>>>> ERROR executor.Executor: Exception in task ID 0
>>>>>> java.lang.ClassNotFoundException:
>>>>>> $line5.$read$$iw$$iw$$iw$$iw$$anonfun$2
>>>>>
>>>>>
>>>>> It is my guess that for some reason Spark is not able to find the
>>>>> anonymous function (_ % 2 == 1). Please note I'm running locally so I did
>>>>> not provide any jars. For some reason when using first() instead of
>>>>> count()
>>>>> it works. Needless to say it also works in Spark Shell but as I stated,
>>>>> working with it is not an option.
>>>>>
>>>>> This issue brings much sadness to my heart, and I could not find a
>>>>> solution on the mailing list archives or elsewhere. I am hoping someone
>>>>> here might offer some help.
>>>>>
>>>>> Thanks,
>>>>> Jon
>>>>>
>>>>
>>>>
>>>
>>
>
