This is a little bit of a hack, but might work for you. You'll need to be
on sbt 0.13.2.
connectInput in run := true
outputStrategy in run := Some (StdoutOutput)
console := {
(runMain in Compile).toTask(" org.apache.spark.repl.Main
-usejavacp").value
}
On Sat, Apr 26, 2014 at 1:05 PM, Jonathan Chayat <
jonatha...@supersonicads.com> wrote:
> Hi Michael, thanks for your prompt reply.
>
> It seems like IntelliJ Scala Console actually runs the Scala REPL (they
> print the same stuff when starting up).
> It is probably the SBT console.
>
> When I tried the same code in the Scala REPL of my project using "sbt
> console" it didn't work either.
> It only worked in spark project's bin/spark-shell
>
> Is there a way to customize the SBT console of a project listing spark as
> a dependency?
>
> Thx,
> Jon
>
>
> On Sat, Apr 26, 2014 at 9:42 PM, Michael Armbrust
> <mich...@databricks.com>wrote:
>
>> The spark-shell is a special version of the Scala REPL that serves the
>> classes created for each line over HTTP. Do you know if the InteliJ Spark
>> console is just the normal Scala repl in a GUI wrapper, or if it is
>> something else entirely? If its the former, perhaps it might be possible
>> to tell InteliJ to bring up the spark version instead.
>>
>>
>> On Sat, Apr 26, 2014 at 10:47 AM, Jonathan Chayat <
>> jonatha...@supersonicads.com> wrote:
>>
>>> Hi all,
>>>
>>> TLDR: running spark locally through IntelliJ IDEA Scala Console results
>>> in java.lang.ClassNotFoundException
>>>
>>> Long version:
>>>
>>> I'm an algorithms developer in SupersonicAds - an ad network. We are
>>> building a major new big data project and we are now in the process of
>>> selecting our tech stack & tools.
>>>
>>> I'm new to Spark, but I'm very excited about it. It is my opinion that
>>> Spark can be a great tool for us, and that we might be able to build most
>>> of our toolchain on top of it.
>>>
>>> We currently develop in Scala and we are using IntelliJ IDEA as our IDE
>>> (we love it). One of the features I love about IDEA is the Scala Console
>>> which lets me work interactively with all of my project's code available
>>> and all of the IDE's features & convenience. That is as opposed to the
>>> Scala Shell & Spark Shell which I dislike because it is based on JLine and
>>> doesn't behave like a good shell would (I cant even Ctrl-c to abort a line
>>> without crashing the whole thing). Of course, as an algo guy, having a good
>>> REPL is crucial to me.
>>>
>>> To get started, I added the following line to build.sbt:
>>>
>>>> "org.apache.spark" %% "spark-core" % "0.9.1"
>>>
>>>
>>> Then, added the following main class:
>>>
>>> import org.apache.spark.SparkContext
>>>> import org.apache.spark.SparkContext._
>>>>
>>>
>>>
>>> object Main extends App {
>>>> val sc = new SparkContext("local", "myApp")
>>>> val r = sc.parallelize(1 to 1000)
>>>> println("r.filter(_ % 2 == 1).first() = " + r.filter(_ % 2 ==
>>>> 1).first())
>>>> println("r.filter(_ % 2 == 1).count() = " + r.filter(_ % 2 ==
>>>> 1).count())
>>>> }
>>>
>>>
>>> Make, Run, Works perfectly.
>>>
>>> Next, I try running the same in the scala console.
>>> Bad news - the last line throws an exception:
>>>
>>>> ERROR executor.Executor: Exception in task ID 0
>>>> java.lang.ClassNotFoundException:
>>>> $line5.$read$$iw$$iw$$iw$$iw$$anonfun$2
>>>
>>>
>>> It is my guess that for some reason Spark is not able to find the
>>> anonymous function (_ % 2 == 1). Please note I'm running locally so I did
>>> not provide any jars. For some reason when using first() instead of count()
>>> it works. Needless to say it also works in Spark Shell but as I stated,
>>> working with it is not an option.
>>>
>>> This issue brings much sadness to my heart, and I could not find a
>>> solution on the mailing list archives or elsewhere. I am hoping someone
>>> here might offer some help.
>>>
>>> Thanks,
>>> Jon
>>>
>>
>>
>
