Problems with Spark Core 1.2.0 SBT project in IntelliJ

[Night Wolf]

Hi,

I'm trying to load up an SBT project in IntelliJ 14 (windows) running 1.7
JDK, SBT 0.13.5 -I seem to be getting errors with the project.

The build.sbt file is super simple;

name := "scala-spark-test1"
version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"


Then I have a super simple test class;

package test

import org.apache.spark.{SparkContext, SparkConf}

case class Blah(s: Int, d: String)

object Test1  {
  def main(args: Array[String]): Unit = {
    val sparkconf = new
SparkConf().setMaster("local[4]").setAppName("test-spark")
    val sc = new SparkContext(sparkconf)

    val rdd = sc.parallelize(Seq(
      Blah(1,"dsdsd"),
      Blah(2,"daaa"),
      Blah(3,"dhghghgh")
    ))

    rdd.collect().foreach(println)

  }
}


When I try to run the Test1 object in IntelliJ I get the following error;

Exception in thread "main" java.lang.NoClassDefFoundError:
javax/servlet/http/HttpServletResponse
at org.apache.spark.HttpServer.org
$apache$spark$HttpServer$$doStart(HttpServer.scala:73)
at org.apache.spark.HttpServer$$anonfun$1.apply(HttpServer.scala:60)
at org.apache.spark.HttpServer$$anonfun$1.apply(HttpServer.scala:60)
at
org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1676)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1667)
at org.apache.spark.HttpServer.start(HttpServer.scala:60)
at org.apache.spark.HttpFileServer.initialize(HttpFileServer.scala:45)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:304)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:159)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:232)
at test.Test1$.main(Test1.scala:10)
at test.Test1.main(Test1.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: java.lang.ClassNotFoundException:
javax.servlet.http.HttpServletResponse
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 18 more


For whatever reason it seems that IntelliJ isnt pulling in these deps.
Doing an sbt run works fine. Looking at the project structure it seems that
7 libs dont get marked as a dependency for my module... But they are on the
dep tree.... http://pastebin.com/REkQh5ux

[image: Inline image 1]

Is this something to do with the libs and scoping or shading in Spark and
its associated libs? Has anyone else seen this issue?

Cheers,
NW