Hi All, Now that the Spark Version 1.0.0 is release there should not be any problem with the local jars. Shrikars-MacBook-Pro:SimpleJob shrikar$ cat simple.sbt name := "Simple Project"
version := "1.0" scalaVersion := "2.10.4" libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % "1.0.0", "org.apache.spark" %% "spark-streaming" % "1.0.0") resolvers += "Akka Repository" at "http://repo.akka.io/releases/" I am still having this issue [error] (run-main) java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse at org.apache.spark.HttpServer.start(HttpServer.scala:54) at org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156) at org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127) at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31) at org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48) at org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218) at org.apache.spark.SparkContext.<init>(SparkContext.scala:202) Any help would be greatly appreciated. Thanks, Shrikar On Fri, May 23, 2014 at 3:58 PM, Shrikar archak <shrika...@gmail.com> wrote: > Still the same error no change > > Thanks, > Shrikar > > > On Fri, May 23, 2014 at 2:38 PM, Jacek Laskowski <ja...@japila.pl> wrote: > >> Hi Shrikar, >> >> How did you build Spark 1.0.0-SNAPSHOT on your machine? My >> understanding is that `sbt publishLocal` is not enough and you really >> need `sbt assembly` instead. Give it a try and report back. >> >> As to your build.sbt, upgrade Scala to 2.10.4 and "org.apache.spark" >> %% "spark-streaming" % "1.0.0-SNAPSHOT" only that will pull down >> spark-core as a transitive dep. The resolver for Akka Repository is >> not needed. Your build.sbt should really look as follows: >> >> name := "Simple Project" >> >> version := "1.0" >> >> scalaVersion := "2.10.4" >> >> libraryDependencies += "org.apache.spark" %% "spark-streaming" % >> "1.0.0-SNAPSHOT" >> >> Jacek >> >> On Thu, May 22, 2014 at 11:27 PM, Shrikar archak <shrika...@gmail.com> >> wrote: >> > Hi All, >> > >> > I am trying to run the network count example as a seperate standalone >> job >> > and running into some issues. >> > >> > Environment: >> > 1) Mac Mavericks >> > 2) Latest spark repo from Github. >> > >> > >> > I have a structure like this >> > >> > Shrikars-MacBook-Pro:SimpleJob shrikar$ find . >> > . >> > ./simple.sbt >> > ./src >> > ./src/main >> > ./src/main/scala >> > ./src/main/scala/NetworkWordCount.scala >> > ./src/main/scala/SimpleApp.scala.bk >> > >> > >> > simple.sbt >> > name := "Simple Project" >> > >> > version := "1.0" >> > >> > scalaVersion := "2.10.3" >> > >> > libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % >> > "1.0.0-SNAPSHOT", >> > "org.apache.spark" %% "spark-streaming" % >> > "1.0.0-SNAPSHOT") >> > >> > resolvers += "Akka Repository" at "http://repo.akka.io/releases/" >> > >> > >> > I am able to run the SimpleApp which is mentioned in the doc but when I >> try >> > to run the NetworkWordCount app I get error like this am I missing >> > something? >> > >> > [info] Running com.shrikar.sparkapps.NetworkWordCount >> > 14/05/22 14:26:47 INFO spark.SecurityManager: Changing view acls to: >> shrikar >> > 14/05/22 14:26:47 INFO spark.SecurityManager: SecurityManager: >> > authentication disabled; ui acls disabled; users with view permissions: >> > Set(shrikar) >> > 14/05/22 14:26:48 INFO slf4j.Slf4jLogger: Slf4jLogger started >> > 14/05/22 14:26:48 INFO Remoting: Starting remoting >> > 14/05/22 14:26:48 INFO Remoting: Remoting started; listening on >> addresses >> > :[akka.tcp://spark@192.168.10.88:49963] >> > 14/05/22 14:26:48 INFO Remoting: Remoting now listens on addresses: >> > [akka.tcp://spark@192.168.10.88:49963] >> > 14/05/22 14:26:48 INFO spark.SparkEnv: Registering MapOutputTracker >> > 14/05/22 14:26:48 INFO spark.SparkEnv: Registering BlockManagerMaster >> > 14/05/22 14:26:48 INFO storage.DiskBlockManager: Created local >> directory at >> > >> /var/folders/r2/mbj08pb55n5d_9p8588xk5b00000gn/T/spark-local-20140522142648-0a14 >> > 14/05/22 14:26:48 INFO storage.MemoryStore: MemoryStore started with >> > capacity 911.6 MB. >> > 14/05/22 14:26:48 INFO network.ConnectionManager: Bound socket to port >> 49964 >> > with id = ConnectionManagerId(192.168.10.88,49964) >> > 14/05/22 14:26:48 INFO storage.BlockManagerMaster: Trying to register >> > BlockManager >> > 14/05/22 14:26:48 INFO storage.BlockManagerInfo: Registering block >> manager >> > 192.168.10.88:49964 with 911.6 MB RAM >> > 14/05/22 14:26:48 INFO storage.BlockManagerMaster: Registered >> BlockManager >> > 14/05/22 14:26:48 INFO spark.HttpServer: Starting HTTP Server >> > [error] (run-main) java.lang.NoClassDefFoundError: >> > javax/servlet/http/HttpServletResponse >> > java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse >> > at org.apache.spark.HttpServer.start(HttpServer.scala:54) >> > at >> > >> org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156) >> > at >> > >> org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127) >> > at >> > >> org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31) >> > at >> > >> org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48) >> > at >> > >> org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35) >> > at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218) >> > at org.apache.spark.SparkContext.<init>(SparkContext.scala:202) >> > at >> > >> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:549) >> > at >> > >> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:561) >> > at >> > >> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:91) >> > at >> com.shrikar.sparkapps.NetworkWordCount$.main(NetworkWordCount.scala:39) >> > at com.shrikar.sparkapps.NetworkWordCount.main(NetworkWordCount.scala) >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > at >> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> > at >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> > at java.lang.reflect.Method.invoke(Method.java:597) >> > >> > >> > Thanks, >> > Shrikar >> > >> >> >> >> -- >> Jacek Laskowski | http://blog.japila.pl >> "Never discourage anyone who continually makes progress, no matter how >> slow." Plato >> > >