All the stuff in lib_managed are what gets downloaded by sbt/maven when you compile. Those are necessary for running spark, spark streaming, etc. But you should not have to add all that to classpath individually and manually when running Spark programs. If you are trying to run your Spark program locally, you should use sbt or maven to compile your project with Spark as a dependency, and sbt/maven will take care of putting all the necessary jars in the classpath (when you run run your program with sbt/maven). If you are trying to run your Spark program on a cluster, then refer to the deployment guide <http://spark.apache.org/docs/latest/cluster-overview.html>. To run a Spark stand alone cluster, you just have to compile spark and place the whole spark directory on the worker nodes. For other deploy modes like Yarn and Mesos, you should just compile spark into a big all-inclusive jar and supply that when you launch your program on Yarn/mesos. See the guide for more details.
TD On Sat, May 3, 2014 at 7:24 PM, Weide Zhang <[email protected]> wrote: > Hi Tathagata, > > I actually have a separate question. What's the usage of lib_managed > folder inside spark source folder ? Are those the library required for > spark streaming to run ? Do they needed to be added to spark classpath when > starting sparking cluster? > > Weide > > > On Sat, May 3, 2014 at 7:08 PM, Weide Zhang <[email protected]> wrote: > >> Hi Tathagata, >> >> I figured out the reason. I was adding a wrong kafka lib along side with >> the version spark uses. Sorry for spamming. >> >> Weide >> >> >> On Sat, May 3, 2014 at 7:04 PM, Tathagata Das < >> [email protected]> wrote: >> >>> I am a little confused about the version of Spark you are using. Are you >>> using Spark 0.9.1 that uses scala 2.10.3 ? >>> >>> TD >>> >>> >>> On Sat, May 3, 2014 at 6:16 PM, Weide Zhang <[email protected]> wrote: >>> >>>> Hi I'm trying to run the kafka-word-count example in spark2.9.1. I >>>> encountered some exception when initialize kafka consumer/producer config. >>>> I'm using scala 2.10.3 and used maven build inside spark streaming kafka >>>> library comes with spark2.9.1. Any one see this exception before? >>>> >>>> Thanks, >>>> >>>> producer: >>>> [java] The args attribute is deprecated. Please use nested arg >>>> elements. >>>> [java] Exception in thread "main" java.lang.NoClassDefFoundError: >>>> scala/Tuple2$mcLL$sp >>>> [java] at >>>> kafka.producer.ProducerConfig.<init>(ProducerConfig.scala:56) >>>> [java] at >>>> com.turn.apache.KafkaWordCountProducer$.main(HelloWorld.scala:89) >>>> [java] at >>>> com.turn.apache.KafkaWordCountProducer.main(HelloWorld.scala) >>>> [java] Caused by: java.lang.ClassNotFoundException: >>>> scala.Tuple2$mcLL$sp >>>> [java] at >>>> java.net.URLClassLoader$1.run(URLClassLoader.java:202) >>>> [java] at java.security.AccessController.doPrivileged(Native >>>> Method) >>>> [java] at >>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190) >>>> [java] at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >>>> [java] at >>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>>> [java] at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >>>> [java] ... 3 more >>>> [java] Java Result: 1 >>>> >>>> >>> >> >
