I dont see the error. The twitter stuff (as well as kafka and flume stuff) are treated as "external" projects and are not included in the spark shell. This is because we dont want the dependencies of such non-core functionalities to cause random conflicts with that of core spark.
Hence its not possible to use twitter spark streaming in the spark shell. Unless you explicitly add the jar generated in external/twitter/target/scala... To the spark shells extra class path (see spark config page) TD On Thu, Jul 31, 2014 at 5:48 PM, durin <m...@simon-schaefer.net> wrote: > As suggested here > < http://apache-spark-user-list.1001560.n3.nabble.com/import-org-apache-spark-streaming-twitter-in-Shell-tp9665p9743.html > > , I want to create a minimal project using sbt to be able to use > org.apache.spark.streaming.twitter in the shell. My Spark version is the > latest Master branch compiled with maven3. > > I was following this > < http://spark.apache.org/docs/latest/quick-start.html#standalone-applications > > tutorial to create a small standalone application. > > My simple.sbt file looks like this: > > > > > However, I will get this error when using sbt package: > > > > > I don't quite understand the whole standalone thing yet. sbt is installed on > a different machine than my Spark Cluster, but the tutorial never mentions > any variables that would have to be set. So I did assume sbt didn't actually > need a Spark installation on the machine. > Also, I could imagine that 1.0.1 isn't the correct version when I compiled > it myself. > > > Can someone help me out? In the end, all I want for now is to be able to use > twitterUtils in the spark-shell. > > > Simon > > > > -- > View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.
