$ jar tvf ./external/flume-sink/target/spark-streaming-flume-sink_2.10-1.6.1.jar | grep SparkFlumeProtocol 841 Thu Mar 03 11:09:36 PST 2016 org/apache/spark/streaming/flume/sink/SparkFlumeProtocol$Callback.class 2363 Thu Mar 03 11:09:36 PST 2016 org/apache/spark/streaming/flume/sink/SparkFlumeProtocol.class
On Sun, Mar 20, 2016 at 9:22 AM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > Hi, > I'm trying to use the Spark Sink with Flume but it seems I'm missing some > of the dependencies. > I'm running the following code: > > ./bin/spark-shell --master yarn --jars > /home/impact/flumeStreaming/spark-streaming-flume_2.10-1.6.1.jar,/home/impact/flumeStreaming/flume-ng-core-1.6.0.jar,/home/impact/flumeStreaming/flume-ng-sdk-1.6.0.jar > > > import org.apache.spark.streaming.flume._ > > import org.apache.spark.streaming._ > > val ssc = new StreamingContext(sc, Seconds(60)) > val flumeStream = FlumeUtils.createPollingStream(ssc, "impact1", 9999) > > flumeStream.print > ssc.start > > > And getting this execption. > > 16/03/20 18:17:17 INFO scheduler.ReceiverTracker: Registered receiver for > stream 0 from impact3.indigo.co.il:51581 > 16/03/20 18:17:17 WARN scheduler.TaskSetManager: Lost task 0.0 in stage > 4.0 (TID 76, impact3.indigo.co.il): java.lang.NoClassDefFoundError: > org/apache/spark/streaming/flume/sink/SparkFlumeProtocol$Callback > at > org.apache.spark.streaming.flume.FlumePollingReceiver$$anonfun$onStart$1.apply(FlumePollingInputDStream.scala:84) > > > What deps am I missing ? > > Thank you. > Daniel >
