Re: Flume and Spark Streaming

2017-01-16 Thread Guillermo Ortiz
Avro sink --> Spark Streaming 2017-01-16 13:55 GMT+01:00 ayan guha : > With Flume, what would be your sink? > > > > On Mon, Jan 16, 2017 at 10:44 PM, Guillermo Ortiz > wrote: > >> I'm wondering to use Flume (channel file)-Spark Streaming. >> >> I have

Re: Flume and Spark Streaming

2017-01-16 Thread ayan guha
With Flume, what would be your sink? On Mon, Jan 16, 2017 at 10:44 PM, Guillermo Ortiz wrote: > I'm wondering to use Flume (channel file)-Spark Streaming. > > I have some doubts about it: > > 1.The RDD size is all data what it comes in a microbatch which you have >

Re: Flume with Spark Streaming Sink

2016-03-20 Thread Luciano Resende
You should use it as described in the documentation and passing it as a package: ./bin/spark-submit --packages org.apache.spark:spark-streaming-flume_2.10:1.6.1 ... On Sun, Mar 20, 2016 at 9:22 AM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > Hi, > I'm trying to use the Spark

Re: Flume with Spark Streaming Sink

2016-03-20 Thread Ted Yu
$ jar tvf ./external/flume-sink/target/spark-streaming-flume-sink_2.10-1.6.1.jar | grep SparkFlumeProtocol 841 Thu Mar 03 11:09:36 PST 2016 org/apache/spark/streaming/flume/sink/SparkFlumeProtocol$Callback.class 2363 Thu Mar 03 11:09:36 PST 2016