Re: query on Spark + Flume integration using push model
Hi Akhil, thank you for your reply. Does that mean that original Spark Streaming only support Avro? If that the case then why only Avro? Is there a particular reason? The project linked is for Scala but I'm using Java. Is there another project? On 10 July 2015 at 08:46, Akhil Das wrote: > Here's an example https://github.com/przemek1990/spark-streaming > > Thanks > Best Regards > > On Thu, Jul 9, 2015 at 4:35 PM, diplomatic Guru > wrote: > >> Hello all, >> >> I'm trying to configure the flume to push data into a sink so that my >> stream job could pick up the data. My events are in JSON format, but the >> "Spark + Flume integration" [1] document only refer to Avro sink. >> >> [1] https://spark.apache.org/docs/latest/streaming-flume-integration.html >> >> I looked at some of the examples online, and they all refer to avro type: >> >> agent.sinks.avroSink.type = avro >> >> If I set the type to avro and send the data in JSON, will it work? I'm >> unable to try this because the Stream job throwing Avro >> 'org.apache.flume.source.avro.AvroFlumeEvent' exception. >> >> >> Please advice how to handle this situation. >> >> >> many thanks >> > >
Re: query on Spark + Flume integration using push model
Here's an example https://github.com/przemek1990/spark-streaming Thanks Best Regards On Thu, Jul 9, 2015 at 4:35 PM, diplomatic Guru wrote: > Hello all, > > I'm trying to configure the flume to push data into a sink so that my > stream job could pick up the data. My events are in JSON format, but the > "Spark + Flume integration" [1] document only refer to Avro sink. > > [1] https://spark.apache.org/docs/latest/streaming-flume-integration.html > > I looked at some of the examples online, and they all refer to avro type: > > agent.sinks.avroSink.type = avro > > If I set the type to avro and send the data in JSON, will it work? I'm > unable to try this because the Stream job throwing Avro > 'org.apache.flume.source.avro.AvroFlumeEvent' exception. > > > Please advice how to handle this situation. > > > many thanks >
query on Spark + Flume integration using push model
Hello all, I'm trying to configure the flume to push data into a sink so that my stream job could pick up the data. My events are in JSON format, but the "Spark + Flume integration" [1] document only refer to Avro sink. [1] https://spark.apache.org/docs/latest/streaming-flume-integration.html I looked at some of the examples online, and they all refer to avro type: agent.sinks.avroSink.type = avro If I set the type to avro and send the data in JSON, will it work? I'm unable to try this because the Stream job throwing Avro 'org.apache.flume.source.avro.AvroFlumeEvent' exception. Please advice how to handle this situation. many thanks
