Re: Should Flume integration be behind a profile?

27;m raising it again to >> the >> >> whole list.) >> >> >> >> Any preferences among: >> >> 1. Put Flume behind a profile, remove examples, deprecate >> >> 2. Put Flume behind a profile, remove examples, but don't deprecate >> >&g

Re: Flume integration

Hi Mich, Thanks. I would prefer not to add another system into the mix as we currently don't use kafka at all. We are still in the prototype phase at the moment and it seems to be working well though it doesn't like you restrating the flume sink part without restarting the SPARK application. Th

Re: Flume integration

Hi Ian, Flume is great for ingesting data into HDFS and Hbase. However, that is part of batch layer. For real time processing, I would go through Kafka into spark streaming. Except your case, I have not established if anyone else does Flume directly into Spark? If so how mature is it. Thanks D

Re: Flume integration

*-Ian* Hi While I am following this discussion with interest, I am trying to comprehend any architectural benefit of a spark sink. Is there any feature in flume makes it more suitable to ingest stream data than sppark streaming, so that we should chain them? For example does it help durabilit

Re: Flume integration

Hi While I am following this discussion with interest, I am trying to comprehend any architectural benefit of a spark sink. Is there any feature in flume makes it more suitable to ingest stream data than sppark streaming, so that we should chain them? For example does it help durability or reliab

Re: Flume integration

Thanks Ian. Was your source of Flume IBM/MQ by any chance? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.co

Re: Flume integration

Hi Mich, Yes, i managed to resolve this one. The issue was because the way described in the docs doesn't work properly as in order for the Flume part to be notified you need to set the storageLevel on the PollingStream like JavaReceiverInputDStream flumeStream = FlumeUtils.createPoll

Re: Flume integration

Hi Ian, Has this been resolved? How about data to Flume and then Kafka and Kafka streaming into Spark? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Flume integration

Hi, I'm currently trying to implement a prototype Spark application that gets data from Flume and processes it. I'm using the pull based method mentioned in https://spark.apache.org/docs/1.6.1/streaming-flume-integration.html The is initially working fine for getting data from Flume, however t

Re: query on Spark + Flume integration using push model

my >> stream job could pick up the data. My events are in JSON format, but the >> "Spark + Flume integration" [1] document only refer to Avro sink. >> >> [1] https://spark.apache.org/docs/latest/streaming-flume-integration.html >> >> I looked at some of the

Re: query on Spark + Flume integration using push model

nts are in JSON format, but the > "Spark + Flume integration" [1] document only refer to Avro sink. > > [1] https://spark.apache.org/docs/latest/streaming-flume-integration.html > > I looked at some of the examples online, and they all refer to avro type: > > agent.sin

query on Spark + Flume integration using push model

Hello all, I'm trying to configure the flume to push data into a sink so that my stream job could pick up the data. My events are in JSON format, but the "Spark + Flume integration" [1] document only refer to Avro sink. [1] https://spark.apache.org/docs/latest/streaming-flume-inte

Re: Spark and Flume integration - do I understand this correctly?

Great, thanks guys, that helped a lot and I've got a sample working. As a follow up, when do worker/masters become necessity? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-and-Flume-integration-do-I-understand-this-correctly-tp10879p10908.html

Re: Spark and Flume integration - do I understand this correctly?

annel thank you in advance for any assistance, -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-and-Flume-integration-do-I-understand-this-correctly-tp10879.html Sent from the Apache Spark User List mailing list archive at Nabble.com. Tathagat

Re: Spark and Flume integration - do I understand this correctly?

> > #Sinks > agent1.sinks = localout > > #http://flume.apache.org/FlumeUserGuide.html#file-roll-sink > agent1.sinks.localout.type = file_roll > agent1.sinks.localout.sink.directory = /home/vagrant/flume/logs > agent1.sinks.localout.sink.rollInterval = 0 > agent1.sinks.l

Spark and Flume integration - do I understand this correctly?

://apache-spark-user-list.1001560.n3.nabble.com/Spark-and-Flume-integration-do-I-understand-this-correctly-tp10879.html Sent from the Apache Spark User List mailing list archive at Nabble.com.