;
>>>> On Thu, Dec 17, 2015 at 11:43 AM, Cody Koeninger <c...@koeninger.org>
>>>> wrote:
>>>>
>>>>> Using spark.streaming.concurrentJobs for this probably isn't a good
>>>>> idea, as it allows the next batch to start processing befo
t;>> I might end up running 3 different streaming apps instead of one.
>>>>>
>>>>> Thanks again!
>>>>>
>>>>> On Thu, Dec 17, 2015 at 11:43 AM, Cody Koeninger <c...@koeninger.org>
>>>>> wrote:
>>>>>
>>>
gt;>>>> them
>>>>>> would be really efficient, unless you know something that I don't.
>>>>>>
>>>>>> As I really don't need any interaction between those streams, I think
treams if you're e.g. joining them?
>>>>
>>>>
>>>>
>>>>> On Wed, Dec 16, 2015 at 3:00 PM, jpocalan <jpoca...@gmail.com> wrote:
>>>>> Nevermind, I found the answer to my questions.
>>>>&g
treams if you're e.g. joining them?
>>>>
>>>>
>>>>
>>>> On Wed, Dec 16, 2015 at 3:00 PM, jpocalan <jpoca...@gmail.com> wrote:
>>>>
>>>>> Nevermind, I found the answer to my questions.
>>>>> The following
;>
>>>
>>> On Wed, Dec 16, 2015 at 3:00 PM, jpocalan <jpoca...@gmail.com> wrote:
>>>
>>>> Nevermind, I found the answer to my questions.
>>>> The following spark configuration property will allow you to process
>>>> multiple KafkaDirectSt
ing.concurrentJobs=
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-topics-tp8678p25723.html
> Sent from the Apache Spark
cess
>> multiple KafkaDirectStream in parallel:
>> --conf spark.streaming.concurrentJobs=
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-stream
e answer to my questions.
>>> The following spark configuration property will allow you to process
>>> multiple KafkaDirectStream in parallel:
>>> --conf spark.streaming.concurrentJobs=
>>>
>>>
>>>
>>>
>>>
>>> -
-streaming-from-multiple-topics-tp8678p25723.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h
offset range.
>>
>> On Wed, Dec 2, 2015 at 2:13 PM, dutrow <dan.dut...@gmail.com> wrote:
>>
>>> I found the JIRA ticket:
>>> https://issues.apache.org/jira/browse/SPARK-2388
>>>
>>> It was marked as invalid.
>>>
>>>
>>
M, dutrow <dan.dut...@gmail.com> wrote:
>
>> I found the JIRA ticket: https://issues.apache.org/jira/browse/SPARK-2388
>>
>> It was marked as invalid.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.10015
PARK-2388
>
> It was marked as invalid.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-topics-tp8678p25550.html
> Sent from the Apache Spark User Lis
this issue?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-topics-tp8678p25549.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
sues.apache.org/jira/browse/SPARK-2388
>>
>> It was marked as invalid.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-topics-tp8678p2555
I found the JIRA ticket: https://issues.apache.org/jira/browse/SPARK-2388
It was marked as invalid.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-topics-tp8678p25550.html
Sent from the Apache Spark User List mailing list
Can you link to the JIRA issue? I'm having to work around this bug and it
would be nice to monitor the JIRA so I can change my code when it's fixed.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-streaming-from-multiple-topics-tp8678p12053.html
Sent
: Thursday, July 3, 2014 at 9:41 PM
To: user@spark.apache.orgmailto:user@spark.apache.org
user@spark.apache.orgmailto:user@spark.apache.org
Subject: Re: Kafka - streaming from multiple topics
Sergey,
On Fri, Jul 4, 2014 at 1:06 AM, Sergey Malov
sma...@collective.commailto:sma...@collective.com
: Re: Kafka - streaming from multiple topics
Sergey,
you might actually consider using two streams, like
val stream1 = KafkaUtils.createStream(ssc,localhost:2181,logs,
Map(retarget - 2))
val stream2 = KafkaUtils.createStream(ssc,localhost:2181,logs,
Map(datapair - 2))
to achieve what
Sergey,
On Fri, Jul 4, 2014 at 1:06 AM, Sergey Malov sma...@collective.com wrote:
On the other hand, under the hood KafkaInputDStream which is create with
this KafkaUtils call, calls ConsumerConnector.createMessageStream which
returns a Map[String, List[KafkaStream] keyed by topic. It is,
HI,
I would like to set up streaming from Kafka cluster, reading multiple topics
and then processing each of the differently.
So, I’d create a stream
val stream = KafkaUtils.createStream(ssc,localhost:2181,logs,
Map(retarget - 2,datapair - 2))
And then based on whether it’s “retarget”
21 matches
Mail list logo
