Sergey,
you might actually consider using two streams, like
* val stream1 = KafkaUtils.createStream(ssc,"localhost:2181","logs",
Map("retarget" -> 2))*
* val stream2 = KafkaUtils.createStream(ssc,"localhost:2181","logs",
Map("datapair" -> 2))*
to achieve what you want. This has the additional advantage that there are
actually two connections to Kafka and data is possibly received on
different cluster nodes, already increasing parallelity in an early stage
of processing.
Tobias
On Thu, Jul 3, 2014 at 6:47 AM, Sergey Malov <sma...@collective.com> wrote:
> HI,
> I would like to set up streaming from Kafka cluster, reading multiple
> topics and then processing each of the differently.
> So, I’d create a stream
>
> * val stream = KafkaUtils.createStream(ssc,"localhost:2181","logs",
> Map("retarget" -> 2,"datapair" -> 2))*
>
> And then based on whether it’s “retarget” topic or “datapair”, set up
> different filter function, map function, reduce function, etc. Is it
> possible ? I’d assume it should be, since ConsumerConnector can map of
> KafkaStreams keyed on topic, but I can’t find that it would be visible to
> Spark.
>
> Thank you,
>
> Sergey Malov
>
>
