[
https://issues.apache.org/jira/browse/SPARK-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14077488#comment-14077488
]
Silver commented on SPARK-2388:
-------------------------------
Cant you use the overload of Kafka's KeyedMessage(topic : String, KEY : String,
data : String)? The key is passed on to Spark nicely.
> Streaming from multiple different Kafka topics is problematic
> -------------------------------------------------------------
>
> Key: SPARK-2388
> URL: https://issues.apache.org/jira/browse/SPARK-2388
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.0.0
> Reporter: Sergey
> Fix For: 1.0.1
>
>
> Default way of creating stream out of Kafka source would be as
> val stream = KafkaUtils.createStream(ssc,"localhost:2181","logs",
> Map("retarget" -> 2,"datapair" -> 2))
> However, if two topics - in this case "retarget" and "datapair" - are very
> different, there is no way to set up different filter, mapping functions,
> etc), as they are effectively merged.
> However, instance of KafkaInputDStream, created with this call internally
> calls ConsumerConnector.createMessageStream() which returns *map* of
> KafkaStreams, keyed by topic. It would be great if this map would be exposed
> somehow, so aforementioned call
> val streamS = KafkaUtils.createStreamS(...)
> returned map of streams.
> Regards,
> Sergey Malov
> Collective Media
--
This message was sent by Atlassian JIRA
(v6.2#6252)
