[
https://issues.apache.org/jira/browse/BEAM-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513629#comment-15513629
]
ASF GitHub Bot commented on BEAM-549:
-------------------------------------
Github user amitsela closed the pull request at:
https://github.com/apache/incubator-beam/pull/822
> SparkRunner should support Beam's KafkaIO instead of providing it's own.
> ------------------------------------------------------------------------
>
> Key: BEAM-549
> URL: https://issues.apache.org/jira/browse/BEAM-549
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
> Reporter: Amit Sela
> Assignee: Amit Sela
>
> For portability, and in the spirit of Apache Beam, the SparkRunner should use
> the Beam implementation of KafkaIO instead of it's own.
> Having said that, the runner will translate the KafkaIO as defined in the
> pipeline into it's own internal implementation, but should still map the
> properties the user defined in the pipeline in a way that the IO behaves the
> same - i.e., brokers, topic, etc.
> Eventually, the SparkRunner will implement reading from Kafka using Spark's
> KafakUtils.createDirectStream() as described here:
> http://spark.apache.org/docs/1.6.2/streaming-kafka-integration.html#approach-2-direct-approach-no-receivers
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
