[
https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274910#comment-15274910
]
Mark Grover commented on SPARK-12177:
-------------------------------------
I spent some time earlier today looking at the latest Kafka 0.10 RC. Thanks
Cody, looks like you poked at it too.
As Cody found out, the new kafka consumer API is in flux. I found and filed a
PR (KAFKA-3669) for an incompatibility in KafkaConfig, which is now fixed in
Kafka 0.10. But, there's more - KAFKA-3633 is of note - which discusses fixing
a compatibility break between 0.9 and 0.10 and remains unresolved.
So, at this point, I can say, that there's no point in committing the PR
associated with this JIRA (https://github.com/apache/spark/pull/11863), until
at least Kafka 0.10.0 is released and we have a good sense that Kafka 0.11.0 is
not going to break compatibility with 0.10.0. Otherwise, we have to bear the
burden of adding and maintaining complexity to build against multiple versions
of Kafka, something Storm folks are already suffering from, having their
KafkaSpout now using the new Kafka consumer API from 0.9.
Given the timing of the Kafka 0.10.0 release and Spark 2 release, this JIRA
likely wouldn't get resolved in 2.0.
> Update KafkaDStreams to new Kafka 0.9 Consumer API
> --------------------------------------------------
>
> Key: SPARK-12177
> URL: https://issues.apache.org/jira/browse/SPARK-12177
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.6.0
> Reporter: Nikita Tarasenko
> Labels: consumer, kafka
>
> Kafka 0.9 already released and it introduce new consumer API that not
> compatible with old one. So, I added new consumer api. I made separate
> classes in package org.apache.spark.streaming.kafka.v09 with changed API. I
> didn't remove old classes for more backward compatibility. User will not need
> to change his old spark applications when he uprgade to new Spark version.
> Please rewiew my changes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
