[jira] [Commented] (SPARK-16917) Spark streaming kafka version compatibility.
[ https://issues.apache.org/jira/browse/SPARK-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418747#comment-15418747 ] Cody Koeninger commented on SPARK-16917: It sounds to me like the documentation is clear, because you have interpreted things correctly. As it says, the 0.10 version works with broker version 0.10 and higher. The 0.8 version works with broker version 0.8 and higher. There is no version specifically for 0.9, nor do I expect there ever will be. > Spark streaming kafka version compatibility. > - > > Key: SPARK-16917 > URL: https://issues.apache.org/jira/browse/SPARK-16917 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0 >Reporter: Sudev >Priority: Trivial > Labels: documentation > > It would be nice to have Kafka version compatibility information in the > official documentation. > It's very confusing now. > * If you look at this JIRA[1], it seems like Kafka is supported in Spark > 2.0.0. > * The documentation lists artifact for (Kafka 0.8) > spark-streaming-kafka-0-8_2.11 > Is Kafka 0.9 supported by Spark 2.0.0 ? > Since I'm confused here even after an hours effort googling on the same, I > think someone should help add the compatibility matrix. > [1] https://issues.apache.org/jira/browse/SPARK-12177 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16917) Spark streaming kafka version compatibility.
[ https://issues.apache.org/jira/browse/SPARK-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418384#comment-15418384 ] Alexey Zotov commented on SPARK-16917: -- [~sowen] [~c...@koeninger.org] It really seems to be confusing: 1. Security is supported in new consumer API which is implemented starting from Kafka v0.9. _spark-streaming-kafka-0-8_2.11_ does not support new consumer API. Based on that it does not look compatible with secured Kafka v0.9. 2. _spark-streaming-kafka-0-10_2.11_ works with brokers 0.10 or higher. Based on the above reasoning looks like it is impossible to use Spark Streaming for secured Kafka v0.9. Please, let me know what I have missed in the above reasoning. Thanks! > Spark streaming kafka version compatibility. > - > > Key: SPARK-16917 > URL: https://issues.apache.org/jira/browse/SPARK-16917 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0 >Reporter: Sudev >Priority: Trivial > Labels: documentation > > It would be nice to have Kafka version compatibility information in the > official documentation. > It's very confusing now. > * If you look at this JIRA[1], it seems like Kafka is supported in Spark > 2.0.0. > * The documentation lists artifact for (Kafka 0.8) > spark-streaming-kafka-0-8_2.11 > Is Kafka 0.9 supported by Spark 2.0.0 ? > Since I'm confused here even after an hours effort googling on the same, I > think someone should help add the compatibility matrix. > [1] https://issues.apache.org/jira/browse/SPARK-12177 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16917) Spark streaming kafka version compatibility.
[ https://issues.apache.org/jira/browse/SPARK-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410623#comment-15410623 ] Cody Koeninger commented on SPARK-16917: I think the doc changes I submitted make it pretty clear that spark--streaming-kafka-0-8 works with brokers 0.8 or higher, and 0-10 works with brokers 0.10 or higher > Spark streaming kafka version compatibility. > - > > Key: SPARK-16917 > URL: https://issues.apache.org/jira/browse/SPARK-16917 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0 >Reporter: Sudev >Priority: Trivial > Labels: documentation > > It would be nice to have Kafka version compatibility information in the > official documentation. > It's very confusing now. > * If you look at this JIRA[1], it seems like Kafka is supported in Spark > 2.0.0. > * The documentation lists artifact for (Kafka 0.8) > spark-streaming-kafka-0-8_2.11 > Is Kafka 0.9 supported by Spark 2.0.0 ? > Since I'm confused here even after an hours effort googling on the same, I > think someone should help add the compatibility matrix. > [1] https://issues.apache.org/jira/browse/SPARK-12177 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16917) Spark streaming kafka version compatibility.
[ https://issues.apache.org/jira/browse/SPARK-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409570#comment-15409570 ] Sean Owen commented on SPARK-16917: --- Yes I think the new "0.10" integration supports the new consumer? CC [~c...@koeninger.org] > Spark streaming kafka version compatibility. > - > > Key: SPARK-16917 > URL: https://issues.apache.org/jira/browse/SPARK-16917 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0 >Reporter: Sudev >Priority: Trivial > Labels: documentation > > It would be nice to have Kafka version compatibility information in the > official documentation. > It's very confusing now. > * If you look at this JIRA[1], it seems like Kafka is supported in Spark > 2.0.0. > * The documentation lists attifact for (Kafka 0.8) > spark-streaming-kafka-0-8_2.11 > Is Kafka 0.9 supported by Spark 2.0.0 ? > Since I'm confused here even after an hours effort googling on the same, I > think someone should help add the compatibility matrix. > [1] https://issues.apache.org/jira/browse/SPARK-12177 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
