[
https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982077#comment-15982077
]
Ismael Juma edited comment on SPARK-18057 at 4/24/17 11:07 PM:
---------------------------------------------------------------
Hi. A few clarifications below.
"Based on previous kafka client upgrades I wouldn't expect them to be binary
compatible, so it's likely to cause someone problems if they were also making
use of kafka client libraries in their spark job. Still may be the path of
least resistance."
We do strive for binary compatibility for APIs that are not marked as Unstable.
The Java consumer was introduced in 0.9.0.0 and the APIs were marked as
Unstable. There were incompatible changes between 0.9.0.0 and 0.10.0.0
(KIP-45), which is what the above comment is probably referring to, but no
other incompatible changes after that.
"For what it's worth, or not, I ran into a wire protocol incompatibility
between 0.10.0.1 and 0.10.1.1 today. I suspect we'd find the same vs 0.10.2.0.
It surprised me."
As Michael has said, 0.10.2 clients are the first version that supports older
brokers (0.10.0 and higher). Before 0.10.2, clients supported newer brokers
(e.g. 0.8.2 clients support 0.10.2 brokers), but not older brokers. I hope that
helps.
was (Author: ijuma):
Hi. A few clarifications below.
"Based on previous kafka client upgrades I wouldn't expect them to be binary
compatible, so it's likely to cause someone problems if they were also making
use of kafka client libraries in their spark job. Still may be the path of
least resistance."
We do strive for binary compatibility for APIs that are not marked as Unstable.
The Java consumer was introduced in 0.9.0.0 and the APIs were marked as
Unstable. There were incompatible changes between 0.9.0.0 and 0.10.0.0
(KIP-45), which is what the above comment is probably referring to, but no
other incompatible changes after that.
"For what it's worth, or not, I ran into a wire protocol incompatibility
between 0.10.0.1 and 0.10.1.1 today. I suspect we'd find the same vs 0.10.2.0.
It surprised me."
As Michael has said, 0.10.2 clients are the first version that supports older
brokers (0.10.0 and higher). Before 0.10.2, clients supported newer brokers
(e.g. 0.8.2 clients support 0.10.2), but not older brokers. I hope that helps.
> Update structured streaming kafka from 10.0.1 to 10.2.0
> -------------------------------------------------------
>
> Key: SPARK-18057
> URL: https://issues.apache.org/jira/browse/SPARK-18057
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Reporter: Cody Koeninger
>
> There are a couple of relevant KIPs here,
> https://archive.apache.org/dist/kafka/0.10.1.0/RELEASE_NOTES.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
