[
https://issues.apache.org/jira/browse/FLINK-9700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556223#comment-16556223
]
Jonathan Miles commented on FLINK-9700:
---------------------------------------
Duplicating the comment I added to FLINK-9690, since this is worth documenting
as well...
Although the protocol is backwards compatible, there is a performance penalty
when using the 0.11 consumer and producer with later Kafka brokers. The
protocol was changed at some point and using older connectors requires the
brokers to do conversions between the on-disk and wire protocols, so we lose
the zero-copy functionality and add extra GC burden among other issues.
There's a note about this in the Kafka documentation. Although that's
discussing conversions between different version numbers than we are, the
effects are the same:
https://kafka.apache.org/0102/documentation.html#upgrade_10_performance_impact
> Document FlinkKafkaProducer behaviour for Kafka versions > 0.11
> ---------------------------------------------------------------
>
> Key: FLINK-9700
> URL: https://issues.apache.org/jira/browse/FLINK-9700
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
> Affects Versions: 1.5.0
> Reporter: Ufuk Celebi
> Assignee: vinoyang
> Priority: Minor
>
> FlinkKafkaProducer for Kafka 0.11 uses reflection to work around API
> limitations of the Kafka client. Using reflection breaks with newer versions
> of the Kafka client (due to internal changes of the client).
> The documentation does not mention newer Kafka versions. We should add the
> following notes:
> - Only package Kafka connector with kafka.version property set to 0.11.*.*
> - Mention that it is possible to use the 0.11 connector with newer versions
> of Kafka as the protocol seems to be backwards compatible (double check that
> this is correct)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
