Hi folks, I wanted to double check the approach to the next release of the Flink Connector for Kafka [1]. Currently, there is no release that supports Flink 2.0.0. Also, the Kafka version used in the current 3.4 connector is quite old and has several low and medium severity CVEs.
The main branch of the connector is currently on version 4.0-SNAPSHOT and is now running the 3.9.0 Kafka client libraries [2]. However, Kafka 4.0.0 is now out and there is a PR [3] available for that. The Kafka 4.0.0 client libraries drop support for Kafka brokers running version 2.0 or older, so moving to these libraries would represent a breaking change and should probably come with a new major version of the connector? So there are a few options and (being new to the community) I wondered what the approach would be: 1) Release Kafka Connector 4.0 with Flink 2.0.0 and Kafka 4.0.0. 2) Release Kafka Connector 4.0 with Flink 2.0.0 and Kafka 3.9.0. Then do a Kafka Connector 4.1 release with Kafka 4.0.0, but as mentioned above would this actually need a 5.0 release of the connector as it drops support for older Kafka versions? 3) Same as option 2 but downgrade to the more stable Kafka 3.8.1 client libraries so we can get a release out ASAP. 4) Same as option 2 but wait for the Kafka 3.9.1 bug fix release at the end of this month. As well as Kafka Connector support for Flink 2.0.0 users, there is also the issue of users of Flink 1.x. I think it would be good to have an updated Kafka connector with Flink 1.20.1 and Kafka 3.9.0/1 (to fix CVEs and get performance improvements for the Flink LTS). Would it make sense to do a Connector 3.5 release with a more up to date Kafka version? I also think it would be a good idea to codify (add to the README or similar) what our Kafka client library update approach is for the connector. Do we want to always rack the x.x.1 version of the Kafka clients so we are on a solid base, or do we want to update ASAP and have a x.1 releases of the Kafka Connector which tracks the more stable clients? Thanks, Tom Cooper @tomcooper.dev | https://tomcooper.dev [1] https://github.com/apache/flink-connector-kafka [2] https://github.com/apache/flink-connector-kafka/pull/138 [3] https://github.com/apache/flink-connector-kafka/pull/161
