Hi Tom, thanks for driving this.
Updating the Kafka client library has been done only occasionally and there was no formal process. It makes sense to give stronger guidelines. I guess a rigid process would only make sense if we enforce it for example through bots and that was highly debated a couple of years ago in Apache projects (I would be very much in favor of getting bot support for dependencies). Here is my proposal: * Kafka-connector-4.0.0 uses kafka-client-3.9.0 because it's the latest minor release in the 3.X. I went over the things that they (plan to) fix in 3.9.1 and nothing major pops out. 3.9.0 gives folks the option to relatively effortless downgrade to 3.8.1 in DataStreams (it's just a transitive dependency). For SQL, folks are bound to 3.9.0 unless they rebuild the module from scratch, which is also a moderate effort. No code change is required. * Kafka-connector-3.5.0 with a kafka-client-3.9.0 for Flink 1.X also makes a lot of sense if Kafka-connector-4.0.0 cannot run on Flink 1.X (to confirm). Else, folks can just use Kafka-connector-4.X. * Release Kafka-connector-4.0.1 once kafka-client-3.9.1 is out + any bug fix that is needed in our code. * Release Kafka-connector-4.1.0 once kafka-client-4.0.1 or 4.1.0 is out. In my experience, the very first version of a major release is usually for early adopters only. Most of the new features in that release are only available once you also update brokers to 4.0.0, which will probably take a longer time for most setups. Of course, the effort can be expedited if there is actual demand for kafka-client-4.0.0. As Yanquan mentioned, the biggest need is currently on Kafka-connector-4.0.0. We need to find a release manager asap and get that release out next week (so probably start this week with cut and voting). We need to bundle our resources on that. Once that is done, we should look at Kafka-connector-3.5.0 release. Then, the others. Note that I don't see the need for another major release to bump kafka-client to 4.0.0. From the user's perspective, this change is mostly transparent. I think that release basically just cuts off some very old broker versions <3.0. I'm assuming most folks are on 3.5+ atm. In contrast, for kafka-connector-4.0.0, Yanquan deleted 22k LOC to drop all the legacy consumer/producer. LMK what you think about that proposal. Arvid On Tue, Apr 1, 2025 at 6:10 PM Yanquan Lv <decq12y...@gmail.com> wrote: > > Hi, Cooper. Thank you for bringing up the discussion about Kafka 4.0.0. > I tend to release Kafka Connector 4.0 with Flink 2.0.0 and Kafka 3.9.0, or > Kafka 3.8.1 if If the community is concerned about 3.9.0. > > Here are some of my opinions: > > 1. The plan for Kafka Connector 4.0.0 was proposed[1] a long time ago, mainly > to support the Flink 2.0 API. Although it has been delayed for some time, > users' demand for this version is becoming increasingly strong (as we can see > from emails or Slack), so it is more important to release a version that > supports Flink 2.0 as soon as possible. > > 2. For Bump to Kafka 4.0.0, I personally believe that this is a decision that > requires careful analysis, because rolling back to Kafka client version 3. x > will require additional workload, and the impact on performance and API > compatibility may not be reflected solely through IT cases(Of course, I > believe it will have a positive impact). > I don't recommend making two major changes in one version,So perhaps 4.1/5.0 > is more suitable for doing this. > > 3. As for Kafka 3.9.0, I don't have a big opinion as we have been using Kafka > client 3.4.0 for almost two years since FLINK-31599[2]. Of course, I also > approve of using Kafka 3.8.1. > But for using 3.9.1, the benefits are not very significant and it will block > the activities of other developers who rely on Flink connector Kafka release > 4.0. As I know, some e2e in Flink rely on Kafka connectors, but they have > been temporarily removed[3], and development activities[4] for Apache Paimon > adaptation to Flink 2.0, etc. > > > Best, Yanquan. > > [1]https://lists.apache.org/thread/rl7prqop7wfn2o8j2j9fd96dgr1bjjnx > [2]https://issues.apache.org/jira/browse/FLINK-31599 > [3]https://issues.apache.org/jira/browse/FLINK-36413 > [4]https://github.com/apache/paimon/issues/5350 > > > > 2025年4月1日 22:50，Tom Cooper <c...@tomcooper.dev> 写道： > > > > Kafka 4.0.0 >
