Hi all, I noticed that we recently altered Kafka Connect to explicitly disable idempotent producers by default [1], with the rationale being that this ensured compatibility with brokers running versions older than 0.11 and ACL-protected brokers running versions older than 2.8. There has also been mention that KIP-318 [2] would have to be approved before reverting this change and enabling idempotent producers by default on Kafka Connect.
This is a pretty restrictive precedent to set and I want to see if we could consider relaxing it a little bit. When KIP-679 [3] was approved and idempotence was enabled by default in all producers starting with 3.0, it was with full understanding that it would lead to breaking changes in these scenarios for all producer applications, and with conviction that the upgrade steps laid out were sufficient for users to be able to handle this gracefully. Kafka Connect is one such producer application, and it already gives users the means to implement the upgrade steps laid out in KIP-679 in order to remain compatible with older broker versions. Given that it's possible for users to reconfigure their clusters to work properly with older broker versions, the changes in KIP-679 were released in a major version bump (3.0.0) that permits breaking of backwards compatibility, and there is no significant operational burden placed on users of Kafka Connect by KIP-679 that isn't also placed on other producer applications, do we really need to go through the process of yet another KIP in order to apply these changes to Kafka Connect? And if so, what differentiates Kafka Connect from all the other producer applications that were affected by KIP-679, including Kafka Streams applications configured with the (default) processing mode of "at least once"? Instead of choosing to emphasize cross-major-version backwards compatibility between Kafka Connect and Kafka brokers, we might instead invest our efforts in creating more detailed upgrade notes that call out changes like these and let users know exactly how to configure their Kafka Connect workers (and producer applications in general) when upgrading Kafka Connect and/or their Kafka brokers. [1] - https://issues.apache.org/jira/browse/KAFKA-13759 [2] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-318%3A+Make+Kafka+Connect+Source+idempotent [3] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default Cheers, Chris
