Hi all, As part of making these changes and testing the clients mentioned in the KIP, we identified a number of adjustments required to satisfy the premise that released versions of the clients mentioned in the KIP would work with Apache Kafka 4.0:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=235838251&selectedPageVersions=23&selectedPageVersions=18 So far so good, nothing too controversial. And that leads us to the remaining piece to achieve the stated goal, we need to workaround a bug in librdkafka where produce v0 needs to be in the api versions response for compressed produce to work with lz4, gzip and snappy[1]. Unfortunately, supporting produce v0 would imply supporting the older record formats (v0 & v1), which would imply reintroducing significant tech debt. I would rather not do that. A better alternative, in my opinion, is to advertise support for produce v0-v2, but return an UnsupportedVersion error if this is attempted. Since clients pick the highest supported version by both client and broker during version negotiation, this solves the problem with minimal tech debt (even though it's not ideal that the protocol definition becomes inconsistent with the actual protocol support). This workaround would be removed in Apache Kafka 5.0. If there are no strong concerns or better alternatives suggested, I will go ahead with this approach as it's pretty straightforward: https://github.com/apache/kafka/pull/18727 Ismael [1] https://issues.apache.org/jira/browse/KAFKA-18659 On Sun, Dec 17, 2023 at 7:40 AM Ismael Juma <m...@ismaeljuma.com> wrote: > Hi all, > > It turns out that kafka-python's protocol api versions support is worse > than originally described - newer versions were defined but not actually > supported in a few cases. I have updated the KIP to address this: > > > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=235838251&selectedPageVersions=18&selectedPageVersions=17 > > The main impact is that we will deprecate and remove fewer consumer > related api versions - fetch, group and offset management primarily (fetch > was the most impacted). > > Ismael > > On Tue, Nov 21, 2023 at 12:06 PM Ismael Juma <m...@ismaeljuma.com> wrote: > >> Hi all, >> >> I would like to start a vote on KIP-896. Please take a look and let us >> know what you think. >> >> Even though most of the changes in this KIP will be done for Apache Kafka >> 4.0, I would like to introduce a new metric and new request log attribute >> in Apache 3.7 to help users identify usage of deprecated protocol api >> versions. >> >> Link: >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0 >> >> Thanks, >> Ismael >> >
