Thanks for letting the community/users know. The proposal seems sensible.
I'm wondering if is it worth to add a note about this on the release notes here: https://www.apache.org/dist/kafka/2.2.1/RELEASE_NOTES.html. On Thu, Sep 12, 2019 at 5:23 AM Matthias J. Sax <matth...@confluent.io> wrote: > Hi, > > recently a user reported an issue upgrading a Kafka Streams application > from 2.2.0 to 2.2.1 (cf > https://mail-archives.apache.org/mod_mbox/kafka-users/201908.mbox/ > <CAG8RrxkDAhCrGQ1%3DpBpX-Jcxagk_xkmRk4MBQeZYAP6ZGHoK_w%40mail.gmail.com>) > > After some investigation, we identified > https://issues.apache.org/jira/browse/KAFKA-7895 to be the root cause of > the problem. > > The fix for KAFKA-7895 is using message headers and thus requires broker > version 0.11.0 (or newer) and message format 0.11 (or newer). Hence, > while a Kafka Streams application version 2.2.0 is compatible to older > brokers (0.10.1 and 0.10.2) and only requires message format 0.10, the > backward compatibility was broken accidentally in 2.2.1. > > The fix is also contained in 2.3.0 release and cherry-picked to 2.1 > branch (2.1.2 is not released yet, and thus 2.1 users are not affected > as this point). > > Note: only users that use `suppress()` operator in their program are > affected. > > We should not break streams-broker backward compatibility in bug-fix > releases at all and avoid for minor releases. However, it seems > necessary to have the fix in 2.3.0 though -- otherwise, `suppress()` is > effectively useless and it does not seem to be a good idea to fix the > bug only in the next major release. Hence, trading-off some backward > compatibility in a minor release seems to be acceptable for this case, > considering that 0.11.0 was release 2 years ago. > > For 2.2.1, it is more challenging to decide how to move forward, because > we should not have broken streams-broker compatibility but 2.2.1 is > already released and we can only react after the fact. > > From my point of view, the best way is to keep the fix and update the > release notes and documentation accordingly. The main reason for my > suggestions is that we would expect a majority of users to be on 0.11.0 > brokers already and the fix will work for them -- reverting the fix in > 2.2.2 seems to be worse for all those users on newer broker versions. We > also know that `suppress()` is a highly demanded feature and a lot of > people waited for a fix. > > The expected minority of users that are on 0.10.1 / 0.10.2 brokers, or > newer brokers but still on message format 0.10 would either need to stay > on Kafka Streams 2.2.0 or upgrade their brokers/message format > accordingly. However, upgrading brokers/message format is de-facto > already required for 2.2.1 and thus keeping the fix would not make the > situation worse. > > For 2.1, I would suggest to revert the fix to make sure we don't break > streams-broker compatibility for 2.1.x users. If those users need the > fix for `suppress()` they need to upgrade to 2.2.1/2.3.0 or newer and > make sure their brokers are on 0.11.0 with message format 0.11, too. > > > TL;DR; the proposal is: > > (1) revert the fix for KAFKA-7895 in 2.1 branch > (2) keep the fix for KAFKA-7895 in 2.2.1 and 2.3.0 > > Impact: > > - Kafka Streams 2.1.x and 2.2.0 applications are backward compatible > back to 0.10.1 brokers, requiring message format 0.10 > - Kafka Streams 2.2.2 / 2.3.0 application (or newer) are backward > compatible back to 0.11.0 brokers, requiring message format 0.11 > > > Please let us know what you think about this proposal. > > > -Matthias > > > >
