Hi Ismael, I agree it would be awesome to drop support for the old formats! A few comments below:
1. Suppose that a partition with 3 replicas begins at v0. One broker is upgraded to v2 with the new "force upgrade" config as part of a rolling restart, which leaves two replicas on v0 and one on v2. The KIP documents that replicas will take the format from the leader without any conversion, so I'm a little concerned that leader changes in this state could result in the version flipping from v2 to v0. Could that happen? Say, for example, if the third replica was catching up while the rolling restart was in progress. Since it is probably an unlikely case in practice, would it make sense to allow up-conversion during replication when "force upgrade" has been enabled? Not sure I see any great alternatives to this. 2. A related question concerns batching when upgrading from the non-batched formats (i.e. v0 and v1 when no compression is in use). I suspect the only reasonable way to do this is to treat each individual record as a batch of 1 during the conversion process. Otherwise, I am not sure we can come up with a deterministic approach to re-batching which can be applied consistently on all replicas (especially taking into account compaction). The reason this is important is that replication is aligned by batches, so if the batches are not consistent, the replication won't work. The only problem with using single-record batches is that the v2 message format has some additional overhead for the batch itself when compared with the v0 and v1 individual record format. So the up-conversion process would likely increase storage. I don't think this is a dealbreaker, but probably worth calling out somewhere. 3. In 4.0, we are dropping support for Fetch versions 0 through 3. I might have missed it in the KIP, but it sounds like we need a similar restriction on Produce versions since up-conversion will also not be supported? Would it be reasonable to drop ListOffsets v0 at the same time? It is not related to the message format, but the Fetch restriction will already effectively kill any client that might still be using this wonky version. Not sure if there are any similar opportunities, but might be worth checking. Thanks, Jason On Fri, Jun 4, 2021 at 6:14 AM Ismael Juma <ism...@juma.me.uk> wrote: > If there are no comments or objections, I will start a vote on this soon. > > On Tue, Jun 1, 2021 at 7:59 AM Ismael Juma <ism...@juma.me.uk> wrote: > > > Hi all, > > > > It's time to start the process of sunsetting message formats v0 and v1 in > > order to establish a new baseline in terms of supported client/broker > > behavior and to improve maintainability & supportability of Kafka. Please > > take a look at the proposal: > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1 > > > > Ismael > > >
