Hi, Dong, Thanks for the KIP. At the high level, this makes sense. A few comments below.
1. It would be useful to support removing partitions as well. The general idea could be bumping the leader epoch for the remaining partitions. For the partitions to be removed, we can make them read-only and remove them after the retention time. 2. If we support removing partitions, I am not sure if it's enough to fence off the producer using total partition number since the total partition number may remain the same after adding and then removing partitions. Perhaps we need some notion of partition epoch. 3. In step 5) of the Proposed Changes, I am not sure that we can always rely upon position 0 for dealing with the new partitions. A consumer will start consuming the new partition when some of the existing records have been removed due to retention. 4. When the consumer is allowed to read messages after the partition expansion point, a key may be moved from one consumer instance to another. In this case, similar to consumer rebalance, it's useful to inform the application about this so that the consumer can save and reload the per key state. So, we need to either add some new callbacks or reuse the existing rebalance callbacks. 5. There is some subtlety in assigning partitions. Currently, the consumer assigns partitions without needing to know the consumption offset. This could mean that a particular consumer may be assigned some new partitions that are not consumable yet, which could lead to imbalanced load temporarily. Not sure if this is super important to address though. Thanks, Jun On Sat, Feb 10, 2018 at 3:35 PM, Dong Lin <lindon...@gmail.com> wrote: > Hi all, > > I have created KIP-253: Support in-order message delivery with partition > expansion. See > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 253%3A+Support+in-order+message+delivery+with+partition+expansion > . > > This KIP provides a way to allow messages of the same key from the same > producer to be consumed in the same order they are produced even if we > expand partition of the topic. > > Thanks, > Dong >
