Hi Koushik, The vote for this KIP already passed.
See https://www.mail-archive.com/dev@kafka.apache.org/msg99636.html best, Colin On Thu, Aug 8, 2019, at 10:50, Koushik Chitta wrote: > Thanks Colin, George. Can we restart the voting for this KIP. > > Thanks, > Koushik > > -----Original Message----- > From: Colin McCabe <cmcc...@apache.org> > Sent: Wednesday, August 7, 2019 5:17 PM > To: dev@kafka.apache.org > Subject: Re: [VOTE] KIP-455: Create an Administrative API for Replica > Reassignment > > On Wed, Aug 7, 2019, at 15:41, George Li wrote: > > This email seemed to get lost in the dev email server. Resending. > > > > > > On Tuesday, August 6, 2019, 10:16:57 PM PDT, George Li > > <sql_consult...@yahoo.com> wrote: > > > > > > The pending reassignments partitions would be reported as URP (Under > > Replicated Partitions). or maybe reported as a separate metrics of > > RURP (Reassignment URP) since now we can derived from the new > > AddingReplicas. An alert could be triggered based on this. > > > > Hi George, > > I agree that this would be a great idea for follow up work. Check out > KIP-352, which discusses creating a such a metric. :) > > > > > > > It would be nice if ListPartitionReassignmentResult could return the > > "elapsed time/duration" of the current pending reassignments, the > > calling client can flag those current long running reassignments and > > alert. However, what I would be interested is probably the total # of > > pending reassignments because I will submit reassignments in batches, > > e.g. 50 reassignments per batch. If the pending reassignments # is > > below that per batch #, submit more new reassignments = (per_batch_# - > > pending_#). > > > > It is definitely useful to know what reassignments exist. If you call > ListPartitionReassignments, you can count how many results you get, in > order to implement a policy like that. > > I'm not sure if knowing how long reassignments have been in progress > will be important or not. I think we should give people some time to > try out the new APIs and see what could be improved based on their > experience. > > > > > > > It seems currently, the ReplicaFetcher threads could quite easily crash > > because of some exceptions. e.g. Java Out Of Memory, and would just > > remain dead (jstack to dump threads to check the # of running > > ReplicaFetcher threads) without getting restarted automatically, so > > needs to bounce the broker. It would be nice to make the > > ReplicaFetcher more robust/resilient of catching more exceptions, and > > if crashed, get restarted after some time. > > > > This has definitely been an issue in the past, I agree. Thankfully, we > recently did improve the robustness of the ReplicaFetcher. Check out > "KIP-461: Improve Replica Fetcher behavior at handling partition > failure." > > cheers, > Colin > > > > > > > Thanks, > > > > George > > > > > > > > On 2019/08/06 23:07:19, "Colin McCabe" <cmcc...@apache.org> wrote: > > > Hi Koushik, > > > > > > Thanks for the idea. This KIP is already pretty big, so I think we'll > > > have to consider ideas like this in follow-on KIPs. > > > > > > In general, figuring out what's wrong with replication is a pretty tough > > > problem. If we had an API for this, we'd probably want it to be unified, > > > and not specific to reassigning partitions. > > > > > > regards, > > > Colin > > > > > > > > > On Tue, Aug 6, 2019, at 10:57, Koushik Chitta wrote: > > > > Hey Colin, > > > > > > > > Can the ListPartitionReassignmentsResult include the status of the > > > > current reassignment progress of each partition? A reassignment can be > > > > in progress for different reasons and the status can give the option to > > > > alter the current reassignment. > > > > > > > > Example - A leaderISRRequest of a new assigned replicas can be > > > > ignored/errored because of a storage exception. And reassignment batch > > > > will be waiting indefinitely for the new assigned replicas to be in > > > > sync with the leader of the partition. > > > > Showing the status will give an option to alter the > > > > affected > > > > partitions and allow the batch to complete reassignment. > > > > > > > > OAR = {1, 2, 3} and RAR = {4,5,6} > > > > > > > > AR leader/isr > > > > {1,2,3,4,5,6} 1/{1,2,3,4,6} => LeaderISRRequest > > > > was lost/skipped for 5 and the reassignment operation will be waiting > > > > indefinitely for the 5 to be insync. > > > > > > > > > > > > > > > > Thanks, > > > > Koushik > > > > > > > > -----Original Message----- > > > > From: Jun Rao <j...@confluent.io> > > > > Sent: Friday, August 2, 2019 10:04 AM > > > > To: dev <dev@kafka.apache.org> > > > > Subject: Re: [VOTE] KIP-455: Create an Administrative API for Replica > > > > Reassignment > > > > > > > > Hi, Colin, > > > > > > > > First, since we are changing the format of LeaderAndIsrRequest, which > > > > is an inter broker request, it seems that we will need IBP during > > > > rolling upgrade. Could we add that to the compatibility section? > > > > > > > > Regarding UnsupportedVersionException, even without ZK node version > > > > bump, we probably want to only use the new ZK value fields after all > > > > brokers have been upgraded to the new binary. Otherwise, the > > > > reassignment task may not be completed if the controller changes to a > > > > broker still on the old binary. > > > > IBP is one way to achieve that. The main thing is that we need some way > > > > for the controller to deal with the new ZK fields. Dealing with the > > > > additional ZK node version bump seems a small thing on top of that? > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Thu, Aug 1, 2019 at 3:05 PM Colin McCabe <cmcc...@apache.org> wrote: > > > > > > > > > On Thu, Aug 1, 2019, at 12:00, Jun Rao wrote: > > > > > > Hi, Colin, > > > > > > > > > > > > 10. Sounds good. > > > > > > > > > > > > 13. Our current convention is to bump up the version of ZK value if > > > > > > there is any format change. For example, we have bumped up the > > > > > > version of the value in /brokers/ids/nnn multiple times and all of > > > > > > those changes are compatible (just adding new fields). This has the > > > > > > slight benefit that it makes it clear there is a format change. > > > > > > Rolling upgrades and downgrades can still be supported with the > > > > > > version bump. For example, if you > > > > > downgrade > > > > > > from a compatible change, you can leave the new format in ZK and > > > > > > the > > > > > > old code will only pick up fields relevant to the old version. > > > > > > Upgrade will > > > > > be > > > > > > controlled by inter broker protocol. > > > > > > > > > > Hmm. If we bump that ZK node version, we will need a new > > > > > inter-broker > > > > > protocol version. We also need to return UnsupportedVersionException > > > > > from the alterPartitionReassignments and listPartitionReassignments > > > > > APIs when the IBP is too low. This sounds doable, although we might > > > > > need a release note that upgrading the IBP is necessary to allow > > > > > reassignment operations after an upgrade. > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > On Wed, Jul 31, 2019 at 1:22 PM Colin McCabe <cmcc...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Hi Jun, > > > > > > > > > > > > > > Thanks for taking another look at this. > > > > > > > > > > > > > > On Wed, Jul 31, 2019, at 09:22, Jun Rao wrote: > > > > > > > > Hi, Stan, > > > > > > > > > > > > > > > > Thanks for the explanation. > > > > > > > > > > > > > > > > 10. If those new fields in LeaderAndIsr are only needed for > > > > > > > > future > > > > > work, > > > > > > > > perhaps they should be added when we do the future work instead > > > > > > > > of > > > > > now? > > > > > > > > > > > > > > I think this ties in with one of the big goals of this KIP, > > > > > > > making > > > > > > > it possible to distinguish reassigning replicas from normal > > > > > > > replicas. > > > > > This is > > > > > > > the key to follow-on work like being able to ensure that > > > > > > > partitions > > > > > with a > > > > > > > reassignment don't get falsely flagged as under-replicated in the > > > > > metrics, > > > > > > > or implementing reassignment quotas that don't accidentally affect > > > > > normal > > > > > > > replication traffic when a replica falls out of the ISR. > > > > > > > > > > > > > > For these follow-on improvements, we need to have that > > > > > > > information > > > > > > > in LeaderAndIsrRequest. We could add the information in a > > > > > > > follow-on KIP, > > > > > of > > > > > > > course, but then all the improvements are blocked on that > > > > > > > follow-on > > > > > KIP. > > > > > > > That would slow things down for all of the downstream KIPs that > > > > > > > are > > > > > blocked > > > > > > > on this. > > > > > > > > > > > > > > Also, to keep things consistent, I think it would be best if the > > > > > format of > > > > > > > the data in the LeaderAndIsrRequest matched the format of the > > > > > > > data > > > > > > > in ZooKeeper. Since we're deciding on the ZK format in this KIP, > > > > > > > I think > > > > > it > > > > > > > makes sense to also decide on the format in the > > > > > > > LeaderAndIsrRequest. > > > > > > > > > > > > > > > > > Should we include those two fields in UpdateMetadata and > > > > > potentially > > > > > > > > > > Metadata requests too? > > > > > > > > > > > > > > We had some discussion earlier about how metadata responses to > > > > > > > clients > > > > > are > > > > > > > getting too large, in part because they include a lot of > > > > > > > information > > > > > that > > > > > > > most clients don't need (such as the ISR). I think reassignment > > > > > > > information definitely falls in the category of something a client > > > > > doesn't > > > > > > > need to know, so we shouldn't include it. > > > > > > > > > > > > > > A program like CruiseControl, or the command-line reassignment > > > > > > > program, just wants to get the most up-to-date information about > > > > > > > the state of reassigning partitions. The MetadataRequest API > > > > > > > wouldn't deliver that, because there are inherently delays in how > > > > > > > we propagate metadata to brokers. That's why the > > > > > > > ListPartitionReassignments API is a better > > > > > choice > > > > > > > for those programs. So I think if we added this information to > > > > > > > the MetadataResponse, nobody would actually use it, and it would > > > > > > > just use > > > > > up > > > > > > > more bandwidth. > > > > > > > > > > > > > > Of course, we can always revisit this later if we find a scenario > > > > > where a > > > > > > > producer or consumer would actually care about this. But I think > > > > > > > we > > > > > should > > > > > > > default to not adding stuff to the metadata response if we don't > > > > > > > have a good use case in mind. > > > > > > > > > > > > > > > > > 11. "If a new reassignment is issued during an on-going > > > > > > > > > > one, > > > > > > > > > > we > > > > > > > cancel the > > > > > > > > > > current one by emptying out both AR and RR, constructing > > > > > > > > > > them > > > > > from > > > > > > > (the > > > > > > > > > > updated from the last-reassignment) R and TR, and starting > > > > > anew." In > > > > > > > this > > > > > > > > > > case, it seems that the controller needs to issue a > > > > > > > > > > StopReplica > > > > > > > request to > > > > > > > > > > remove those unneeded replicas. > > > > > > > > > > > > > > Good catch. Yes, we should document this in the KIP. > > > > > > > > > > > > > > > > > 12. "Essentially, once a cancellation is called we subtract > > > > > > > > > > AR > > > > > from > > > > > > > R, > > > > > > > > > > empty out both AR and RR, and send LeaderAndIsr requests to > > > > > cancel > > > > > > > the > > > > > > > > > > replica movements that have not yet completed." Similar to > > > > > > > > > > the > > > > > > > above, it > > > > > > > > > > seems the controller needs to issue a StopReplica request to > > > > > remove > > > > > > > those > > > > > > > > > > unneeded replicas. > > > > > > > > > > > > > > Right. Let's add this. > > > > > > > > > > > > > > > > > 13. Since we changed the format of the topics/[topic] zNode, > > > > > should > > > > > > > we bump > > > > > > > > > > up the version number in the json value? > > > > > > > > > > > > > > The change to the zNode is backwards compatible, though. Older > > > > > > > brokers will continue to work, but just ignore the new fields. > > > > > > > If > > > > > > > we bump that version number, then downgrades will require > > > > > > > hand-editing zookeeper. > > > > > (Of > > > > > > > course downgrade isn't officially supported, but it would be nice > > > > > > > not > > > > > to > > > > > > > break it if we don't need to...) Changing the version number > > > > > > > would > > > > > also > > > > > > > create problems during a rolling upgrade. > > > > > > > > > > > > > > best, > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > On Mon, Jul 22, 2019 at 8:38 AM Colin McCabe > > > > > > > > > > <cmcc...@apache.org > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > With three non-binding +1 votes from Viktor Somogyi-Vass, > > > > > Robert > > > > > > > > > Barrett, > > > > > > > > > > > and George Li, and 3 binding +1 votes from Gwen Shapira, > > > > > > > > > > > Jason > > > > > > > > > Gustafson, > > > > > > > > > > > and myself, the vote passes. Thanks, everyone! > > > > > > > > > > > > > > > > > > > > > > best, > > > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 19, 2019, at 08:55, Robert Barrett wrote: > > > > > > > > > > > > +1 (non-binding). Thanks for the KIP! > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 18, 2019 at 5:59 PM George Li < > > > > > > > sql_consult...@yahoo.com > > > > > > > > > > > .invalid> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for addressing the comments. > > > > > > > > > > > > > George > > > > > > > > > > > > > > > > > > > > > > > > > > On Thursday, July 18, 2019, 05:03:58 PM PDT, Gwen > > > > > Shapira < > > > > > > > > > > > > > g...@confluent.io> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Renewing my +1, thank you Colin and Stan for working > > > > > through > > > > > > > all > > > > > > > > > the > > > > > > > > > > > > > questions, edge cases, requests and alternatives. We > > > > > > > > > > > > > ended > > > > > up > > > > > > > with > > > > > > > > > a > > > > > > > > > > > > > great protocol. > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 18, 2019 at 4:54 PM Jason Gustafson < > > > > > > > > > ja...@confluent.io> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 Thanks for the KIP. Really looking forward to > > > > > > > > > > > > > > this! > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Jason > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 17, 2019 at 1:41 PM Colin McCabe < > > > > > > > cmcc...@apache.org > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Stanislav. Let's restart the vote to > > > > > > > > > > > > > > > reflect > > > > > the > > > > > > > fact > > > > > > > > > > that > > > > > > > > > > > > > we've > > > > > > > > > > > > > > > made significant changes. The new vote will go > > > > > > > > > > > > > > > for 3 > > > > > days > > > > > > > as > > > > > > > > > > > usual. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'll start with my +1 (binding). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > best, > > > > > > > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 17, 2019, at 08:56, Stanislav > > > > > > > > > > > > > > > Kozlovski > > > > > wrote: > > > > > > > > > > > > > > > > Hey everybody, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We have further iterated on the KIP in the > > > > > accompanying > > > > > > > > > > > discussion > > > > > > > > > > > > > thread > > > > > > > > > > > > > > > > and I'd like to propose we resume the vote. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Some notable changes: > > > > > > > > > > > > > > > > - we will store reassignment information in the > > > > > > > > > > > > > `/brokers/topics/[topic]` > > > > > > > > > > > > > > > > - we will internally use two collections to > > > > > represent a > > > > > > > > > > > reassignment > > > > > > > > > > > > > - > > > > > > > > > > > > > > > > "addingReplicas" and "removingReplicas". > > > > > LeaderAndIsr has > > > > > > > > > been > > > > > > > > > > > > > updated > > > > > > > > > > > > > > > > accordingly > > > > > > > > > > > > > > > > - the Alter API will still use the > > > > > > > > > > > > > > > > "targetReplicas" > > > > > > > > > collection, > > > > > > > > > > > but > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > List API will now return three separate > > > > > > > > > > > > > > > > collections > > > > > - the > > > > > > > > > full > > > > > > > > > > > > > replica > > > > > > > > > > > > > > > set, > > > > > > > > > > > > > > > > the replicas we are adding as part of this > > > > > reassignment > > > > > > > > > > > > > > > ("addingReplicas") > > > > > > > > > > > > > > > > and the replicas we are removing > > > > > > > > > > > > > > > > ("removingReplicas") > > > > > > > > > > > > > > > > - cancellation of a reassignment now means a > > > > > > > > > > > > > > > > proper > > > > > > > rollback > > > > > > > > > of > > > > > > > > > > > the > > > > > > > > > > > > > > > > assignment to its original state prior to the > > > > > > > > > > > > > > > > API > > > > > call > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As always, you can re-read the KIP here > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwik > > > > > i.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-455%253A%2BCreate%2B > > > > > an%2BAdministrative%2BAPI%2Bfor%2BReplica%2BReassignment&data=02%7 > > > > > C01%7Ckchitta%40microsoft.com%7C09235110291f453cb18608d7176b8505%7C72f > > > > > 988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637003622876331137&sdata=w > > > > > GYsHbEN5ApACnH5XOswidEnaUjjLOdmbWD0B3Z3xCs%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > Stanislav > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, May 22, 2019 at 6:12 PM Colin McCabe < > > > > > > > > > > cmcc...@apache.org > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi George, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for taking a look. I am working on > > > > > > > > > > > > > > > > > getting > > > > > a PR > > > > > > > > > done > > > > > > > > > > > as a > > > > > > > > > > > > > > > > > proof-of-concept. I'll post it soon. Then > > > > > > > > > > > > > > > > > we'll > > > > > > > finish up > > > > > > > > > > the > > > > > > > > > > > > > vote. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > best, > > > > > > > > > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, May 21, 2019, at 17:33, George Li > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi Colin, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Great! Looking forward to these features. > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > (non-binding) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What is the estimated timeline to have this > > > > > > > implemented? > > > > > > > > > > If > > > > > > > > > > > any > > > > > > > > > > > > > help > > > > > > > > > > > > > > > > > > is needed in the implementation of > > > > > > > > > > > > > > > > > > cancelling > > > > > > > > > > > reassignments, I > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > > help if there is spare cycle. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > George > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thursday, May 16, 2019, 9:48:56 AM > > > > > > > > > > > > > > > > > > PDT, > > > > > Colin > > > > > > > > > McCabe > > > > > > > > > > > > > > > > > > <cmcc...@apache.org> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi George, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, KIP-455 allows the reassignment of > > > > > individual > > > > > > > > > > > partitions to > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > cancelled. I think it's very important for > > > > > > > > > > > > > > > > > > these > > > > > > > > > > operations > > > > > > > > > > > to > > > > > > > > > > > > > be at > > > > > > > > > > > > > > > > > > the partition level. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > best, > > > > > > > > > > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, May 14, 2019, at 16:34, George Li > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Colin, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the updated KIP. It has very > > > > > > > > > > > > > > > > > > > good > > > > > > > > > > improvements > > > > > > > > > > > of > > > > > > > > > > > > > Kafka > > > > > > > > > > > > > > > > > > > reassignment operations. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > One question, looks like the KIP includes > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > Cancellation > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > individual pending reassignments as well > > > > > > > > > > > > > > > > > > > when > > > > > the > > > > > > > > > > > > > > > > > > > AlterPartitionReasisgnmentRequest has > > > > > > > > > > > > > > > > > > > empty > > > > > > > replicas > > > > > > > > > for > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > topic/partition. Will you also be > > > > > > > > > > > > > > > > > > > implementing > > > > > the > > > > > > > the > > > > > > > > > > > > > partition > > > > > > > > > > > > > > > > > > > cancellation/rollback in the PR ? If > > > > > > > > > > > > > > > > > > > yes, > > > > > it > > > > > > > will > > > > > > > > > > make > > > > > > > > > > > > > KIP-236 > > > > > > > > > > > > > > > (it > > > > > > > > > > > > > > > > > > > has PR already) trivial, since the cancel > > > > > > > > > > > > > > > > > > > all > > > > > > > pending > > > > > > > > > > > > > > > reassignments, > > > > > > > > > > > > > > > > > > > one just needs to do a > > > > > > > ListPartitionRessignmentRequest, > > > > > > > > > > > then > > > > > > > > > > > > > submit > > > > > > > > > > > > > > > > > > > empty replicas for all those > > > > > > > > > > > > > > > > > > > topic/partitions > > > > > in > > > > > > > > > > > > > > > > > > > one AlterPartitionReasisgnmentRequest. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > George > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Friday, May 10, 2019, 8:44:31 PM > > > > > > > > > > > > > > > > > > > PDT, > > > > > Colin > > > > > > > > > McCabe > > > > > > > > > > > > > > > > > > > <cmcc...@apache.org> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 10, 2019, at 17:34, Colin > > > > > > > > > > > > > > > > > > > McCabe > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > On Fri, May 10, 2019, at 16:43, Jason > > > > > Gustafson > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Colin, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think storing reassignment state at > > > > > > > > > > > > > > > > > > > > > the > > > > > > > partition > > > > > > > > > > > level > > > > > > > > > > > > > is > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > right move > > > > > > > > > > > > > > > > > > > > > and I also agree that replicas should > > > > > > > understand > > > > > > > > > that > > > > > > > > > > > > > there is > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > reassignment in progress. This makes > > > > > KIP-352 a > > > > > > > > > > trivial > > > > > > > > > > > > > > > follow-up > > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > example. The only doubt I have is > > > > > > > > > > > > > > > > > > > > > whether > > > > > the > > > > > > > > > leader > > > > > > > > > > > and > > > > > > > > > > > > > isr > > > > > > > > > > > > > > > znode > > > > > > > > > > > > > > > > > is the > > > > > > > > > > > > > > > > > > > > > right place to store the target > > > > > reassignment. > > > > > > > It > > > > > > > > > is a > > > > > > > > > > > bit > > > > > > > > > > > > > odd > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > keep the > > > > > > > > > > > > > > > > > > > > > target assignment in a separate place > > > > > > > > > > > > > > > > > > > > > from > > > > > the > > > > > > > > > > current > > > > > > > > > > > > > > > assignment, > > > > > > > > > > > > > > > > > right? I > > > > > > > > > > > > > > > > > > > > > assume the thinking is probably that > > > > > although > > > > > > > the > > > > > > > > > > > current > > > > > > > > > > > > > > > > > assignment should > > > > > > > > > > > > > > > > > > > > > probably be in the leader and isr > > > > > > > > > > > > > > > > > > > > > znode as > > > > > > > well, it > > > > > > > > > > is > > > > > > > > > > > > > hard to > > > > > > > > > > > > > > > > > move the > > > > > > > > > > > > > > > > > > > > > state in a compatible way. Is that > > > > > > > > > > > > > > > > > > > > > right? > > > > > But > > > > > > > if we > > > > > > > > > > > have no > > > > > > > > > > > > > > > plan > > > > > > > > > > > > > > > > > to remove > > > > > > > > > > > > > > > > > > > > > the assignment znode, do you see a > > > > > downside to > > > > > > > > > > storing > > > > > > > > > > > the > > > > > > > > > > > > > > > target > > > > > > > > > > > > > > > > > > > > > assignment there as well? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Jason, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That's a good point -- it's probably > > > > > > > > > > > > > > > > > > > > better > > > > > to > > > > > > > keep > > > > > > > > > the > > > > > > > > > > > > > target > > > > > > > > > > > > > > > > > > > > assignment in the same znode as the > > > > > > > > > > > > > > > > > > > > current > > > > > > > > > assignment, > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > consistency. I'll change the KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Jason, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks again for the review. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I took another look at this, and I think > > > > > > > > > > > > > > > > > > > we > > > > > should > > > > > > > > > stick > > > > > > > > > > > with > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > initial proposal of putting the > > > > > > > > > > > > > > > > > > > reassignment > > > > > state > > > > > > > into > > > > > > > > > > > > > > > > > > > > > > > > > > /brokers/topics/[topic]/partitions/[partitionId]/state. > > > > > > > > > > > The > > > > > > > > > > > > > > > reason is > > > > > > > > > > > > > > > > > > > because we'll want to bump the leader > > > > > > > > > > > > > > > > > > > epoch > > > > > for the > > > > > > > > > > > partition > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > > > > changing the reassignment state, and the > > > > > > > > > > > > > > > > > > > leader > > > > > > > epoch > > > > > > > > > > > resides > > > > > > > > > > > > > in > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > znode anyway. I agree there is some > > > > > inconsistency > > > > > > > > > here, > > > > > > > > > > > but > > > > > > > > > > > > > so be > > > > > > > > > > > > > > > it: > > > > > > > > > > > > > > > > > > > if we were to greenfield these zookeeper > > > > > > > > > > > > > > > > > > > data > > > > > > > > > structures, > > > > > > > > > > > we > > > > > > > > > > > > > might > > > > > > > > > > > > > > > do > > > > > > > > > > > > > > > > > > > it differently, but the proposed scheme > > > > > > > > > > > > > > > > > > > will > > > > > work > > > > > > > fine > > > > > > > > > > and > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > extensible for the future. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > A few additional questions: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Should > > > > > > > > > > > > > > > > > > > > > `alterPartitionReassignments` be > > > > > > > > > > > > > > > > > `alterPartitionAssignments`? > > > > > > > > > > > > > > > > > > > > > It's the current assignment we're > > > > > > > > > > > > > > > > > > > > > altering, > > > > > > > right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That's fair. AlterPartitionAssigments > > > > > > > > > > > > > > > > > > > > reads > > > > > a > > > > > > > little > > > > > > > > > > > > > better, and > > > > > > > > > > > > > > > > > I'll > > > > > > > > > > > > > > > > > > > > change it to that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1. I've changed the RPC and API name in > > > > > > > > > > > > > > > > > > > +the > > > > > wiki. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Does this change affect the > > > > > > > > > > > > > > > > > > > > > Metadata > > > > > API? In > > > > > > > > > other > > > > > > > > > > > > > words, > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > clients > > > > > > > > > > > > > > > > > > > > > aware of reassignments? If so, then we > > > > > probably > > > > > > > > > need > > > > > > > > > > a > > > > > > > > > > > > > change > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > UpdateMetadata as well. The only > > > > > alternative I > > > > > > > can > > > > > > > > > > > think of > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > be to > > > > > > > > > > > > > > > > > > > > > represent the replica set in the > > > > > > > > > > > > > > > > > > > > > Metadata > > > > > > > request > > > > > > > > > as > > > > > > > > > > > the > > > > > > > > > > > > > union > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > current and target replicas, but I > > > > > > > > > > > > > > > > > > > > > can't > > > > > think > > > > > > > of > > > > > > > > > any > > > > > > > > > > > > > benefit > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > hiding > > > > > > > > > > > > > > > > > > > > > reassignments. Note that if we did > > > > > > > > > > > > > > > > > > > > > this, we > > > > > > > > > probably > > > > > > > > > > > > > wouldn't > > > > > > > > > > > > > > > need > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > separate API to list reassignments. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I thought about this a bit... and I > > > > > > > > > > > > > > > > > > > > think on > > > > > > > balance, > > > > > > > > > > > you're > > > > > > > > > > > > > > > right. > > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > > > > > should keep this information together > > > > > > > > > > > > > > > > > > > > with > > > > > the > > > > > > > > > replica > > > > > > > > > > > > > nodes, isr > > > > > > > > > > > > > > > > > > > > nodes, and offline replicas, and that > > > > > > > information is > > > > > > > > > > > > > available in > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > MetadataResponse. > > > > > > > > > > > > > > > > > > > > However, I do think in order to do > > > > > > > > > > > > > > > > > > > > this, > > > > > we'll > > > > > > > need > > > > > > > > > a > > > > > > > > > > > flag > > > > > > > > > > > > > in > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > MetadataRequest that specifiies "only > > > > > > > > > > > > > > > > > > > > show me > > > > > > > > > > reassigning > > > > > > > > > > > > > > > > > partitions". > > > > > > > > > > > > > > > > > > > > I'll add this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I revisited this, and I think we should > > > > > > > > > > > > > > > > > > > stick > > > > > with > > > > > > > the > > > > > > > > > > > original > > > > > > > > > > > > > > > > > > > proposal of having a separate > > > > > > > > > ListPartitionReassignments > > > > > > > > > > > API. > > > > > > > > > > > > > > > There > > > > > > > > > > > > > > > > > > > really is no use case where the Producer > > > > > > > > > > > > > > > > > > > or > > > > > > > Consumer > > > > > > > > > > needs > > > > > > > > > > > to > > > > > > > > > > > > > know > > > > > > > > > > > > > > > > > > > about a reassignment. They should just be > > > > > notified > > > > > > > > > when > > > > > > > > > > > the > > > > > > > > > > > > > set of > > > > > > > > > > > > > > > > > > > partitions changes, which doesn't require > > > > > changes > > > > > > > to > > > > > > > > > > > > > > > > > > > MetadataRequest/Response. The Admin > > > > > > > > > > > > > > > > > > > client > > > > > only > > > > > > > cares > > > > > > > > > if > > > > > > > > > > > > > someone > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > managing the reassignment. So adding this > > > > > state > > > > > > > to the > > > > > > > > > > > > > > > > > > > MetadataResponse adds overhead for no real > > > > > > > benefit. In > > > > > > > > > > the > > > > > > > > > > > > > common > > > > > > > > > > > > > > > > > case > > > > > > > > > > > > > > > > > > > where there is no ongoing reassignment, it > > > > > would > > > > > > > be 4 > > > > > > > > > > > bytes per > > > > > > > > > > > > > > > > > > > partition of extra overhead in the > > > > > > > MetadataResponse. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I think we have a problem of > > > > > > > oversharing in > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > MetadataRequest/Response. As we 10x or > > > > > > > > > > > > > > > > > > > 100x > > > > > the > > > > > > > number > > > > > > > > > > of > > > > > > > > > > > > > > > partitions > > > > > > > > > > > > > > > > > > > we support, we'll need to get stricter > > > > > > > > > > > > > > > > > > > about > > > > > giving > > > > > > > > > > clients > > > > > > > > > > > > > only > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > information they actually need, about the > > > > > > > partitions > > > > > > > > > they > > > > > > > > > > > > > actually > > > > > > > > > > > > > > > > > care > > > > > > > > > > > > > > > > > > > about. Reassignment state clearly falls > > > > > > > > > > > > > > > > > > > in the > > > > > > > > > category > > > > > > > > > > of > > > > > > > > > > > > > state > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > isn't needed by clients (except very > > > > > specialized > > > > > > > > > > > rebalancing > > > > > > > > > > > > > > > programs). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Another important consideration here is > > > > > > > > > > > > > > > > > > > that > > > > > > > someone > > > > > > > > > > > managing > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > ongoing reassignment wants the most > > > > > > > > > > > > > > > > > > > up-to-date > > > > > > > > > > information, > > > > > > > > > > > > > which > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > be found on the controller. Therefore > > > > > > > > > > > > > > > > > > > adding > > > > > this > > > > > > > > > state > > > > > > > > > > to > > > > > > > > > > > > > > > listTopics > > > > > > > > > > > > > > > > > > > or describeTopics, which could contact any > > > > > node in > > > > > > > the > > > > > > > > > > > > > cluster, is > > > > > > > > > > > > > > > > > > > sub-optimal. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Finally, adding this to listTopics or > > > > > > > describeTopics > > > > > > > > > > feels > > > > > > > > > > > > > like a > > > > > > > > > > > > > > > > > warty > > > > > > > > > > > > > > > > > > > API. It's an extra boolean which > > > > > > > > > > > > > > > > > > > interacts > > > > > with > > > > > > > other > > > > > > > > > > > extra > > > > > > > > > > > > > > > booleans > > > > > > > > > > > > > > > > > > > like "show internal", etc. in weird ways. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I > > > > > think > > > > > > > a > > > > > > > > > > > separate > > > > > > > > > > > > > API > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > cleaner. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. As replicas come into sync, they > > > > > > > > > > > > > > > > > > > > > will > > > > > join > > > > > > > the > > > > > > > > > > ISR. > > > > > > > > > > > > > Will we > > > > > > > > > > > > > > > > > await all > > > > > > > > > > > > > > > > > > > > > target replicas joining the ISR before > > > > > taking > > > > > > > the > > > > > > > > > > > replica > > > > > > > > > > > > > out > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > the target > > > > > > > > > > > > > > > > > > > > > replicas set? Also, I assume that > > > > > > > > > > > > > > > > > > > > > target > > > > > > > replicas > > > > > > > > > can > > > > > > > > > > > > > still be > > > > > > > > > > > > > > > > > elected as > > > > > > > > > > > > > > > > > > > > > leader? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We'll take a replica out of the target > > > > > replicas > > > > > > > set > > > > > > > > > as > > > > > > > > > > > soon > > > > > > > > > > > > > as > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > replica is in the ISR. Let me clarify > > > > > > > > > > > > > > > > > > > > this > > > > > in > > > > > > > the > > > > > > > > > KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4. Probably useful to mention > > > > > > > > > > > > > > > > > > > > > permissions > > > > > for > > > > > > > the > > > > > > > > > new > > > > > > > > > > > APIs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Good point. I think > > > > > alterPartitionAssignments > > > > > > > should > > > > > > > > > > > require > > > > > > > > > > > > > > > ALTER > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > CLUSTER. MetadataRequest permissions > > > > > > > > > > > > > > > > > > > > will be > > > > > > > > > > unchanged. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I added permission information. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > best, > > > > > > > > > > > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > best, > > > > > > > > > > > > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Jason > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 10, 2019 at 9:30 AM Gwen > > > > > Shapira < > > > > > > > > > > > > > > > g...@confluent.io> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > > > > > > > Looks great, and will be awesome to > > > > > > > > > > > > > > > > > > > > > > have > > > > > > > this new > > > > > > > > > > > > > capability. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, May 8, 2019 at 10:23 PM > > > > > > > > > > > > > > > > > > > > > > Colin > > > > > McCabe > > > > > > > < > > > > > > > > > > > > > > > cmcc...@apache.org> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to start the vote for > > > > > > > > > > > > > > > > > > > > > > > KIP-455: > > > > > > > Create > > > > > > > > > an > > > > > > > > > > > > > > > > > Administrative API for > > > > > > > > > > > > > > > > > > > > > > > Replica Reassignment. I think > > > > > > > > > > > > > > > > > > > > > > > this > > > > > KIP is > > > > > > > > > > > important > > > > > > > > > > > > > since > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > will unlock > > > > > > > > > > > > > > > > > > > > > > > many follow-on improvements to > > > > > > > > > > > > > > > > > > > > > > > Kafka > > > > > > > > > reassignment > > > > > > > > > > > (see > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > "Future work" > > > > > > > > > > > > > > > > > > > > > > > section, plus a lot of the other > > > > > > > discussions > > > > > > > > > > we've > > > > > > > > > > > had > > > > > > > > > > > > > > > > > recently about > > > > > > > > > > > > > > > > > > > > > > > reassignment). It also furthers > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > important > > > > > > > > > > > KIP-4 > > > > > > > > > > > > > goal > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > removing > > > > > > > > > > > > > > > > > > > > > > > direct access to ZK. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I made a few changes based on the > > > > > > > discussion in > > > > > > > > > > the > > > > > > > > > > > > > > > [DISCUSS] > > > > > > > > > > > > > > > > > thread. As > > > > > > > > > > > > > > > > > > > > > > > Robert suggested, I removed the > > > > > > > > > > > > > > > > > > > > > > > need to > > > > > > > > > > explicitly > > > > > > > > > > > > > cancel a > > > > > > > > > > > > > > > > > reassignment > > > > > > > > > > > > > > > > > > > > > > > for a partition before setting up > > > > > > > > > > > > > > > > > > > > > > > a > > > > > > > different > > > > > > > > > > > > > reassignment > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > > > > specific partition. I also > > > > > > > > > > > > > > > > > > > > > > > simplified > > > > > the > > > > > > > API > > > > > > > > > a > > > > > > > > > > > bit by > > > > > > > > > > > > > > > adding > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > PartitionReassignment class which > > > > > > > > > > > > > > > > > > > > > > > is > > > > > used > > > > > > > by > > > > > > > > > both > > > > > > > > > > > the > > > > > > > > > > > > > alter > > > > > > > > > > > > > > > > > and list > > > > > > > > > > > > > > > > > > > > > > APIs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I modified the proposal so that > > > > > > > > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > now > > > > > > > > > deprecate > > > > > > > > > > > the > > > > > > > > > > > > > old > > > > > > > > > > > > > > > > > znode-based API > > > > > > > > > > > > > > > > > > > > > > > rather than removing it > > > > > > > > > > > > > > > > > > > > > > > completely. > > > > > That > > > > > > > > > should > > > > > > > > > > > give > > > > > > > > > > > > > > > external > > > > > > > > > > > > > > > > > > > > > > rebalancing > > > > > > > > > > > > > > > > > > > > > > > tools some time to transition to > > > > > > > > > > > > > > > > > > > > > > > the > > > > > new > > > > > > > API. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > To clarify a question Viktor > > > > > > > > > > > > > > > > > > > > > > > asked, I > > > > > > > added a > > > > > > > > > > note > > > > > > > > > > > > > that the > > > > > > > > > > > > > > > > > > > > > > > kafka-reassign-partitions.sh will > > > > > > > > > > > > > > > > > > > > > > > now > > > > > use a > > > > > > > > > > > > > > > --bootstrap-server > > > > > > > > > > > > > > > > > argument > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > contact the admin APIs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks, Colin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > *Gwen Shapira* Product Manager | > > > > > > > > > > > > > > > > > > > > > > Confluent > > > > > > > > > > > > > > > > > > > > > > 650.450.2760 | @gwenshap Follow us: > > > > > > > > > > > > > > > > > > > > > > Twitter < > > > > > > > > > > https: > > > > > > > > > > %2F%2Ftwitter.com%2FConfluentInc&data=02%7C01%7Ckchitta% > > > > > > > > > > 40microsoft.com%7C09235110291f453cb18608d7176b8505%7C72f988b > > > > > > > > > > f86f141af91ab2d7cd011db47%7C1%7C0%7C637003622876341136&s > > > > > > > > > > data=nk22CVhfV%2FTqNRFQc04LLldSHmFHTvrxcxYZcORpITg%3D&re > > > > > > > > > > served=0> > > > > > > > > > > > | > > > > > > > > > > > > > blog > > > > > > > > > > > > > > > > > > > > > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnam06.safelinks.protection&data=02%7C01%7Ckchitta%40microsoft.com%7C3e662dd1d60949f2ff6808d71b95e68f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637008202957720437&sdata=vdDj2GPhjVP5TQj1B4%2BoCjGU6cgDqGVYOEnqaMurz68%3D&reserved=0. > > > > > > > > > > > > > > > > > > > > > > outlook.com/?url=http%3A%2F%2Fwww.co > > > > > > > > > > > > > > > > > > > > > > nfluent.io%2Fblog&data=02%7C01%7 > > > > > > > > > > > > > > > > > > > > > > Ckchitta%40microsoft.com%7C092351102 > > > > > > > > > > > > > > > > > > > > > > 91f453cb18608d7176b8505%7C72f988bf86 > > > > > > > > > > > > > > > > > > > > > > f141af91ab2d7cd011db47%7C1%7C0%7C637 > > > > > > > > > > > > > > > > > > > > > > 003622876341136&sdata=TwKB1ey0Do > > > > > > > > > > > > > > > > > > > > > > pQ3gmZJ88vPHxHF4hOZEhQH8z8LqhElQc%3D > > > > > > > > > > > > > > > > > > > > > > &reserved=0> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > Gwen Shapira > > > > > > > > > > > > > Product Manager | Confluent > > > > > > > > > > > > > 650.450.2760 | @gwenshap Follow us: Twitter | blog > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
