I've made two changes to the document: - Removed the TMR evolution piece since we agreed to retain this. - Added two new API's to the admin client spec. (Alter and Describe config).
Please review. Aditya ________________________________________ From: Ashish Singh [asi...@cloudera.com] Sent: Friday, May 29, 2015 8:36 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2) +1 on discussing this on next KIP hangout. I will update KIP-24 before that. On Fri, May 29, 2015 at 3:40 AM, Andrii Biletskyi < andrii.bilets...@stealth.ly> wrote: > Guys, > > I won't be able to attend next meeting. But in the latest patch for KIP-4 > Phase 1 > I didn't even evolve TopicMetadataRequest to v1 since we won't be able > to change config with AlterTopicRequest, hence with this patch TMR will > still > return isr. Taking this into account I think yes - it would be good to fix > ISR issue, > although I didn't consider it to be a critical one (isr was part of TMR > from the very > beginning and almost no code relies on this piece of request). > > Thanks, > Andrii Biletskyi > > On Fri, May 29, 2015 at 8:50 AM, Aditya Auradkar < > aaurad...@linkedin.com.invalid> wrote: > > > Thanks. Perhaps we should leave TMR unchanged for now. Should we discuss > > this during the next hangout? > > > > Aditya > > > > ________________________________________ > > From: Jun Rao [j...@confluent.io] > > Sent: Thursday, May 28, 2015 5:32 PM > > To: dev@kafka.apache.org > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized > administrative > > operations (Thread 2) > > > > There is a reasonable use case of ISR in KAFKA-2225. Basically, for > > economical reasons, we may want to let a consumer fetch from a replica in > > ISR that's in the same zone. In order to support that, it will be > > convenient to have TMR return the correct ISR for the consumer to choose. > > > > So, perhaps it's worth fixing the ISR inconsistency issue in KAFKA-1367 > > (there is some new discussion there on what it takes to fix this). If we > do > > that, we can leave TMR unchanged. > > > > Thanks, > > > > Jun > > > > On Tue, May 26, 2015 at 1:13 PM, Aditya Auradkar < > > aaurad...@linkedin.com.invalid> wrote: > > > > > Andryii, > > > > > > I made a few edits to this document as discussed in the KIP-21 thread. > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > With these changes. the only difference between > TopicMetadataResponse_V1 > > > and V0 is the removal of the ISR field. I've altered the KIP with the > > > assumption that this is a good enough reason by itself to evolve the > > > request/response protocol. Any concerns there? > > > > > > Thanks, > > > Aditya > > > > > > ________________________________________ > > > From: Mayuresh Gharat [gharatmayures...@gmail.com] > > > Sent: Thursday, May 21, 2015 8:29 PM > > > To: dev@kafka.apache.org > > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized > > administrative > > > operations (Thread 2) > > > > > > Hi Jun, > > > > > > Thanks a lot. I get it now. > > > Point 4) will actually enable clients to who don't want to create a > > topic > > > with default partitions, if it does not exist and then can manually > > create > > > the topic with their own configs(#partitions). > > > > > > Thanks, > > > > > > Mayuresh > > > > > > On Thu, May 21, 2015 at 6:16 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > Mayuresh, > > > > > > > > The current plan is the following. > > > > > > > > 1. Add TMR v1, which still triggers auto topic creation. > > > > 2. Change the consumer client to TMR v1. Change the producer client > to > > > use > > > > TMR v1 and on UnknownTopicException, issue TopicCreateRequest to > > > explicitly > > > > create the topic with the default server side partitions and > replicas. > > > > 3. At some later time after the new clients are released and > deployed, > > > > disable auto topic creation in TMR v1. This will make sure consumers > > > never > > > > create new topics. > > > > 4. If needed, we can add a new config in the producer to control > > whether > > > > TopicCreateRequest should be issued or not on UnknownTopicException. > If > > > > this is disabled and the topic doesn't exist, send will fail and the > > user > > > > is expected to create the topic manually. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat < > > > > gharatmayures...@gmail.com > > > > > wrote: > > > > > > > > > Hi, > > > > > I had a question about TopicMetadata Request. > > > > > Currently the way it works is : > > > > > > > > > > 1) Suppose a topic T1 does not exist. > > > > > 2) Client wants to produce data to T1 using producer P1. > > > > > 3) Since T1 does not exist, P1 issues a TopicMetadata request to > > kafka. > > > > > This in turn creates the default number of partition. The number of > > > > > partitions is a cluster wide config. > > > > > 4) Same goes for a consumer. If the topic does not exist and new > > topic > > > > will > > > > > be created when the consumer issues TopicMetadata request. > > > > > > > > > > Here are 2 use cases where it might not be suited : > > > > > > > > > > The auto creation flag for topics is turned ON. > > > > > > > > > > a) Some clients might not want to create topic with default number > of > > > > > partitions but with lower number of partitions. Currently in a > > > > multi-tenant > > > > > environment this is not possible without changing the cluster wide > > > > default > > > > > config. > > > > > > > > > > b) Some clients might want to just check if the topic exist or not > > but > > > > > currently the topic gets created automatically using default number > > of > > > > > partitions. > > > > > > > > > > Here are some ideas to address this : > > > > > > > > > > 1) The way this can be addressed is that TopicMetadata request > > should > > > > have > > > > > a way to specify whether it should only check if the topic exist or > > > check > > > > > and create a topic with given number of partitions. If the number > of > > > > > partitions is not specified use the default cluster wide config. > > > > > > > > > > OR > > > > > > > > > > 2) We should only allow TopicMetadata Request to get the metadata > > > > > explicitly and not allow it to create a new topic. We should have > > > another > > > > > Request that takes in config parameters from the user regarding how > > > > he/she > > > > > wants the topic to be created. This request can be used if we get > an > > > > empty > > > > > TopicMetadata Response. > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Mayuresh > > > > > > > > > > > > > > > On Thu, May 14, 2015 at 10:22 AM, Jun Rao <j...@confluent.io> > wrote: > > > > > > > > > > > For ListTopics, we decided not to add a ListTopics request for > now > > > and > > > > > just > > > > > > rely on passing in an empty list to TMR. We can revisit this in > the > > > > > future > > > > > > if it becomes an issue. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy <jjkosh...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > Just had a few minor questions before I join the vote thread. > > > > > > > Apologies if these have been discussed: > > > > > > > > > > > > > > - Do we need DecreasePartitionsNotAllowed? i.e., can we just > > return > > > > > > > InvalidPartitions instead? > > > > > > > - AdminClient.listTopics: should we allow listing all > partitions? > > > Or > > > > > > > do you intend for the client to issue listTopics followed by > > > > > > > describeTopics? > > > > > > > - On returning future<void> for partition reassignments: do we > > need > > > > to > > > > > > > return any future especially since you have the > > > > > > > verifyReassignPartitions method? For e.g., what happens if > the > > > > > > > controller moves? The get should fail right? The client will > > then > > > > > > > need to connect to the new controller and reissue the request > > but > > > > > > > will then get ReassignPartitionsInProgress. So in that case > the > > > > > > > client any way needs to rely in verifyReassignPartitions. > > > > > > > - In past hangouts I think either you/Joe were mentioning the > > need > > > to > > > > > > > locate the controller (and possibly other cluster metadata). > It > > > > > > > appears we decided to defer this for a future KIP. Correct? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Joel > > > > > > > > > > > > > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi > wrote: > > > > > > > > Guys, > > > > > > > > > > > > > > > > I've updated the wiki to reflect all previously discussed > items > > > > > > > > (regarding the schema only - this is included to phase 1). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > > > > > > > > > > > I think we can have the final discussion today (for phase 1) > > and > > > > > > > > in case no new remarks I will start the voting thread. > > > > > > > > > > > > > > > > With regards to AlterTopicRequest semantics. I agree with > Jun, > > > > > > > > and I think it's my bad I focused on "multiple topics in one > > > > > request". > > > > > > > > The same situation is possible in ProduceRequest, Fetch, > > > > > TopicMetadata > > > > > > > > and we handle it naturally and in the most transparent way - > we > > > > > > > > put all separate instructions into a map and thus silently > > ignore > > > > > > > > duplicates. > > > > > > > > This also makes Response part simple too - it's just a map > > > > > > > Topic->ErrorCode. > > > > > > > > I think we need to follow the same approach for Alter (and > > > Create, > > > > > > > Delete) > > > > > > > > request. With this we add nothing new in terms of batch > > requests > > > > > > > > semantics. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -Regards, > > > > > Mayuresh R. Gharat > > > > > (862) 250-7125 > > > > > > > > > > > > > > > > > > > > > -- > > > -Regards, > > > Mayuresh R. Gharat > > > (862) 250-7125 > > > > > > -- Regards, Ashish
