Hi Viktor, +1 to this KIP.
I would very much like to see AdminClient in TopicCommand. This would also allow us to efficiently implement new features like the "--under-min-isr" option I proposed in KIP-351 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-351%3A+Add+--under-min-isr+option+to+describe+topics+command> . Thanks. Regards, Kevin On Sat, Oct 20, 2018 at 10:52 PM Colin McCabe <cmcc...@apache.org> wrote: > Hi Viktor, > > Sounds good. If you want to propose a way of improving the metadata > protocol so that "[deleted]" could be supported, you could probably create > that KIP in parallel. > > The last KIP in that area that I can remember is KIP-142, which didn't get > adopted (yet?) > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-142%3A+Add+ListTopicsRequest+to+efficiently+list+all+the+topics+in+a+cluster > > There have been other discussions though. In general there are a lot of > features that would be nice to have in the metadata protocol (pagniation, > regexes, skip stuff we don't need). > > best, > Colin > > > On Tue, Oct 16, 2018, at 10:11, Viktor Somogyi-Vass wrote: > > Hi Colin, > > > > Thanks, it makes sense and simplifies this KIP tremendously. I'll move > this > > section to the rejected alternatives with a note that KIP-142 will have > > this feature. > > On a similar note: is there a KIP for describe topics protocol or have > you > > been thinking about it? I guess there it's the same problem, we often > don't > > want to forward the entire metadata. > > > > Viktor > > > > On Fri, Oct 12, 2018 at 12:03 PM Colin McCabe <cmcc...@apache.org> > wrote: > > > > > Hi Viktor, > > > > > > Thanks for bumping this thread. > > > > > > I think we should just focus on transitioning the TopicCommand to use > > > AdminClient, and talk about protocol changes in a separate KIP. > Protocol > > > changes often involve a lot of discussion. This does mean that we > couldn't > > > implement the "list topics under deletion" feature when using > AdminClient > > > at the moment. We could add a note to the tool output indicating this. > > > > > > We should move the protocol discussion to a separate thread. Probably > > > also look at KIP-142 as well. > > > > > > best, > > > Colin > > > > > > > > > On Tue, Oct 9, 2018, at 07:45, Viktor Somogyi-Vass wrote: > > > > Hi All, > > > > > > > > Would like to bump this as the conversation sank a little bit, but > more > > > > importantly I'd like to validate my plans/ideas on extending the > Metadata > > > > protocol. I was thinking about two other alternatives, namely: > > > > 1. Create a ListTopicUnderDeletion protocol. This however would be > > > > unnecessary: it'd have one very narrow functionality which we can't > > > extend. > > > > I'd make sense to have a list topics or describe topics protocol > where we > > > > can list/describe topics under deletion but for normal > listing/describing > > > > we already use the metadata, so it would be a duplication of > > > functionality. > > > > 2. DeleteTopicsResponse could return the topics under deletion if the > > > > request's argument list is empty which might make sense at the first > > > look, > > > > but actually we'd mix the query functionality with the delete > > > functionality > > > > which is counterintuitive. > > > > > > > > Even though most clients won't need these "limbo" topics (which are > under > > > > deletion) in the foreseeable future, it can be considered as part of > the > > > > cluster state or metadata and to me it makes sense. Also it doesn't > have > > > a > > > > big overhead in the response size as typically users don't delete > topics > > > > too often as far as I experienced. > > > > > > > > I'd be happy to receive some ideas/feedback on this. > > > > > > > > Cheers, > > > > Viktor > > > > > > > > > > > > On Fri, Sep 28, 2018 at 4:51 PM Viktor Somogyi-Vass < > > > viktorsomo...@gmail.com> > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > I made an update to the KIP. Just in short: > > > > > Currently KafkaAdminClient.describeTopics() and > > > > > KafkaAdminClient.listTopics() uses the Metadata protocol to acquire > > > topic > > > > > information. The returned response however won't contain the topics > > > that > > > > > are under deletion but couldn't complete yet (for instance because > of > > > some > > > > > replicas offline), therefore it is not possible to implement the > > > current > > > > > command's "marked for deletion" feature. To get around this I > > > introduced > > > > > some changes in the Metadata protocol. > > > > > > > > > > Thanks, > > > > > Viktor > > > > > > > > > > On Fri, Sep 28, 2018 at 4:48 PM Viktor Somogyi-Vass < > > > > > viktorsomo...@gmail.com> wrote: > > > > > > > > > >> Hi Mickael, > > > > >> > > > > >> Thanks for the feedback, I also think that many customers wanted > this > > > for > > > > >> a long time. > > > > >> > > > > >> Cheers, > > > > >> Viktor > > > > >> > > > > >> On Fri, Sep 28, 2018 at 11:45 AM Mickael Maison < > > > mickael.mai...@gmail.com> > > > > >> wrote: > > > > >> > > > > >>> Hi Viktor, > > > > >>> Thanks for taking this task! > > > > >>> This is a very nice change as it will allow users to use this > tool in > > > > >>> many Cloud environments where direct zookeeper access is not > > > > >>> available. > > > > >>> > > > > >>> > > > > >>> On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass > > > > >>> <viktorsomo...@gmail.com> wrote: > > > > >>> > > > > > >>> > Hi All, > > > > >>> > > > > > >>> > This is the continuation of the old KIP-375 with the same > title: > > > > >>> > > > > > >>> > > > > https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E > > > > >>> > > > > > >>> > The problem there was that two KIPs were created around the > same > > > time > > > > >>> and I > > > > >>> > chose to reorganize mine a bit and give it a new number to > avoid > > > > >>> > duplication. > > > > >>> > > > > > >>> > The KIP summary here once again: > > > > >>> > > > > > >>> > I wrote up a relatively simple KIP about improving the Kafka > > > protocol > > > > >>> and > > > > >>> > the TopicCommand tool to support the new Java based > AdminClient and > > > > >>> > hopefully to deprecate the Zookeeper side of it. > > > > >>> > > > > > >>> > I would be happy to receive some opinions about this. In > general I > > > > >>> think > > > > >>> > this would be an important addition as this is one of the few > left > > > but > > > > >>> > important tools that still uses direct Zookeeper connection. > > > > >>> > > > > > >>> > Here is the link for the KIP: > > > > >>> > > > > > >>> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient > > > > >>> > > > > > >>> > Cheers, > > > > >>> > Viktor > > > > >>> > > > > >> > > > >
