Just to add: I wasn't too pushy by raising a KIP for this as so far I had the experience that the community is fine with them or at least people learned to live with the current tools but if there's a need I'd be happy to jump back working on it or helping you if you have time :)
On Wed, May 8, 2019 at 11:35 AM Viktor Somogyi-Vass <viktorsomo...@gmail.com> wrote: > Hi Sönke, > > In KAFKA-1774 I have created a Kafka Shell java tool that unfortunately > didn't get much attention so far from the creators of the jira. It works > similarly to what Colin mentioned, like "kafka.sh topics create -n my-topic > -p 3 -r 3" or has long names like "kafka.sh topics create --name my-topic > --partitions 3 --replicas 3". The bootstrap server everywhere defaults to > :9092 or reads up the configs from a path so in many scenarios you can omit > it, also it uses the admin client of course and all in all provides a much > better experience than the current tools. It has interactive mode and help > too. Wanted to implement "code completion" too but for that I'd have to > exercise the code a little bit more :). > It is currently based on a quite old trunk but if you think it's > interesting I can rebase it and we can continue with raising a KIP. > https://github.com/viktorsomogyi/kafka/tree/kafka_shell > > Viktor > > On Tue, May 7, 2019 at 11:10 AM Sönke Liebau > <soenke.lie...@opencore.com.invalid> wrote: > >> Hi Colin, >> >> thanks! >> While I personally don't like the current command line tools I >> realistically think we'll be stuck with them for a while yet, so a cleanup >> might make sense. >> So I'll start looking into that. >> >> Regarding a central entrypoint, that would indeed be brilliant and I'd >> love >> to work on that, but I currently have enough other open things to look at, >> so I won't draw that one to myself as well for now. >> >> But I'll keep it in mind for when some time frees up :) >> >> Best regards, >> Sönke >> >> >> >> Colin McCabe <cmcc...@apache.org> schrieb am Di., 7. Mai 2019, 00:56: >> >> > On Mon, May 6, 2019, at 10:21, Sönke Liebau wrote: >> > > Hi Colin, >> > > >> > > it was my intention to keep the structure of the commands mostly >> intact >> > > while doing the refactoring - if that is possible, have not really >> > checked >> > > yet to be honest. >> > > >> > > But what I wanted to try and do is recreate the current parsing with >> > > argparse as much as possible. And in the process simply adding >> synonyms, >> > > for example make the kafka-console-producer understand a >> > > bootstrap-parameter in addition to broker-list. >> > > There is a bit of custom logic about which parameters go together >> etc. in >> > > the current classes, so output may look different here and there, but >> in >> > > principle I do believe that it should be possible to recreate the >> current >> > > structure. >> > >> > Sounds like a good idea. Thanks for the clarification. >> > >> > > >> > > If there is an appetite for a new, hadoop-like entrypoint anyway, then >> > all >> > > of this might be "wasted" effort, or rather effort better spent >> though, >> > you >> > > are right. >> > >> > I don't think anyone is working on a new entry point right now -- or if >> > they are, they haven't said anything yet :) >> > >> > I just wanted to mention it as a possible approach in case you wanted to >> > do a bigger project. >> > >> > best, >> > Colin >> > >> > > >> > > Best regards, >> > > Sönke >> > > >> > > >> > > >> > > On Mon, May 6, 2019 at 7:13 PM Colin McCabe <cmcc...@apache.org> >> wrote: >> > > >> > > > Hi Sönke, >> > > > >> > > > #2 is a bit tough because people have come to rely on the way the >> > commands >> > > > are structured right now. >> > > > >> > > > If we want to make big changes, it might be easier just to create a >> > > > separate tool and deprecate the old one(s). One thing we've talked >> > about >> > > > doing in the past is creating a single entry point for all the tool >> > > > functionality, kind of like hadoop did with the "hadoop" command Or >> > git >> > > > with the "git" command, etc. Then we could deprecate the standalone >> > > > commands and remove them after enough time had passed-- kind of like >> > the >> > > > old consumer. >> > > > >> > > > On the other hand, a more incremental change would be standardizing >> > flags >> > > > a bit. So for example, at least setting it up so that there is a >> > standard >> > > > way of supplying bootstrap brokers, etc. We could keep the old >> flags >> > > > around for a while as variants to ease the transition. >> > > > >> > > > best, >> > > > Colin >> > > > >> > > > >> > > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote: >> > > > > Hi Colin, >> > > > > >> > > > > I totally agree! Especially the differently named bootstrap server >> > > > options >> > > > > have been annoying me a long time. >> > > > > >> > > > > I'd propose a two-step approach: >> > > > > 1. Add new default options objects similar to CommandLineUtils and >> > > > > CommandDefaultOptions (based on argparse4j) but in the clients >> > project, >> > > > as >> > > > > this is referenced by all command line tools as far as I can tell >> > > > > 2. Refactor tools one by one to use these new helper classes (and >> > thus >> > > > > argparse) and add standardized synonyms for parameters as >> necessary >> > > > > >> > > > > I think for step 1 we can get away with no KIP, as this doesn't >> > change >> > > > any >> > > > > public interfaces or behavior. >> > > > > Step 2 probably needs a KIP as we are adding new parameters? We >> can >> > pick >> > > > up >> > > > > KIP-14 again for that I think. A lot of work has been done on that >> > > > already. >> > > > > >> > > > > Does that sound useful to everybody? >> > > > > >> > > > > Best regards, >> > > > > Sönke >> > > > > >> > > > > >> > > > > On Thu, Apr 18, 2019 at 1:44 AM Colin McCabe <cmcc...@apache.org> >> > wrote: >> > > > > >> > > > > > If we are going to standardize on one argument parsing library, >> it >> > > > should >> > > > > > certainly be argparse4j, I think. >> > > > > > argparse4j is simply a better argument parsing library with >> > support >> > > > for >> > > > > > more features. One example is mutually exclusive options. >> > argparse4j >> > > > > > supports this with MutuallyExclusiveGroup. jopt doesn't support >> > this, >> > > > so >> > > > > > when it is needed, we have to add extra code to manually check >> that >> > > > > > mutually exclusive options are not set. >> > > > > > >> > > > > > argparse4j also has subcommands. If you want something like >> "git >> > add" >> > > > > > with some set of flags, and "git remove" with another, you can >> do >> > this >> > > > with >> > > > > > argparse4j, but not with jopt. This would be very helpful for >> > > > clearing up >> > > > > > confusion in a lot of our shell scripts which have accumulated >> > dozens >> > > > of >> > > > > > arguments, most of which are only relevant to a very specific >> > > > operation. >> > > > > > But you just can't do it with jopt. >> > > > > > >> > > > > > Just to give an example, argparse4j with subcommands would allow >> > you to >> > > > > > run something like ./kafka-topics.sh list --help and get just >> > options >> > > > that >> > > > > > were relevant for listing topics, not the full dozens of options >> > that >> > > > might >> > > > > > relate to adding topics, removing them, etc. >> > > > > > >> > > > > > To be honest, though, what would help users the most is >> > standardizing >> > > > the >> > > > > > option flags across tools. We should have a standard way of >> > specifying >> > > > > > bootstrap brokers, for example. (We can continue to support the >> > old >> > > > > > synonyms for a while, of course.) >> > > > > > >> > > > > > best, >> > > > > > Colin >> > > > > > >> > > > > > >> > > > > > On Wed, Apr 17, 2019, at 08:56, Guozhang Wang wrote: >> > > > > > > I took another look at the PR itself and I think it would be >> > great to >> > > > > > have >> > > > > > > this cleanup too -- I cannot remember at the beginning why we >> > > > gradually >> > > > > > > moved to different mechanism (argparse4j) for different cmds, >> if >> > > > there's >> > > > > > no >> > > > > > > rationales behind it we should just make them consistent. >> > > > > > > >> > > > > > > Thanks for driving this! >> > > > > > > >> > > > > > > Guozhang >> > > > > > > >> > > > > > > On Wed, Apr 17, 2019 at 7:19 AM Ryanne Dolan < >> > ryannedo...@gmail.com> >> > > > > > wrote: >> > > > > > > >> > > > > > > > Sönke, I'd find this very helpful. It's annoying to keep >> track >> > of >> > > > which >> > > > > > > > commands use which form -- I always seem to guess wrong. >> > > > > > > > >> > > > > > > > Though I don't think there is any reason to deprecate >> existing >> > > > forms, >> > > > > > e.g. >> > > > > > > > consumer.config vs consumer-config. I think it's perfectly >> > > > reasonable >> > > > > > to >> > > > > > > > have multiple spellings of the same arguments. I don't >> really >> > see a >> > > > > > > > downside to keeping the aliases around indefinitely. >> > > > > > > > >> > > > > > > > Ryanne >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau >> > > > > > > > <soenke.lie...@opencore.com.invalid> wrote: >> > > > > > > > >> > > > > > > > > Hi everybody, >> > > > > > > > > >> > > > > > > > > Jason and I were recently discussing command line argument >> > > > parsing on >> > > > > > > > > KAFKA-8131 (or rather the related pull request) [1]. >> > > > > > > > > >> > > > > > > > > Command line tools and their arguments are somewhat >> diverse >> > at >> > > > the >> > > > > > > > moment. >> > > > > > > > > Most of the tools use joptsimple for argument parsing, >> some >> > newer >> > > > > > java >> > > > > > > > > tools use argparse4j instead and some tools use nothing at >> > all. >> > > > > > > > > I've looked for a reason as to why there are two libraries >> > being >> > > > > > used, >> > > > > > > > but >> > > > > > > > > couldn't really find anything. Paolo brought up the same >> > > > question on >> > > > > > the >> > > > > > > > > mailing list a while back [7], but got no response either. >> > > > > > > > > Does anybody know why this is the case? >> > > > > > > > > >> > > > > > > > > This results in no central place to add universal >> parameters >> > like >> > > > > > help >> > > > > > > > and >> > > > > > > > > version, as well as the help output looking different >> between >> > > > some >> > > > > > of the >> > > > > > > > > tools. >> > > > > > > > > Also, there are a number of parameters that should be >> > renamed to >> > > > > > adhere >> > > > > > > > to >> > > > > > > > > defaults. >> > > > > > > > > >> > > > > > > > > There have been a few discussions and initiatives around >> > this in >> > > > the >> > > > > > > > past. >> > > > > > > > > Just of the top of my head (and a 5 minute jira search) >> there >> > > > are: >> > > > > > > > > - KIP-14 [2] >> > > > > > > > > - KAFKA-2111 [3] >> > > > > > > > > - KIP-316 [4] >> > > > > > > > > - KAFKA-1292 [5] >> > > > > > > > > - KAFKA-3530 [6] >> > > > > > > > > - and probably many more >> > > > > > > > > >> > > > > > > > > Would people generally be in favor of revisiting this >> topic? >> > > > > > > > > >> > > > > > > > > What I'd propose to do is: >> > > > > > > > > - comb through jira and KIPs, clean up old stuff and >> creae a >> > new >> > > > > > umbrella >> > > > > > > > > issue to track this (maybe reuse KIP-4 as well) >> > > > > > > > > - agree on one library for parsing command line arguments >> > (don't >> > > > care >> > > > > > > > which >> > > > > > > > > one, but two is one too many I think) >> > > > > > > > > - refactor tools to use one library and default way of >> > argument >> > > > > > parsing >> > > > > > > > > with central help and version parameter >> > > > > > > > > - add aliases for options that should be renamed >> according to >> > > > KIP-4 >> > > > > > (and >> > > > > > > > > maybe others) so that both new and old work for a while, >> > > > deprecate >> > > > > > old >> > > > > > > > > parameters for a cycle or two and then remove them >> > > > > > > > > >> > > > > > > > > I'll shut up now and see if people would consider this >> > useful or >> > > > > > have any >> > > > > > > > > other input :) >> > > > > > > > > >> > > > > > > > > Best regards, >> > > > > > > > > Sönke >> > > > > > > > > >> > > > > > > > > [1] >> > > > https://github.com/apache/kafka/pull/6481#discussion_r273773003 >> > > > > > > > > <https://issues.apache.org/jira/browse/KAFKA-8131> >> > > > > > > > > [2] >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > >> > > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization >> > > > > > > > > [3] https://issues.apache.org/jira/browse/KAFKA-2111 >> > > > > > > > > [4] >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > >> > > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-316%3A+Command-line+overrides+for+ConnectDistributed+worker+properties >> > > > > > > > > [5] https://issues.apache.org/jira/browse/KAFKA-1292 >> > > > > > > > > [6] https://issues.apache.org/jira/browse/KAFKA-3530 >> > > > > > > > > [7] >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > >> > > > >> > >> https://sematext.com/opensee/m/Kafka/uyzND10ObP01p77VS?subj=From+Scala+to+Java+based+tools+joptsimple+vs+argparse4j >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > -- >> > > > > > > -- Guozhang >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > Sönke Liebau >> > > > > Partner >> > > > > Tel. +49 179 7940878 >> > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - >> Germany >> > > > > >> > > > >> > > >> > > >> > > -- >> > > Sönke Liebau >> > > Partner >> > > Tel. +49 179 7940878 >> > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany >> > > >> > >> >
