Hi,
I was digging into it because I need something like an Admin Client alter API for my work on rewriting the TopicCommand tool using them. The AlterConfigs API is used for changing topic level configuration (i.e. retention.ms, retention.bytes and so on). A new AlterTopic API could be better in order to change topic "high level" structure so number of partitions, replication factors and so on. My opinion is that we need separate API because from my point of view they are different settings. Thanks, Paolo. Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoT Microsoft Azure Advisor Twitter : @ppatierno<http://twitter.com/ppatierno> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> Blog : DevExperience<http://paolopatierno.wordpress.com/> ________________________________ From: Tom Bentley <t.j.bent...@gmail.com> Sent: Tuesday, July 25, 2017 9:02 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient Hi Ismael, I assume that's the --partitions for kafka-topics.sh? I must admit I hadn't considered that tool, only kafka-reassign-partitions.sh. Thanks for pointing it out, because obviously the AdminClient API needs to be suitable for reuse in kafka-topics.sh too. Since AdminClient doesn't currently support an alterTopics() I guess the API should mirror the API for newTopics(), so something like: public AlterTopicsResult alterTopics(Set<AlteredTopic> topics); // where: public class AlteredTopic { // constructor etc public String name(); public int numPartitions(); public int replicationFactor(); Map<Integer,List<Integer>> replicasAssignment(); } Note that although NewTopic contains a `Map<String,String> configs`, I think the API for changing a topic's config already exists: alterConfigs(). This API is better than having separate methods to set the number of partitions/replicas and assigning them to brokers, since sometimes people will want to set the assignment at the same time as changing the partitions/replicas. An API like this could then be used by both tools. On 24 July 2017 at 16:23, Ismael Juma <ism...@juma.me.uk> wrote: > Hi Tom, > > I think it makes sense to keep progress reporting simple in the initial > version. As you say, time to completion is tricky to compute and it seems > like it should be tackled as its own KIP. > > Regarding waiting for reassignment completion, it's easy enough for people > to do that via a script, so I don't think we need to add it to the tool. > > One question that occurred to me, the tool allows one to add partitions to > existing topics: > > val partitionsOpt = parser.accepts("partitions", "The number of partitions > for the topic being created or " + > "altered (WARNING: If partitions are increased for a topic that has a > key, the partition logic or ordering of the messages will be affected") > > It seems like it may make sense to have that as an AdminClient API as well. > If we agree to do that, then we need to decide if it should be implemented > client-side or by adding a protocol API. The former is simpler, but the > latter would allow non Java clients to use it without duplicating the logic > of assigning replicas to the new partitions. What are your thoughts? > > Ismael > > On Sat, Jul 22, 2017 at 10:14 AM, Tom Bentley <t.j.bent...@gmail.com> > wrote: > > > Thinking about this some more, I release that the proposed API for > tracking > > progress is a bit specific to this reassignment use case. A more > generally > > useful API would be to be able to find out, for a partition on a broker: > > > > * When the broker became a follower > > * When the broker was last in the ISR > > * If the broker is not in the ISR, how many messages behind it is > > > > That's not enough to be able to calculate a percentage completion of a > > reassignment though (we don't know how far behind it was when it became a > > follower). But maybe we could maintain a maximum of how far behind it has > > fallen behind the ISR, since it became a follower/dropped out of the ISR. > > > > btw, we need the middle bullet to cope with following sequence: > > > > 1. Start syncing > > 2. Finish syncing > > 3. Fall behind (drop out of ISR for some reason) > > 4. User queries for if reassignment has finished (it has, but just > looking > > at the ISR would give the impression that it has not). > > > > > > > > > > On 21 July 2017 at 11:09, Tom Bentley <t.j.bent...@gmail.com> wrote: > > > > > Aside: I've started this new DISCUSS thread for KIP-179 since the > > original > > > one had the incorrect KIP number 178. The original thread can be found > > > here: http://mail-archives.apache.org/mod_mbox/kafka-dev/201707. > > > mbox/%3cCAMd5YszudP+-8z5KTbFh6JscT2p4xFi1=VZWWX+ > > > 5dccpxry...@mail.gmail.com%3e > > > > > > I've just updated KIP-179 to support Ismael's request for the command > to > > > be able to support progress reporting of an ongoing partition > > reassignment. > > > > > > I'll call out two things which I'm not sure about since I don't yet > have > > > much experience of Kafka being used operationally: > > > > > > 1. As currently constructed the --progress option could report an > overall > > > progress percentage, per-partition percentages and errors. It cannot > > > provide any kind of time-to-completion estimate. Part of me is loath to > > do > > > this, as I'm sure we all remember file transfer dialogs that provide > > > amusing/baffling time-to-completion estimates. So it might be hard to > do > > > _well_. On the other hand I expect the thing people will be interested > in > > > will often be "when will it be finished?" > > > > > > 2. There is no option for the tool to wait for reassignment > completion. I > > > can imagine users might want to script something to happen after the > > > reassignment is complete, and without some kind of --wait option they > > will > > > have to poll for completion "manually". Having a --wait optin and > putting > > > this polling in the tool means we have a lot more control over how > often > > > such polling happens. > > > > > > The KIP is available here: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-179+-+Change+ > > > ReassignPartitionsCommand+to+use+AdminClient > > > > > > Thanks, > > > > > > Tom > > > > > >
