Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API

Hi Paolo, Replies in line... On 28 July 2017 at 11:14, Paolo Patierno wrote: > Hi committers, > > in my understanding there is the common idea to move all tools from Scala > to Java and then using the new Admin Client API instead of using the > Zookeeper connection. > >

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

July 2017 at 11:45, Tom Bentley <t.j.bent...@gmail.com> wrote: > I've updated the KIP to fix those niggles, but I've not factored out the > topic name from the ReplicaStatusRequest, yet. > > Looking at the topic creation APIs in more detail, the CreateTopicsOptions > has > >

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

Hi again Ismael, 1. It's worth emphasising that reassigning partitions is a different >> process than what happens when a topic is created, so not sure trying to >> make it symmetric is beneficial. In addition to what was already >> discussed, >> one should also enable replication throttling

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

t; TopicPartitionReplica, which identifies the topic, partition and the > brokerId. replicaStatus can then take a list of TopicPartitionReplica as > input. And its output maps the replica to replica status. The latter API > seems simpler and also matches the method name better. What do

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

ming no one can see any glaring holes in what I'm proposing here, or wants to suggest a workable alternative set of APIs and algorithms, then I'll update KIP-179 to this effect. Thanks for taking the time to read this far! Tom On 10 August 2017 at 11:56, Tom Bentley <t.j.bent...@gmail.c

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Jun and Dong, Thanks for your replies... On 10 August 2017 at 20:43, Dong Lin wrote: > This is a very good idea. I have updated the KIP-113 so that > DescribeDirResponse returns lag instead of LEO. Excellent! On Thu, Aug 10, 2017 at 10:21 AM, Jun Rao

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

Hi Jun, Thanks for your reply, I've got a few comment inline... On 11 August 2017 at 01:51, Jun Rao wrote: > Hi, Tom, > > Thanks for the KIP. Looks good overall. A few minor comments. > > 1. In most requests with topic partitions, we nest partitions inside topic. > So,

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

mail, I thought you were proposing to have > PartitionReassignmentRequest > dealing with both inter and intra broker data movement (i.e., include log > dirs in the request). Then, I am not sure how this request will be > processed on the broker. So, you were not proposing that? > > Thanks, > &

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

; wrote: > > > >> > > > > >> > > Hi, Dong, > > > >> > > > > > >> > > I think Tom was suggesting to have the AlterTopicsRequest sent > to > > > any > > > >> > > broker, which just write

Comments on JIRAs

The project recently switched from all JIRA events being sent to the dev mailling list, to just issue creations. This seems like a good thing because the dev mailling list was very noisy before, and if you want to see all the JIRA comments etc you can subscribe to the JIRA list. If you don't

Re: From Scala to Java based tools : joptsimple vs argparse4j

Hi Paolo, I don't have an opinion about which you should use, but I certainly agree that two option parsing dependencies appears to be 1 too many. Is there a reason why you prefer joptsimple? Cheers, Tom On 10 July 2017 at 08:38, Paolo Patierno wrote: > Hi devs, > > >

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

, Tom On 19 July 2017 at 16:08, Tom Bentley <t.j.bent...@gmail.com> wrote: > Ah, thank you! I took the number from the "Next KIP Number: 178" on the > KIP index and didn't check the tables. So this is now KIP-179. The old link > will point you to the right place. >

[DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

Aside: I've started this new DISCUSS thread for KIP-179 since the original one had the incorrect KIP number 178. The original thread can be found here: http://mail-archives.apache.org/mod_mbox/kafka-dev/201707.mbox/%3cCAMd5YszudP+-8z5KTbFh6JscT2p4xFi1=vzwwx+5dccpxry...@mail.gmail.com%3e I've just

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

smael > > On Thu, Jul 20, 2017 at 4:12 AM, Tom Bentley <t.j.bent...@gmail.com> > wrote: > > > Hi Ismael, > > > > I've been working on the progress reporting assuming that it would be > > acceptable for the ReassignPartitionsCommand to poll the AdminClie

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

would give the impression that it has not). On 21 July 2017 at 11:09, Tom Bentley <t.j.bent...@gmail.com> wrote: > Aside: I've started this new DISCUSS thread for KIP-179 since the original > one had the incorrect KIP number 178. The original thread can be found > he

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

gt; If we agree to do that, then we need to decide if it should be implemented > client-side or by adding a protocol API. The former is simpler, but the > latter would allow non Java clients to use it without duplicating the logic > of assigning replicas to the new partitions. Wh

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

API to track the progress of that replication, I'm inclined to think that having a timeout is a bit pointless. But should the replicaStatus() API have a timeout? I suppose it probably should. On 26 July 2017 at 10:58, Tom Bentley <t.j.bent...@gmail.com> wrote: > Tha

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

Thanks Paolo, * in the "Public Interfaces" section you wrote > alterTopics(Set) but then a collection is used (instead of a > set) in the Proposed Changes section. I'm ok with collection. > Agree it should be Collection. > * in the summary of the alterTopics method you say "The request

[DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi, I've written KIP-178 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-178+-+Change+ReassignPartitionsCommand+to+use+AdminClient) for changing the ReassignPartitionsCommand (a.k.a. kafka-reassign-partitions.sh)to remove the dependency on ZooKeeper and instead provide the same

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

Hi Ismael, Answers in-line: 1. Have you considered how progress would be reported? Partition > reassignment can take a long time and it would be good to have a mechanism > for progress reporting. > The ReassignPartitionsCommand doesn't currently have a mechanism to track progress. All you can

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

on can often be done in stages. > > Does that make sense? > > Ismael > > On Wed, Jul 19, 2017 at 6:23 AM, Tom Bentley <t.j.bent...@gmail.com> > wrote: > > > Hi Ismael, > > > > Answers in-line: > > > > 1. Have you considered how progress would b

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

ke there is already a KIP-178: > > KIP-178: Size-based log directory selection strategy > > Ismael > > On Wed, Jul 19, 2017 at 7:05 AM, Tom Bentley <t.j.bent...@gmail.com> > wrote: > > > OK, I will work on adding support for this to the KIP, with the intention &g

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

ler that was changing it, avoiding the possibility of races. For now I've kept this error code in the KIP, but happy to revise that if this interpretation is wrong. > -Ewen > > On Mon, Aug 7, 2017 at 8:21 AM, Tom Bentley <t.j.bent...@gmail.com> wrote: > > > Hi, > > > &

Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy

Hi Hu, I wonder whether changing, or configuring a size-balancing strategy would be sufficient for all users. I would expect that users might want to take other factors into account. For example, with KIP-113, balancing IO across the disks might also be a factor, in addition to balancing free

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

rth thinking whether having 2 separate protocol APIs would be > better. I can see pros and cons, so I'd be interested in what you and > others think. > > Ismael > > On Tue, Aug 1, 2017 at 10:19 AM, Tom Bentley <t.j.bent...@gmail.com> > wrote: > > > I have

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

> Regarding adding the possibility to alter the topic config through the > AlterTopic API, the current TopicCommand implementation provides a warning > on doing this suggesting to use the ConfigCommand tool. So it would be a > step back allowing to do the configs change with the alter topic as

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

ten include multiple error codes (due to batching). > > 3. In my opinion, the warning when someone updates configs via > `kafka-topic.sh` was a mistake. Doing the same thing via the configs tool > is quite clunky in comparison. > > Ismael > > On Tue, Aug 1, 2017 at 4:52 PM, To

[DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

In a similar vein to KIP-179 I've created KIP-183 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-183+-+Change+PreferredReplicaLeaderElectionCommand+to+use+AdminClient) which is about deprecating the --zookeeper option to kafka-preferred-replica-election.sh and replacing it with an option

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

> > > > Also, how do you think things would work in the context of KIP-179? Would > > the tool still invoke these requests or would it be done by the broker > > receiving the alterTopics/reassignPartitions protocol call? > > > > My gut feel is that the tool will still invoke these requests. But I

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Dong, Thanks for your reply. Yeah I agree with you that the total disk capacity can be useful > particularly if it is different across brokers but it is probably of > limited use in most cases. I also expect that most users would have their > own customized tool across to determine the new

Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

The KIP is here for any one, like me, who hasn't seen it yet: https://cwiki.apache.org/confluence/display/KAFKA/KIP-176:+Remove+deprecated+new-consumer+option+for+tools Paolo, the KIP says "On the next release cycle we could totally remove the option." Exactly which release are you proposing that

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Dong, Replies inline, as usual > As I originally envisaged it, KIP-179's support for reassigning partitions > > would have more-or-less taken the logic currently in the > > ReassignPartitionsCommand (that is, writing JSON to the > > ZkUtils.ReassignPartitionsPath) > > and put it behind a

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

nager a homogeneous cluster. If this is not the case then I think we > should include this information in the response. > > Thanks, > Dong > > > On Mon, Aug 7, 2017 at 3:44 AM, Tom Bentley <t.j.bent...@gmail.com> wrote: > > > Hi Dong, > > > > Your comments on KIP-17

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

ully caught up. I am wondering if it's better to instead > > return the lag in offset per replica. This way, the status can probably > be > > reported more reliably. > > > > Thanks, > > > > Jun > > > > On Tue, Aug 8, 2017 at 11:23 AM, Dong Lin <lindon...@gm

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

PI in AdminClient instead: > > > > AlterTopicsResult alterTopics(Map<TopicPartition, List> > > partitionAssignment, AlterTopicsOptions options) > > > > - Do you think "reassignPartitions" may be a better name than > > "alterTopics"? This is more co

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Dong, Your comments on KIP-179 prompted me to look at KIP-113, and I have a question: AFAICS the DescribeDirsResponse (via ReplicaInfo) can be used to get the size of a partition on a disk, but I don't see a mechanism for knowing the total capacity of a disk (and/or the free capacity of a

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

by topic name. This would make for smaller messages when triggering elections for multiple partitions of the same topic. I'd be grateful for any feedback you may have. Cheers, Tom On 2 August 2017 at 18:34, Tom Bentley <t.j.bent...@gmail.com> wrote: > In a similar vein to KIP-179 I've cr

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

t, any new > PreferredLeaderElectionRequest > will be responded immediately with a REPLICA_LEADER_ELECTION_IN_PROGRESS > error. > > Does this sound good to you? > > Thanks, > > Jun > > On Fri, Aug 11, 2017 at 4:55 AM, Tom Bentley <t.j.bent...@gmail.com> > wrote: > > > Hi Jun,

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

Hi Colin, Thanks for your input. A couple of comments inline. On 22 August 2017 at 21:42, Colin McCabe <cmcc...@apache.org> wrote: > Hi, > > Thanks for the KIP. It looks good overall. > > On Tue, Aug 22, 2017, at 08:54, Tom Bentley wrote: > > Hi Jun, >

Re: each file of the kafka data will be opened 70+ times

I think your lsof output is listing threads, not simply processes (if you look at the output instead of piping it to wc you'll see that the PID is the same and only the TID is different). If you use `lsof /opt/kafka/data/...` (i.e without the grep) instead it will just show the processes. On 22

Re: JIRA contributor list

According to the website [1] I need to ask to be able to assign JIRAs to myself, but I'm still unable to do this. Could someone set this up for me please? Thanks, Tom [1]: https://kafka.apache.org/contributing On 14 June 2017 at 13:43, Tom Bentley <t.j.bent...@gmail.com> wrote: > Ple

Re: Contributor

Please can I also be added? My username is tombentley. Thanks Tom On 21 June 2017 at 12:03, Damian Guy wrote: > Hi Andras, > > You should have access now. > > Thanks, > Damian > > On Wed, 21 Jun 2017 at 10:45 Andras Beni wrote: > > > Hi All, > >

Open PRs

I realise that 0.11.0.0 is imminent and so the committers are rightly going to be rather focussed on that, but I opened some PRs nearly a week ago and they don't seem to have been looked at. Even a comment on the PR to the effect of "We'll look at this right after 0.11.0.0" would at least

Re: Contributor

Thanks! On 21 Jun 2017 4:20 pm, "Damian Guy" <damian@gmail.com> wrote: > Done - thanks > > On Wed, 21 Jun 2017 at 12:19 Tom Bentley <t.j.bent...@gmail.com> wrote: > > > Please can I also be added? My username is tombentley. > > > > Thanks

Re: Mirroring documentation improvement

Hi Evgeniy, The wiki page you link to there is about contributing improvements for kafka.apache.org. But since you say "mirroring doc in Confluence" I assume you want to edit https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330, if so you will need one of the committers to

Re: Mirroring documentation improvement

> > My confluence username is evis. > > Best regards, > Evgeny > > С уважением, > Евгений Веретенников > > 2017-06-23 16:43 GMT+03:00 Tom Bentley <t.j.bent...@gmail.com>: > > > Hi Evgeniy, > > > > The wiki page you link to there is

Re: Mirroring documentation improvement

Hi Damian, my username is tombentley Thanks Tom On 23 June 2017 at 14:57, Damian Guy <damian@gmail.com> wrote: > Evgeniy, you should now have access. > > Tom & Paolo what are your wiki usernames? > > Thanks, > Damian > > On Fri, 23 Jun 2017 at 14:52

Re: Test fail reason on doc fix

Hi Paolo, Usually you can just follow the links added by asfbot on the PR, but these are currently giving 404 (and not just for your failures, so maybe an ASF infrastructure problem?) Cheers, Tom On 12 June 2017 at 09:20, Paolo Patierno wrote: > Hi all, > > > I opened

JIRA contributor list

Please could I be added to the JIRA contributor list so that I can assign issues to myself? Thanks, Tom

Re: [jira] [Commented] (KAFKA-5434) Console consumer hangs if not existing partition is specified

It seems to me that in the most common case the partition isn't going to be magically created and in that case it's more helpful to quit with an error than to sit there waiting for the partition to be created.

Improving tools: --help

Hi, I noticed that the command line tools could use a little love. For instance, I was surprised that most of them don't support `--help`, and generally there are a few inconsistencies. KIP-14 is dormant and AFAICS no one is working on https://issues.apache.org/jira/browse/KAFKA-2111 either. So

Re: Improving tools: --help

are close to 20 commands I have made changes to. I will try to push > something out by end of day today. So I think I will cover 1) from your > list below and not 2) and 3). > > Thanks Tom. > > Regards, > Mariam. > > > > [image: Inactive hide details for Tom Bentley -

Re: Improving tools: --help

> I just checked kafka-topics.sh and the only required argument there is > --zookeeper option. Not sure if you were thinking of some other command. > > I meant that for kafka-topics.sh --create requires --topic (amongst others), but --list does not. (For example, using these methods we can say

Re: [DISCUSS] KIP-190: Handle client-ids consistently between clients and brokers

Hi Mickael, I was just wondering why the restriction was imposed for Java clients the first place, do you know? Cheers, Tom On 14 September 2017 at 09:16, Ismael Juma wrote: > Thanks for the KIP Mickael. I suggest starting a vote. > > Ismael > > On Mon, Aug 21, 2017 at

Re: [VOTE] KIP-195: AdminClient.createPartitions()

open for a while longer, but it won't be in 1.0.0 even if the vote is eventually successful. Cheers, Tom On 13 September 2017 at 11:43, Ismael Juma <ism...@juma.me.uk> wrote: > Tom, > > Thanks for the KIP, +1 (binding) from me. > > Ismael > > On Fri, Sep 8, 2017 at 5:4

Re: KIP-203: Add toLowerCase support to sasl.kerberos.principal.to.local rule

What locale is used for the case conversion, the JVM default one or a specific one? On 18 Sep 2017 5:31 pm, "Manikumar" wrote: > Hi all, > > I've created a small KIP to extend the sasl.kerberos.principal.to.local > rule syntax to convert short names to lower case. > >

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Hi Paolo, Thanks for the KIP. What errors can be anticipated for the API method (See https://issues.apache.org/jira/browse/KAFKA-5445)? It's not mentioned in KIP-107, but maybe now is a good time to consider whether there should be some kind of DeleteRecordsPolicy, like there is for creating

Re: [VOTE] KIP-195: AdminClient.createPartitions()

t; specified by the user, what will be sent in the protocol? A null assignment > array? Probably worth mentioning this case explicitly in the KIP. > > Thanks, > Jason > > On Wed, Sep 13, 2017 at 7:53 AM, Tom Bentley <t.j.bent...@gmail.com> > wrote: > > >

Re: [DISCUSS] KIP-207: Offsets returned by ListOffsetsResponse should be monotonically increasing even during a partition leader change

Thanks Colin, it makes sense now, it was the HWM party I was missing. Cheers, Tom On 6 Oct 2017 6:44 pm, "Colin McCabe" <cmcc...@apache.org> wrote: On Thu, Oct 5, 2017, at 12:06, Tom Bentley wrote: > Hi Colin, > > Is it really true that "the period when th

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

ber of partitions). > > thanks > Edo > -- > > Edoardo Comar > > IBM Message Hub > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > From: Tom Bentley <t.j.bent...@gmail.com> > To: dev@kafka.apache.org > Date:

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

that can't send errors to clients is much less useful > 2. Testing policies is much easier with `validateOnly` > > Ismael > > On Wed, Oct 4, 2017 at 9:20 AM, Tom Bentley <t.j.bent...@gmail.com> wrote: > > > Thanks Edoardo, > > > > I've added that motivati

Re: [DISCUSS] KIP-59 : Proposal for a kafka broker command

Hi Jayesh, Thanks, for the KIP. I few questions/points: 1. Could you elaborate on the motivation a little? Currently it seems to boil down to "Kafka doesn't have this, yet", but that's not, in itself, a reason to add it. What can't be done without this change? 2. The second bullet in the

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

citly requested by the client. */ boolean generatedReplicaAssignments(); } Thoughts? On 4 October 2017 at 11:06, Tom Bentley <t.j.bent...@gmail.com> wrote: > Good point. Then I guess I can do those items too. I would also need to do > the same changes for DeleteRecordsRequest and Respons

Re: [DISCUSS] KIP-209 Connection String Support

Hi Clebert, The motivation section is written as more of a summary and doesn't really give any motivation for this change. Can you explain why it would be beneficial for Kafka to have this change? For example, if you have use cases where the current way of instantiating a producer, consumer or

[VOTE] KIP-201: Rationalising policy interfaces

I would like to start a vote on KIP-201, which proposes to replace the existing policy interfaces with a single new policy interface that also extends policy support to cover new and existing APIs in the AdminClient.

Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

Hi Ismael, Thanks for the comments. My bad for not noticing the custom assignment requirement. The current > proposal has the following example (I updated it a little so that 2 > partitions are added): > > increasePartitionCount(4, asList(asList(1, 2), asList(2, 3)) > > Why not simply provide

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

ther the leader is balanced for each > partition in ElectPreferredLeadersResult, instead of just being attempted? > > Thanks, > > Jun > > On Wed, Sep 6, 2017 at 4:03 PM, Colin McCabe <cmcc...@apache.org> wrote: > > > On Wed, Sep 6, 2017, at 01:18, Tom Bentley wrote: >

Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

at length as part of the reassign > partitions KIP, I suggest starting the vote tomorrow if there are no > objections from others. > I'll start the vote today unless anyone raises points that I can't address. Thanks, Tom > > Thanks, > Ismael > > On Thu, Sep 7, 2

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

I can't speak for the committers, but there's nothing to stop you submitting PRs against each others branches. It just needs you to agree which of you will host the integration branch. This would be pretty much exactly the same developer experience as of the branch was in the main Kafak repo

Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

>> >> Cheers >> >> On Fri, Sep 8, 2017 at 5:34 AM, Ismael Juma <ism...@juma.me.uk> wrote: >> >> > Hmm, maybe it should be createPartitions for symmetry with createTopics? >> > >> > Ismael >> > >> > On Fri, Sep

[VOTE] KIP-195: AdminClient.createPartitions()

I would like to start the vote on KIP-195 which adds an AdminClient API for increasing the number of partitions of a topic. The details are here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient.createPartitions Cheers, Tom

Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

cutive partition ids. > > +1 on not supporting non-consecutive partition ids > > Cheers > > On Fri, Sep 8, 2017 at 5:34 AM, Ismael Juma <ism...@juma.me.uk> wrote: > > > Hmm, maybe it should be createPartitions for symmetry with createTopics? > > > &

Re: [VOTE] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

no objections from the voted committer by end of the > day, I think you can mark it as accepted. > > > Guozhang > > > On Wed, Sep 6, 2017 at 7:42 AM, Tom Bentley <t.j.bent...@gmail.com> wrote: > > > Unfortunately I've had to make a small change to the > &

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

nded to mention the changes in the VOTE thread as a > heads up. Generally, we don't restart the vote unless the changes are > significant. > > Ismael > > On Wed, Aug 30, 2017 at 4:26 PM, Tom Bentley <t.j.bent...@gmail.com> > wrote: > > > Hi Ismael, > > > >

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

Hi Ted and Colin, Thanks for the comments. It seems you're both happier with reassign rather than assign, so I'm happy to stick with that. On 5 September 2017 at 18:46, Colin McCabe wrote: > ... > Do we expect that reducing the number of partitions will ever be >

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

Hi Colin, Thanks for taking the time to respond. On 5 September 2017 at 22:22, Colin McCabe wrote: > ... > Why does there need to be a map at all in the API? >From a purely technical PoV there doesn't, but doing something else would make the API inconsistent with other

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

eople have. On 30 August 2017 at 16:17, Tom Bentley <t.j.bent...@gmail.com> wrote: > Hi all, > > I've updated the KIP as follows: > > * remove the APIs supporting progress reporting in favour of the APIs > being implemented in KIP-113. > * added some APIs to co

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

thout the user having to re-run the tool. So, yes, > maybe it should be a separate KIP as well. > > Not sure if we need it in the template, but you're welcome to mention any > dependencies when there are some. > > Thanks, > Ismael > > On Thu, Sep 7, 2017 at 5:15 PM, Tom Bentle

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

ReassignPartitions is more complex and we can probably aim to include that > in the January release. What do you think? > > Ismael > > On Wed, Sep 6, 2017 at 11:42 PM, Colin McCabe <cmcc...@apache.org> wrote: > > > On Wed, Sep 6, 2017, at 00:20, Tom Bentley wr

[DISCUSS] KIP-195: AdminClient.increasePartitions

As suggested by Ismael, I've factored the increasePartitionCounts() API out of KIP-179 out into a separate KIP which hopefully can progress more quickly. https://cwiki.apache.org/confluence/display/KAFKA/KIP-195%3A+AdminClient.increasePartitions If you've looked at KIP-179 in the last few days

Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

2. About using the create topics policy, I'm not sure. Aside from the > naming issue, there's also the problem that the policy doesn't know if a > creation or update is taking place. This matters because one may not want > to allow the number of partitions to be changed after creation as it >

Re: [VOTE] KIP-195: AdminClient.createPartitions()

are reflected in the metadata cache. Cheers, Tom On 8 September 2017 at 17:42, Tom Bentley <t.j.bent...@gmail.com> wrote: > I would like to start the vote on KIP-195 which adds an AdminClient API > for increasing the number of partitions of a topic. The details are here: > > https:

Re: [VOTE] KIP-195: AdminClient.createPartitions()

Hi Jun, Thanks for the comments. On 12 September 2017 at 18:15, Jun Rao wrote: > Hi, Tom, > > Thanks for the KIP. +1. Just a couple of minor comments below. > > 1. The KIP has "INVALID_PARTITIONS (37) If the partition count was <= the > current partition count for the

Re: [VOTE] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

his so that we don't increase > the tech debt that already exists. > > Ismael > > On Mon, Sep 4, 2017 at 10:11 AM, Tom Bentley <t.j.bent...@gmail.com> > wrote: > > > Hi Jun, > > > > You're correct about those other expected errors. If it's OK to update >

Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and introduction of TopicDeletePolicy

Hi Edoardo, KIP-179 will support changing topic configurations via the AdminClient and this would open a loophole to avoid the current CreateTopicPolicy: 1. Create a topic that sticks to the policy 2. Modify the topic after creation to something which would have violated the policy. For this

Re: 1.0.0 KIPs Update

Would it be possible to add KIP-183 to the list too, please? Thanks, Tom On 6 September 2017 at 22:04, Guozhang Wang wrote: > Hi Vahid, > > Yes I have just added it while sending this email :) > > > Guozhang > > On Wed, Sep 6, 2017 at 1:54 PM, Vahid S Hashemian < >

Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

from me. > > Ismael > > On Tue, Sep 12, 2017 at 10:26 AM, Tom Bentley <t.j.bent...@gmail.com> > wrote: > > > 2. About using the create topics policy, I'm not sure. Aside from the > > > naming issue, there's also the problem that the policy doesn't know if >

[DISCUSS] KIP-201: Rationalising Policy interfaces

Hi, I'd like to start a discussion for KIP-201. The basic point is that new AdminClient APIs for modifying topics should have a configurable policy to allow the administrator to veto the modifications. Just adding a ModifyTopicPolicy would make for awkwardness by having separate policies for

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Ted, Thanks for the feedback! bq. topic.action.policy.class.name > > Since the policy would cover more than one action, how about using actions > for the second word ? > Good point, done. > For TopicState interface, the abstract modifier for its methods are not > needed. > Fixed. bq.

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Senior Software Engineer (IoT) @ Red Hat > Microsoft MVP on Azure & IoT > Microsoft Azure Advisor > > Twitter : @ppatierno<http://twitter.com/ppatierno> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> > Blog : DevExperience<http://paolopatie

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Edoardo, what about a single method in ClusterState > > interface ClusterState { > public Map topicsState(); > > } > > which could return a read-only snapshot of the cluster metadata ? > Sure that would work too. A concern with that is that we end up

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

esponse > (to tell why it's failed). > > I'm happy if you incorporate the enhancements to create/alter that allow a > > check against the cluster metadata > and leave out - to anther KIP, or maybe I'll rewrite 170 the changes to > delete. > > thanks > Edo > > ---

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

I'm starting to think about KIP-179 again. In order to have more manageably-scoped KIPs and PRs I think it might be worth factoring-out the throttling part into a separate KIP. Wdyt? Keeping the throttling discussion in this thread for the moment... The throttling behaviour is currently spread

Re: [DISCUSS] KIP-204 : adding records deletion operation to the new Admin Client API

Hi Edoardo and Paolo, On 26 September 2017 at 14:10, Paolo Patierno wrote: > What could be useful use cases for having a RecordsDeletePolicy ? Records > can't be deleted for a topic name ? Starting from a specific offset ? > I can imagine some users wanting to prohibit

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

ng between an operation and the policy. > That > > > is, we initially thought we would have create/alter/delete topics, but > > that > > > didn't work out as the alter case is better served by multiple request > > > types. Given that, it's a bit awkward to m

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

ul for a broad range of use cases, rather than being too narrowly focussed on what's needed by the existing CLI tools. Thanks, Tom On 28 September 2017 at 17:22, Tom Bentley <t.j.bent...@gmail.com> wrote: > I'm starting to think about KIP-179 again. In order to have more > manage

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Mickael, Thanks for the reply. Thanks for the KIP. Is this meant to superseed KIP-170 ? > If so, one of our key requirements was to be able to access the > topics/partitions list from the policy, so an administrator could > enforce a partition limit for example. > It's not meant to replace

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Ismael, On 25 September 2017 at 17:51, Ismael Juma wrote: > We don't have this policy today for what it's worth. > Thanks for the clarification. On re-reading I realise I misinterpreted Guozhang Wang's suggestion when 1.0.0 was first mooted: > Just to clarify, my

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

terState that the server will pass > to the policy.validate method > would just lazily tap into MetadataCache. No need for big upfront > allocations. > > ciao, > Edo > -- > > Edoardo Comar > > IBM Message Hub >

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Ismael, Thanks for looking at the KIP and explaining the thinking behind the original API. Looking at the updated KIP, I notice that we actually have a > TopicDeletionPolicy with a separate config. That seems to be a halfway > house. Not sure about that. > I can certainly see that point of

  1   2   3   4   5   6   >