Would it be possible to add KIP-183 to the list too, please?
Thanks,
Tom
On 6 September 2017 at 22:04, Guozhang Wang wrote:
> Hi Vahid,
>
> Yes I have just added it while sending this email :)
>
>
> Guozhang
>
> On Wed, Sep 6, 2017 at 1:54 PM, Vahid S Hashemian <
>
his so that we don't increase
> the tech debt that already exists.
>
> Ismael
>
> On Mon, Sep 4, 2017 at 10:11 AM, Tom Bentley <t.j.bent...@gmail.com>
> wrote:
>
> > Hi Jun,
> >
> > You're correct about those other expected errors. If it's OK to update
>
Hi Colin,
Thanks for taking the time to respond.
On 5 September 2017 at 22:22, Colin McCabe wrote:
> ...
> Why does there need to be a map at all in the API?
>From a purely technical PoV there doesn't, but doing something else would
make the API inconsistent with other
Hi Ted and Colin,
Thanks for the comments.
It seems you're both happier with reassign rather than assign, so I'm happy
to stick with that.
On 5 September 2017 at 18:46, Colin McCabe wrote:
> ...
> Do we expect that reducing the number of partitions will ever be
>
Hi Edoardo,
KIP-179 will support changing topic configurations via the AdminClient and
this would open a loophole to avoid the current CreateTopicPolicy:
1. Create a topic that sticks to the policy
2. Modify the topic after creation to something which would have violated
the policy.
For this
eople have.
On 30 August 2017 at 16:17, Tom Bentley <t.j.bent...@gmail.com> wrote:
> Hi all,
>
> I've updated the KIP as follows:
>
> * remove the APIs supporting progress reporting in favour of the APIs
> being implemented in KIP-113.
> * added some APIs to co
partitions argument was null, and if there was an
error let the exception propagate out of electPreferredLeaders() directly.
Sorry about having to ask about this once people have already voted, but
what do people think about these options?
Thanks,
Tom
On 30 August 2017 at 16:55, Tom Bentley <t.j.bent...@gm
this, but it would be helpful for people using the
AdminClient to know the kinds of errors they should expect, for testing
purposes for example. On the other hand it's a maintenance burden. Should
we start documenting likely errors like this?
Cheers,
Tom
On 4 September 2017 at 10:10, Tom Bentley
gress exception, (3) can't move to
> > the preferred replica exception (i.e., preferred replica not in sync
> yet).
> >
> > Jun
> >
> > On Tue, Aug 29, 2017 at 8:56 AM, Tom Bentley <t.j.bent...@gmail.com>
> > wrote:
> >
> > > Hi all,
Hi Viktor,
KIP-179's reassignPartitions() API already has a validateOnly() option.
There's no technical barrier to adding similar options to the other
state-changing APIs in that KIP, if people think it would be useful. Please
comment to say this on the [DISCUSS] thread for the KIP though.
In
nded to mention the changes in the VOTE thread as a
> heads up. Generally, we don't restart the vote unless the changes are
> significant.
>
> Ismael
>
> On Wed, Aug 30, 2017 at 4:26 PM, Tom Bentley <t.j.bent...@gmail.com>
> wrote:
>
> > Hi Ismael,
> >
> >
er`. The next
> point is that this is a batch API, so it should ideally be plural like the
> other AdminClient methods. Maybe `electPreferredReplicasAsLeaders`, but
> that's quite a mouthful. Maybe we should shorten it to
> `electPreferredLeaders`. Thoughts?
>
> Ismael
>
> On Wed,
es
that the options make sense for all alterations.
Cheers,
Tom
On 22 August 2017 at 23:49, Colin McCabe <cmcc...@apache.org> wrote:
> On Wed, Aug 9, 2017, at 07:17, Tom Bentley wrote:
> > Hi Dong and Jun,
> >
> > Thanks again for your input in this discussion and on KIP
Hi all,
I would like to start the vote on KIP-183 which will provide an AdminClient
interface for electing the preferred replica, and refactor the
kafka-preferred-replica-election.sh tool to use this interface. More
details here:
Hi Colin,
Thanks for your input. A couple of comments inline.
On 22 August 2017 at 21:42, Colin McCabe <cmcc...@apache.org> wrote:
> Hi,
>
> Thanks for the KIP. It looks good overall.
>
> On Tue, Aug 22, 2017, at 08:54, Tom Bentley wrote:
> > Hi Jun,
>
I think your lsof output is listing threads, not simply processes (if you
look at the output instead of piping it to wc you'll see that the PID is
the same and only the TID is different). If you use `lsof
/opt/kafka/data/...` (i.e without the grep) instead it will just show the
processes.
On 22
t, any new
> PreferredLeaderElectionRequest
> will be responded immediately with a REPLICA_LEADER_ELECTION_IN_PROGRESS
> error.
>
> Does this sound good to you?
>
> Thanks,
>
> Jun
>
> On Fri, Aug 11, 2017 at 4:55 AM, Tom Bentley <t.j.bent...@gmail.com>
> wrote:
>
> > Hi Jun,
mail, I thought you were proposing to have
> PartitionReassignmentRequest
> dealing with both inter and intra broker data movement (i.e., include log
> dirs in the request). Then, I am not sure how this request will be
> processed on the broker. So, you were not proposing that?
>
> Thanks,
>
&
Hi Jun and Dong,
Thanks for your replies...
On 10 August 2017 at 20:43, Dong Lin wrote:
> This is a very good idea. I have updated the KIP-113 so that
> DescribeDirResponse returns lag instead of LEO.
Excellent!
On Thu, Aug 10, 2017 at 10:21 AM, Jun Rao
Hi Jun,
Thanks for your reply, I've got a few comment inline...
On 11 August 2017 at 01:51, Jun Rao wrote:
> Hi, Tom,
>
> Thanks for the KIP. Looks good overall. A few minor comments.
>
> 1. In most requests with topic partitions, we nest partitions inside topic.
> So,
ming no one can see any glaring holes in what I'm proposing here, or
wants to suggest a workable alternative set of APIs and algorithms, then
I'll update KIP-179 to this effect.
Thanks for taking the time to read this far!
Tom
On 10 August 2017 at 11:56, Tom Bentley <t.j.bent...@gmail.c
; wrote:
> > > >> >
> > > >> > > Hi, Dong,
> > > >> > >
> > > >> > > I think Tom was suggesting to have the AlterTopicsRequest sent
> to
> > > any
> > > >> > > broker, which just write
PI in AdminClient instead:
> >
> > AlterTopicsResult alterTopics(Map<TopicPartition, List>
> > partitionAssignment, AlterTopicsOptions options)
> >
> > - Do you think "reassignPartitions" may be a better name than
> > "alterTopics"? This is more co
ully caught up. I am wondering if it's better to instead
> > return the lag in offset per replica. This way, the status can probably
> be
> > reported more reliably.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Aug 8, 2017 at 11:23 AM, Dong Lin <lindon...@gm
Hi Hu,
I wonder whether changing, or configuring a size-balancing strategy would
be sufficient for all users. I would expect that users might want to take
other factors into account. For example, with KIP-113, balancing IO across
the disks might also be a factor, in addition to balancing free
ler that was changing it, avoiding the possibility
of races.
For now I've kept this error code in the KIP, but happy to revise that if
this interpretation is wrong.
> -Ewen
>
> On Mon, Aug 7, 2017 at 8:21 AM, Tom Bentley <t.j.bent...@gmail.com> wrote:
>
> > Hi,
> >
> &
Hi Dong,
Replies inline, as usual
> As I originally envisaged it, KIP-179's support for reassigning partitions
>
> would have more-or-less taken the logic currently in the
> > ReassignPartitionsCommand (that is, writing JSON to the
> > ZkUtils.ReassignPartitionsPath)
> > and put it behind a
The KIP is here for any one, like me, who hasn't seen it yet:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-176:+Remove+deprecated+new-consumer+option+for+tools
Paolo, the KIP says "On the next release cycle we could totally remove the
option." Exactly which release are you proposing that
Hi Dong,
Thanks for your reply.
Yeah I agree with you that the total disk capacity can be useful
> particularly if it is different across brokers but it is probably of
> limited use in most cases. I also expect that most users would have their
> own customized tool across to determine the new
> >
> > Also, how do you think things would work in the context of KIP-179? Would
> > the tool still invoke these requests or would it be done by the broker
> > receiving the alterTopics/reassignPartitions protocol call?
> >
>
> My gut feel is that the tool will still invoke these requests. But I
nager a homogeneous cluster. If this is not the case then I think we
> should include this information in the response.
>
> Thanks,
> Dong
>
>
> On Mon, Aug 7, 2017 at 3:44 AM, Tom Bentley <t.j.bent...@gmail.com> wrote:
>
> > Hi Dong,
> >
> > Your comments on KIP-17
by topic name. This would make for smaller messages when triggering
elections for multiple partitions of the same topic.
I'd be grateful for any feedback you may have.
Cheers,
Tom
On 2 August 2017 at 18:34, Tom Bentley <t.j.bent...@gmail.com> wrote:
> In a similar vein to KIP-179 I've cr
Hi Dong,
Your comments on KIP-179 prompted me to look at KIP-113, and I have a
question:
AFAICS the DescribeDirsResponse (via ReplicaInfo) can be used to get the
size of a partition on a disk, but I don't see a mechanism for knowing the
total capacity of a disk (and/or the free capacity of a
t; TopicPartitionReplica, which identifies the topic, partition and the
> brokerId. replicaStatus can then take a list of TopicPartitionReplica as
> input. And its output maps the replica to replica status. The latter API
> seems simpler and also matches the method name better. What do
In a similar vein to KIP-179 I've created KIP-183 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-183+-+Change+PreferredReplicaLeaderElectionCommand+to+use+AdminClient)
which is about deprecating the --zookeeper option to
kafka-preferred-replica-election.sh and replacing it with an option
Tom Bentley created KAFKA-5693:
--
Summary: TopicCreationPolicy and AlterConfigsPolicy overlap
Key: KAFKA-5693
URL: https://issues.apache.org/jira/browse/KAFKA-5693
Project: Kafka
Issue Type: Bug
Tom Bentley created KAFKA-5692:
--
Summary: Refactor PreferredReplicaLeaderElectionCommand to use
AdminClient
Key: KAFKA-5692
URL: https://issues.apache.org/jira/browse/KAFKA-5692
Project: Kafka
Hi again Ismael,
1. It's worth emphasising that reassigning partitions is a different
>> process than what happens when a topic is created, so not sure trying to
>> make it symmetric is beneficial. In addition to what was already
>> discussed,
>> one should also enable replication throttling
ten include multiple error codes (due to batching).
>
> 3. In my opinion, the warning when someone updates configs via
> `kafka-topic.sh` was a mistake. Doing the same thing via the configs tool
> is quite clunky in comparison.
>
> Ismael
>
> On Tue, Aug 1, 2017 at 4:52 PM, To
> Regarding adding the possibility to alter the topic config through the
> AlterTopic API, the current TopicCommand implementation provides a warning
> on doing this suggesting to use the ConfigCommand tool. So it would be a
> step back allowing to do the configs change with the alter topic as
rth thinking whether having 2 separate protocol APIs would be
> better. I can see pros and cons, so I'd be interested in what you and
> others think.
>
> Ismael
>
> On Tue, Aug 1, 2017 at 10:19 AM, Tom Bentley <t.j.bent...@gmail.com>
> wrote:
>
> > I have
July 2017 at 11:45, Tom Bentley <t.j.bent...@gmail.com> wrote:
> I've updated the KIP to fix those niggles, but I've not factored out the
> topic name from the ReplicaStatusRequest, yet.
>
> Looking at the topic creation APIs in more detail, the CreateTopicsOptions
> has
>
>
Hi Paolo,
Replies in line...
On 28 July 2017 at 11:14, Paolo Patierno wrote:
> Hi committers,
>
> in my understanding there is the common idea to move all tools from Scala
> to Java and then using the new Admin Client API instead of using the
> Zookeeper connection.
>
>
API to track the progress of that
replication, I'm inclined to think that having a timeout is a bit pointless.
But should the replicaStatus() API have a timeout? I suppose it probably
should.
On 26 July 2017 at 10:58, Tom Bentley <t.j.bent...@gmail.com> wrote:
> Tha
Thanks Paolo,
* in the "Public Interfaces" section you wrote
> alterTopics(Set) but then a collection is used (instead of a
> set) in the Proposed Changes section. I'm ok with collection.
>
Agree it should be Collection.
> * in the summary of the alterTopics method you say "The request
gt; If we agree to do that, then we need to decide if it should be implemented
> client-side or by adding a protocol API. The former is simpler, but the
> latter would allow non Java clients to use it without duplicating the logic
> of assigning replicas to the new partitions. Wh
would give the impression that it has not).
On 21 July 2017 at 11:09, Tom Bentley <t.j.bent...@gmail.com> wrote:
> Aside: I've started this new DISCUSS thread for KIP-179 since the original
> one had the incorrect KIP number 178. The original thread can be found
> he
smael
>
> On Thu, Jul 20, 2017 at 4:12 AM, Tom Bentley <t.j.bent...@gmail.com>
> wrote:
>
> > Hi Ismael,
> >
> > I've been working on the progress reporting assuming that it would be
> > acceptable for the ReassignPartitionsCommand to poll the AdminClie
Aside: I've started this new DISCUSS thread for KIP-179 since the original
one had the incorrect KIP number 178. The original thread can be found
here:
http://mail-archives.apache.org/mod_mbox/kafka-dev/201707.mbox/%3cCAMd5YszudP+-8z5KTbFh6JscT2p4xFi1=vzwwx+5dccpxry...@mail.gmail.com%3e
I've just
,
Tom
On 19 July 2017 at 16:08, Tom Bentley <t.j.bent...@gmail.com> wrote:
> Ah, thank you! I took the number from the "Next KIP Number: 178" on the
> KIP index and didn't check the tables. So this is now KIP-179. The old link
> will point you to the right place.
>
ke there is already a KIP-178:
>
> KIP-178: Size-based log directory selection strategy
>
> Ismael
>
> On Wed, Jul 19, 2017 at 7:05 AM, Tom Bentley <t.j.bent...@gmail.com>
> wrote:
>
> > OK, I will work on adding support for this to the KIP, with the intention
&g
on can often be done in stages.
>
> Does that make sense?
>
> Ismael
>
> On Wed, Jul 19, 2017 at 6:23 AM, Tom Bentley <t.j.bent...@gmail.com>
> wrote:
>
> > Hi Ismael,
> >
> > Answers in-line:
> >
> > 1. Have you considered how progress would b
Hi Ismael,
Answers in-line:
1. Have you considered how progress would be reported? Partition
> reassignment can take a long time and it would be good to have a mechanism
> for progress reporting.
>
The ReassignPartitionsCommand doesn't currently have a mechanism to track
progress. All you can
Hi,
I've written KIP-178 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-178+-+Change+ReassignPartitionsCommand+to+use+AdminClient)
for changing the ReassignPartitionsCommand (a.k.a.
kafka-reassign-partitions.sh)to remove the dependency on ZooKeeper and
instead provide the same
Tom Bentley created KAFKA-5601:
--
Summary: Refactor ReassignPartitionsCommand to use AdminClient
Key: KAFKA-5601
URL: https://issues.apache.org/jira/browse/KAFKA-5601
Project: Kafka
Issue Type
The project recently switched from all JIRA events being sent to the dev
mailling list, to just issue creations. This seems like a good thing
because the dev mailling list was very noisy before, and if you want to see
all the JIRA comments etc you can subscribe to the JIRA list. If you don't
Hi Paolo,
I don't have an opinion about which you should use, but I certainly agree
that two option parsing dependencies appears to be 1 too many.
Is there a reason why you prefer joptsimple?
Cheers,
Tom
On 10 July 2017 at 08:38, Paolo Patierno wrote:
> Hi devs,
>
>
>
Tom Bentley created KAFKA-5554:
--
Summary: Hilight config settings for particular common use cases
Key: KAFKA-5554
URL: https://issues.apache.org/jira/browse/KAFKA-5554
Project: Kafka
Issue Type
I realise that 0.11.0.0 is imminent and so the committers are rightly going
to be rather focussed on that, but I opened some PRs nearly a week ago and
they don't seem to have been looked at.
Even a comment on the PR to the effect of "We'll look at this right after
0.11.0.0" would at least
Tom Bentley created KAFKA-5517:
--
Summary: Support linking to particular configuration parameters
Key: KAFKA-5517
URL: https://issues.apache.org/jira/browse/KAFKA-5517
Project: Kafka
Issue Type
Tom Bentley created KAFKA-5508:
--
Summary: Documentation for altering topics
Key: KAFKA-5508
URL: https://issues.apache.org/jira/browse/KAFKA-5508
Project: Kafka
Issue Type: Bug
Hi Damian, my username is tombentley
Thanks
Tom
On 23 June 2017 at 14:57, Damian Guy <damian@gmail.com> wrote:
> Evgeniy, you should now have access.
>
> Tom & Paolo what are your wiki usernames?
>
> Thanks,
> Damian
>
> On Fri, 23 Jun 2017 at 14:52
>
> My confluence username is evis.
>
> Best regards,
> Evgeny
>
> С уважением,
> Евгений Веретенников
>
> 2017-06-23 16:43 GMT+03:00 Tom Bentley <t.j.bent...@gmail.com>:
>
> > Hi Evgeniy,
> >
> > The wiki page you link to there is
Hi Evgeniy,
The wiki page you link to there is about contributing improvements for
kafka.apache.org. But since you say "mirroring doc in Confluence" I assume
you want to edit
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330,
if so you will need one of the committers to
Tom Bentley created KAFKA-5496:
--
Summary: Consistency in documentation
Key: KAFKA-5496
URL: https://issues.apache.org/jira/browse/KAFKA-5496
Project: Kafka
Issue Type: Improvement
Thanks!
On 21 Jun 2017 4:20 pm, "Damian Guy" <damian@gmail.com> wrote:
> Done - thanks
>
> On Wed, 21 Jun 2017 at 12:19 Tom Bentley <t.j.bent...@gmail.com> wrote:
>
> > Please can I also be added? My username is tombentley.
> >
> > Thanks
Please can I also be added? My username is tombentley.
Thanks
Tom
On 21 June 2017 at 12:03, Damian Guy wrote:
> Hi Andras,
>
> You should have access now.
>
> Thanks,
> Damian
>
> On Wed, 21 Jun 2017 at 10:45 Andras Beni wrote:
>
> > Hi All,
> >
Tom Bentley created KAFKA-5479:
--
Summary: Docs for authorization omit authorizer.class.name
Key: KAFKA-5479
URL: https://issues.apache.org/jira/browse/KAFKA-5479
Project: Kafka
Issue Type
According to the website [1] I need to ask to be able to assign JIRAs to
myself, but I'm still unable to do this. Could someone set this up for me
please?
Thanks,
Tom
[1]: https://kafka.apache.org/contributing
On 14 June 2017 at 13:43, Tom Bentley <t.j.bent...@gmail.com> wrote:
> Ple
> I just checked kafka-topics.sh and the only required argument there is
> --zookeeper option. Not sure if you were thinking of some other command.
>
> I meant that for kafka-topics.sh --create requires --topic (amongst
others), but --list does not.
(For example, using these methods we can say
are close to 20 commands I have made changes to. I will try to push
> something out by end of day today. So I think I will cover 1) from your
> list below and not 2) and 3).
>
> Thanks Tom.
>
> Regards,
> Mariam.
>
>
>
> [image: Inactive hide details for Tom Bentley -
Hi,
I noticed that the command line tools could use a little love. For
instance, I was surprised that most of them don't support `--help`, and
generally there are a few inconsistencies.
KIP-14 is dormant and AFAICS no one is working on
https://issues.apache.org/jira/browse/KAFKA-2111 either. So
Tom Bentley created KAFKA-5459:
--
Summary: Support kafka-console-producer.sh messages as whole file
Key: KAFKA-5459
URL: https://issues.apache.org/jira/browse/KAFKA-5459
Project: Kafka
Issue
It seems to me that in the most common case the partition isn't going to be
magically created and in that case it's more helpful to quit with an error
than to sit there waiting for the partition to be created.
[
https://issues.apache.org/jira/browse/KAFKA-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049265#comment-16049265
]
Tom Bentley commented on KAFKA-2111:
I'm happy work work on this if [~johnma] is no longer looking
Please could I be added to the JIRA contributor list so that I can assign
issues to myself?
Thanks,
Tom
[
https://issues.apache.org/jira/browse/KAFKA-5421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049037#comment-16049037
]
Tom Bentley commented on KAFKA-5421:
The root cause is probably bitrot on the disk storing
Hi Paolo,
Usually you can just follow the links added by asfbot on the PR, but these
are currently giving 404 (and not just for your failures, so maybe an ASF
infrastructure problem?)
Cheers,
Tom
On 12 June 2017 at 09:20, Paolo Patierno wrote:
> Hi all,
>
>
> I opened
501 - 578 of 578 matches
Mail list logo