, June 11, 2015 1:28 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
Discussion aside, was there any significant material change besides
the additions below? If so, then we can avoid the overhead of another
vote unless
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
Discussion aside, was there any significant material change besides
the additions below? If so, then we can avoid the overhead of another
vote unless someone wants to down-vote these changes.
Joel
From: Ashish Singh [asi...@cloudera.com]
Sent: Friday, May 29, 2015 8:36 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
+1 on discussing this on next KIP hangout. I will update KIP-24
: Friday, May 29, 2015 8:36 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
+1 on discussing this on next KIP hangout. I will update KIP-24 before that.
On Fri, May 29, 2015 at 3:40 AM, Andrii Biletskyi
andrii.bilets
: RE: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
I've made two changes to the document:
- Removed the TMR evolution piece since we agreed to retain this.
- Added two new API's to the admin client spec. (Alter and Describe config).
Please review.
Aditya
[gharatmayures...@gmail.com]
Sent: Thursday, May 21, 2015 8:29 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized
administrative
operations (Thread 2)
Hi Jun,
Thanks a lot. I get it now.
Point 4) will actually enable clients to who don't want to create
From: Jun Rao [j...@confluent.io]
Sent: Thursday, May 28, 2015 5:32 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized
administrative
operations (Thread 2)
There is a reasonable use case of ISR in KAFKA-2225
: Thursday, May 21, 2015 8:29 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
Hi Jun,
Thanks a lot. I get it now.
Point 4) will actually enable clients to who don't want to create a topic
with default partitions
:29 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
Hi Jun,
Thanks a lot. I get it now.
Point 4) will actually enable clients to who don't want to create a topic
with default partitions, if it does not exist and then can
Hi Jun,
Thanks a lot. I get it now.
Point 4) will actually enable clients to who don't want to create a topic
with default partitions, if it does not exist and then can manually create
the topic with their own configs(#partitions).
Thanks,
Mayuresh
On Thu, May 21, 2015 at 6:16 PM, Jun Rao
For ListTopics, we decided not to add a ListTopics request for now and just
rely on passing in an empty list to TMR. We can revisit this in the future
if it becomes an issue.
Thanks,
Jun
On Wed, May 13, 2015 at 3:31 PM, Joel Koshy jjkosh...@gmail.com wrote:
Just had a few minor questions
Joel,
- DecreasePartitionNotAllowed. Yeah, that's kind of subcase of
InvalidPartitions...
But since client can always request topic metadata and check what exactly is
was wrong with Partitions argument, I think, yes, we can remove
DecreasePartitionNotAllowed
and use InvalidPartitions instead.
Just had a few minor questions before I join the vote thread.
Apologies if these have been discussed:
- Do we need DecreasePartitionsNotAllowed? i.e., can we just return
InvalidPartitions instead?
- AdminClient.listTopics: should we allow listing all partitions? Or
do you intend for the
Guys,
I've updated the wiki to reflect all previously discussed items
(regarding the schema only - this is included to phase 1).
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
I think we can have the final discussion today (for
The following is a description of some of my concerns on allowing the same
topic multiple times in AlterTopicRequest.
ATP has an array of entries, each corresponding to a topic. We allow
multiple changes to a topic in a single entry. Those changes may fail to
apply independently (e.g., the config
Guys,
A quick summary of our today's meeting.
There were no additional issues/questions. The only item about which
we are not 100% sure is multiple instructions for one topic in one
request case.
It was proposed by Jun to explain reasons behind not allowing users doing
that again
here in mailing
Guys,
It seems that there are no open questions left so prior to our weekly call
let me summarize what I'm going to implement as part of phase one for KIP-4.
1. Add 3 new Wire Protocol requests - Create-, Alter- and DeleteTopicRequest
2. Topic requests are batch requests, errors are returned
Okay, I had some doubts in terms of reassign-partitions case. I was
not sure whether we need ISR to check post condition of partition
reassignment. But I think we can rely on assigned replicas - the workflow
in reassignPartitions is the following:
1. Update AR in ZK with OAR + RAR.
...
10. Update
Yes, to verify if a partition reassignment completes or not, we just need
to make sure AR == RAR. So, we don't need ISR for this. It's probably still
useful to know ISR for monitoring in general though.
Thanks,
Jun
On Mon, Apr 27, 2015 at 4:15 AM, Andrii Biletskyi
andrii.bilets...@stealth.ly
Andrii,
Another thing. We decided not to add the lag info in TMR. To be consistent,
we probably also want to remove ISR from TMR since only the leader knows
it. We can punt on adding any new request from getting ISR. ISR is mostly
useful for monitoring. Currently, one can determine if a replica
Andrii,
Thanks for the update.
For your second point, I agree that if a single AlterTopicRequest can make
multiple changes, there is no need to support the same topic included more
than once in the request.
Now about the semantics in your first question. I was thinking that we can
do the
Jun,
I like your approach to AlterTopicReques semantics! Sounds like
we linearize all request fields to ReplicaAssignment - I will definitely
try this out to ensure there are no other pitfalls.
With regards to multiple instructions in one batch per topic. For me
this sounds reasonable too. We
Guys,
Can we come to some agreement in terms of the second item from
the email above? This blocks me from updating and uploading the
patch. Also the new schedule for the weekly calls doesn't work very
well for me - it's 1 am in my timezone :) - so I'd rather we confirm
everything that is possible
As said above, I spent some time thinking about AlterTopicRequest
semantics and batching.
Firstly, about AlterTopicRequest. Our goal here is to see whether we
can suggest some simple semantics and at the same time let users
change different things in one instruction (hereinafter instruction - is
1. Yes, this will be much easier. Okay, let's add it.
2, Okay. This will differ a little bit from the way currently
kafka-topics.sh handles alter-topic command, but I think it's
a reasonable restriction.
I'll update KIP acordingly to our weekly call.
Thanks,
Andrii Biletskyi
On Mon, Apr 20,
Hi all,
I've updated KIP-4 page to include all previously discussed items such as:
new error codes, merged alter-topic and reassign-partitions requests, added
TMR_V1.
It'd be great if we concentrate on the Errors+Wire Protocol schema and
discuss
any remaining issues today, since first patch will
Hey Andrii, thanks for all the hard work on this, it has come a long way.
A couple questions and comments on this.
For the errors, can we do the following:
1. Remove IllegalArgument from the name, we haven't used that convention
for other errors.
2. Normalize this list with the existing errors.
Guys,
Thank you for your time. A short summary of our discussion.
Answering previous items:
1. 2. I will double check existing error codes to align the list of
errors that needs to be added.
3. We agreed to think again about the batch requests semantics.
The main concern is that users would
1. Yes, lag is probably only going to be useful for the admin client.
However, so is isr. It seems to me that we should get lag and isr from the
same request. I was thinking that we can just extend TMR by changing
replicas from an array of int to an array of (int, lag) pairs. Is that too
1. agreed
2. agree new error
3. having discrete operations for tasks makes sense, combining them is
confusing for users I think. + 1 for let user change only one thing at a
time
4. lets be consistent both to the new code and existing code. lets not
confuse the user but give them the right error
1. For the lags, we can add a new field lags per partition. It will
return for each replica that's not in isr, the replica id and the lag in
messages. Also, if TMR is sent to a non-leader, the response can just
include an empty array for isr and lags.
2. So, we will just return a topic level
Guys,
Thanks for the discussion!
Summary:
1. Q: How KAFKA-1367 (isr is inconsistent in brokers' metadata cache) can
affect implementation?
A: We can fix this issue for the leading broker - ReplicaManager (or
Partition)
component should have accurate isr list, then with
Jun,
404. Great, thanks!
500. If I understand correctly KAFKA-1367 says ISR part of TMR may
be inconsistent. If so, then I believe all admin commands but describeTopic
are not affected. Let me emphasize that it's about AdminClient operations,
not about Wire Protocol requests. What I mean:
To
Andrii,
404. Jay and I chatted a bit. We agreed to leave createTopicRequest as
async for now.
There is another thing.
500. Currently, we have this issue (KAFKA-1367) that the ISR in the
metadata cache can be out of sync. The reason is that ISR is really
maintained at the leader. We can
Hi all,
I wasn't able to send email to our thread (it says we exceeded message size
limit :)).
So I'm starting the new one.
Jun,
Thanks again for the review. Answering your comments:
201. I'm not sure I understand how can we evolve Cluster in backward
compatible way. In my understanding topic
Hi all,
A summary of our discussion:
201. Q: Cluster updates in backward compatible way.
A: Add topicConfigs map property and change constructor, this
shouldn't break Consumer/Producer since TMR is used in NetworkClient,
not directly by Consumer/Producer.
300. Q: Can we merge AlterTopic
36 matches
Mail list logo
