I may be missing some context but hopefully this will also be covered today: I thought the earlier proposal where there was an explicit ClusterMetadata request was clearer and explicit. During the course of this thread I think the conclusion was that the main need was for controller information and that can be rolled into the topic metadata response but that seems a bit irrelevant to topic metadata. FWIW I think the full broker-list is also irrelevant to topic metadata, but it is already there and in use. I think there is still room for an explicit ClusterMetadata request since there may be other cluster-level information that we may want to add over time (and that have nothing to do with topic metadata).
On Tue, Mar 17, 2015 at 02:45:30PM +0200, Andrii Biletskyi wrote: > Jun, > > 101. Okay, if you say that such use case is important. I also think > using clientId for these purposes is fine - if we already have this field > as part of all Wire protocol messages, why not use that. > I will update KIP-4 page if nobody has other ideas (which may come up > during the call today). > > 102.1 Agree, I'll update the KIP accordingly. I think we can add new, > fine-grained error codes if some error code received in specific case > won't give enough context to return a descriptive error message for user. > > Look forward to discussing all outstanding issues in detail today during > the call. > > Thanks, > Andrii Biletskyi > > > > On Mon, Mar 16, 2015 at 10:59 PM, Jun Rao <j...@confluent.io> wrote: > > > 101. There may be a use case where you only want the topics to be created > > manually by admins. Currently, you can do that by disabling auto topic > > creation and issue topic creation from the TopicCommand. If we disable auto > > topic creation completely on the broker and don't have a way to distinguish > > between topic creation requests from the regular clients and the admin, we > > can't support manual topic creation any more. I was thinking that another > > way of distinguishing the clients making the topic creation requests is > > using clientId. For example, the admin tool can set it to something like > > admin and the broker can treat that clientId specially. > > > > Also, there is a related discussion in KAFKA-2020. Currently, we do the > > following in TopicMetadataResponse: > > > > 1. If leader is not available, we set the partition level error code to > > LeaderNotAvailable. > > 2. If a non-leader replica is not available, we take that replica out of > > the assigned replica list and isr in the response. As an indication for > > doing that, we set the partition level error code to ReplicaNotAvailable. > > > > This has a few problems. First, ReplicaNotAvailable probably shouldn't be > > an error, at least for the normal producer/consumer clients that just want > > to find out the leader. Second, it can happen that both the leader and > > another replica are not available at the same time. There is no error code > > to indicate both. Third, even if a replica is not available, it's still > > useful to return its replica id since some clients (e.g. admin tool) may > > still make use of it. > > > > One way to address this issue is to always return the replica id for > > leader, assigned replicas, and isr regardless of whether the corresponding > > broker is live or not. Since we also return the list of live brokers, the > > client can figure out whether a leader or a replica is live or not and act > > accordingly. This way, we don't need to set the partition level error code > > when the leader or a replica is not available. This doesn't change the wire > > protocol, but does change the semantics. Since we are evolving the protocol > > of TopicMetadataRequest here, we can potentially piggyback the change. > > > > 102.1 For those types of errors due to invalid input, shouldn't we just > > guard it at parameter validation time and throw InvalidArgumentException > > without even sending the request to the broker? > > > > Thanks, > > > > Jun > > > > > > On Mon, Mar 16, 2015 at 10:37 AM, Andrii Biletskyi < > > andrii.bilets...@stealth.ly> wrote: > > > > > Jun, > > > > > > Answering your questions: > > > > > > 101. If I understand you correctly, you are saying future producer > > versions > > > (which > > > will be ported to TMR_V1) won't be able to automatically create topic (if > > > we > > > unconditionally remove topic creation from there). But we need to this > > > preserve logic. > > > Ok, about your proposal: I'm not a big fan too, when it comes to > > > differentiating > > > clients directly in protocol schema. And also I'm not sure I understand > > at > > > all why > > > auto.create.topics.enable is a server side configuration. Can we > > deprecate > > > this setting > > > in future versions, add this setting to producer and based on that upon > > > receiving > > > UnknownTopic create topic explicitly by a separate producer call via > > > adminClient? > > > > > > 102.1. Hm, yes. It's because we want to support batching and at the same > > > time we > > > want to give descriptive error messages for clients. Since AdminClient > > > holds the context > > > to construct such messages (e.g. AdminClient layer can know that > > > InvalidArgumentsCode > > > means two cases: either invalid number - e.g. -1; or replication-factor > > was > > > provided while > > > partitions argument wasn't) - I wrapped responses in Exceptions. But I'm > > > open to any > > > other ideas, this was just initial version. > > > 102.2. Yes, I agree. I'll change that to probably some other dto. > > > > > > Thanks, > > > Andrii Biletskyi > > > > > > On Fri, Mar 13, 2015 at 7:16 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > Andrii, > > > > > > > > 101. That's what I was thinking too, but it may not be that simple. In > > > > TopicMetadataRequest_V1, > > > > we can let it not trigger auto topic creation. Then, in the producer > > > side, > > > > if it gets an UnknownTopicException, it can explicitly issue a > > > > createTopicRequest for auto topic creation. On the consumer side, it > > will > > > > never issue createTopicRequest. This works when auto topic creation is > > > > enabled on the broker side. However, I am not sure how things will work > > > > when auto topic creation is disabled on the broker side. In this case, > > we > > > > want to have a way to manually create a topic, potentially through > > admin > > > > commands. However, then we need a way to distinguish createTopicRequest > > > > issued from the producer clients and the admin tools. May be we can > > add a > > > > new field in createTopicRequest and set it differently in the producer > > > > client and the admin client. However, I am not sure if that's the best > > > > approach. > > > > > > > > 2. Yes, refactoring existing requests is a non-trivial amount of work. > > I > > > > posted some comments in KAFKA-1927. We will probably have to fix > > > KAFKA-1927 > > > > first, before adding the new logic in KAFKA-1694. Otherwise, the > > changes > > > > will be too big. > > > > > > > > 102. About the AdminClient: > > > > 102.1. It's a bit weird that we return exception in the api. It seems > > > that > > > > we should either return error code or throw an exception when getting > > the > > > > response state. > > > > 102.2. We probably shouldn't explicitly use the request object in the > > > api. > > > > Not every request evolution requires an api change. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > On Fri, Mar 13, 2015 at 4:08 AM, Andrii Biletskyi < > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > Jun, > > > > > > > > > > Thanks for you comments. Answers inline: > > > > > > > > > > 100. There are a few fields such as ReplicaAssignment, > > > > > > ReassignPartitionRequest, > > > > > > and PartitionsSerialized that are represented as a string, but > > > contain > > > > > > composite structures in json. Could we flatten them out directly in > > > the > > > > > > protocol definition as arrays/records? > > > > > > > > > > > > > > > Yes, now with Admin Client this looks a bit weird. My initial > > > motivation > > > > > was: > > > > > ReassignPartitionCommand accepts input in json, we want to remain > > > tools' > > > > > interfaces unchanged, where possible. > > > > > If we port it to deserialized format, in CLI (/tools project) we will > > > > have > > > > > to add some > > > > > json library since /tools is written in java and we'll need to > > > > deserialize > > > > > json file > > > > > provided by a user. Can we quickly agree on what this library should > > be > > > > > (Jackson, GSON, whatever)? > > > > > > > > > > 101. Does TopicMetadataRequest v1 still trigger auto topic creation? > > > This > > > > > > will be a bit weird now that we have a separate topic creation api. > > > > Have > > > > > > you thought about how the new createTopicRequest and > > > > TopicMetadataRequest > > > > > > v1 will be used in the producer/consumer client, in addition to > > admin > > > > > > tools? For example, ideally, we don't want TopicMetadataRequest > > from > > > > the > > > > > > consumer to trigger auto topic creation. > > > > > > > > > > > > > > > I agree, this strange logic should be fixed. I'm not confident in > > this > > > > > Kafka part so > > > > > correct me if I'm wrong, but it doesn't look like a hard thing to > > do, I > > > > > think we can > > > > > leverage AdminClient for that in Producer and unconditionally remove > > > > topic > > > > > creation from the TopicMetadataRequest_V1. > > > > > > > > > > 2. I think Jay meant getting rid of scala classes > > > > > > like HeartbeatRequestAndHeader and HeartbeatResponseAndHeader. We > > did > > > > > that > > > > > > as a stop-gap thing when adding the new requests for the consumers. > > > > > > However, the long term plan is to get rid of all those and just > > reuse > > > > the > > > > > > java request/response in the client. Since this KIP proposes to > > add a > > > > > > significant number of new requests, perhaps we should bite the > > bullet > > > > to > > > > > > clean up the existing scala requests first before adding new ones? > > > > > > > > > > > > > > > > Yes, looks like I misunderstood the point of ...RequestAndHeader. > > > Okay, I > > > > > will > > > > > rework that. The only thing is that I don't see any example how it > > was > > > > done > > > > > for at > > > > > least one existing protocol message. Thus, as I understand, I have to > > > > think > > > > > how we > > > > > are going to do it. > > > > > Re porting all existing RQ/RP in this patch. Sounds reasonable, but > > if > > > > it's > > > > > an *obligatory* > > > > > requirement to have Admin KIP done, I'm afraid this can be a serious > > > > > blocker for us. > > > > > There are 13 protocol messages and all that would require not only > > unit > > > > > tests but quite > > > > > intensive manual testing, no? I'm afraid I'm not the right guy to > > cover > > > > > pretty much all > > > > > Kafka core internals :). Let me know your thoughts on this item. Btw > > > > there > > > > > is a ticket to > > > > > follow-up this issue ( > > https://issues.apache.org/jira/browse/KAFKA-2006 > > > ). > > > > > > > > > > Thanks, > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > On Fri, Mar 13, 2015 at 6:40 AM, Jun Rao <j...@confluent.io> wrote: > > > > > > > > > > > Andrii, > > > > > > > > > > > > > > > > > > A few more comments. > > > > > > > > > > > > 100. There are a few fields such as ReplicaAssignment, > > > > > > ReassignPartitionRequest, > > > > > > and PartitionsSerialized that are represented as a string, but > > > contain > > > > > > composite structures in json. Could we flatten them out directly in > > > the > > > > > > protocol definition as arrays/records? > > > > > > > > > > > > 101. Does TopicMetadataRequest v1 still trigger auto topic > > creation? > > > > This > > > > > > will be a bit weird now that we have a separate topic creation api. > > > > Have > > > > > > you thought about how the new createTopicRequest and > > > > TopicMetadataRequest > > > > > > v1 will be used in the producer/consumer client, in addition to > > admin > > > > > > tools? For example, ideally, we don't want TopicMetadataRequest > > from > > > > the > > > > > > consumer to trigger auto topic creation. > > > > > > > > > > > > 2. I think Jay meant getting rid of scala classes > > > > > > like HeartbeatRequestAndHeader and HeartbeatResponseAndHeader. We > > did > > > > > that > > > > > > as a stop-gap thing when adding the new requests for the consumers. > > > > > > However, the long term plan is to get rid of all those and just > > reuse > > > > the > > > > > > java request/response in the client. Since this KIP proposes to > > add a > > > > > > significant number of new requests, perhaps we should bite the > > bullet > > > > to > > > > > > clean up the existing scala requests first before adding new ones? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 12, 2015 at 3:37 PM, Andrii Biletskyi < > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > As said above - I list again all comments from this thread so we > > > > > > > can see what's left and finalize all pending issues. > > > > > > > > > > > > > > Comments from Jay: > > > > > > > 1. This is much needed functionality, but there are a lot of the > > so > > > > > let's > > > > > > > really think these protocols through. We really want to end up > > > with a > > > > > set > > > > > > > of well thought-out, orthoganol apis. For this reason I think it > > is > > > > > > really > > > > > > > important to think through the end state even if that includes > > APIs > > > > we > > > > > > > won't implement in the first phase. > > > > > > > > > > > > > > A: Definitely behind this. Would appreciate if there are concrete > > > > > > comments > > > > > > > how this can be improved. > > > > > > > > > > > > > > 2. Let's please please please wait until we have switched the > > > server > > > > > over > > > > > > > to the new java protocol definitions. If we add upteen more ad > > hoc > > > > > scala > > > > > > > objects that is just generating more work for the conversion we > > > know > > > > we > > > > > > > have to do. > > > > > > > > > > > > > > A: Fixed in the latest patch - removed scala protocol classes. > > > > > > > > > > > > > > 3. This proposal introduces a new type of optional parameter. > > This > > > is > > > > > > > inconsistent with everything else in the protocol where we use -1 > > > or > > > > > some > > > > > > > other marker value. You could argue either way but let's stick > > with > > > > > that > > > > > > > for consistency. For clients that implemented the protocol in a > > > > better > > > > > > way > > > > > > > than our scala code these basic primitives are hard to change. > > > > > > > > > > > > > > A: Fixed in the latest patch - removed MaybeOf type and changed > > > > > protocol > > > > > > > accordingly. > > > > > > > > > > > > > > 4. ClusterMetadata: This seems to duplicate TopicMetadataRequest > > > > which > > > > > > has > > > > > > > brokers, topics, and partitions. I think we should rename that > > > > request > > > > > > > ClusterMetadataRequest (or just MetadataRequest) and include the > > id > > > > of > > > > > > the > > > > > > > controller. Or are there other things we could add here? > > > > > > > > > > > > > > A: I agree. Updated the KIP. Let's extends TopicMetadata to > > > version 2 > > > > > and > > > > > > > include controller. > > > > > > > > > > > > > > 5. We have a tendency to try to make a lot of requests that can > > > only > > > > go > > > > > > to > > > > > > > particular nodes. This adds a lot of burden for client > > > > implementations > > > > > > (it > > > > > > > sounds easy but each discovery can fail in many parts so it ends > > up > > > > > > being a > > > > > > > full state machine to do right). I think we should consider > > making > > > > > admin > > > > > > > commands and ideally as many of the other apis as possible > > > available > > > > on > > > > > > all > > > > > > > brokers and just redirect to the controller on the broker side. > > > > Perhaps > > > > > > > there would be a general way to encapsulate this re-routing > > > behavior. > > > > > > > > > > > > > > A: It's a very interesting idea, but seems there are some > > concerns > > > > > about > > > > > > > this > > > > > > > feature (like performance considerations, how this will > > complicate > > > > > server > > > > > > > etc). > > > > > > > I believe this shouldn't be a blocker. If this feature is > > > implemented > > > > > at > > > > > > > some > > > > > > > point it won't affect Admin changes - at least no changes to > > public > > > > API > > > > > > > will be required. > > > > > > > > > > > > > > 6. We should probably normalize the key value pairs used for > > > configs > > > > > > rather > > > > > > > than embedding a new formatting. So two strings rather than one > > > with > > > > an > > > > > > > internal equals sign. > > > > > > > > > > > > > > A: Fixed in the latest patch - normalized configs and changed > > > > protocol > > > > > > > accordingly. > > > > > > > > > > > > > > 7. Is the postcondition of these APIs that the command has begun > > or > > > > > that > > > > > > > the command has been completed? It is a lot more usable if the > > > > command > > > > > > has > > > > > > > been completed so you know that if you create a topic and then > > > > publish > > > > > to > > > > > > > it you won't get an exception about there being no such topic. > > > > > > > > > > > > > > A: For long running requests (like reassign partitions) - the > > post > > > > > > > condition is > > > > > > > command has begun - so we don't block the client. In case of your > > > > > > example - > > > > > > > topic commands, this will be refactored and topic commands will > > be > > > > > > executed > > > > > > > immediately, since the Controller will serve Admin requests > > > > > > > (follow-up ticket KAFKA-1777). > > > > > > > > > > > > > > 8. Describe topic and list topics duplicate a lot of stuff in the > > > > > > metadata > > > > > > > request. Is there a reason to give back topics marked for > > > deletion? I > > > > > > feel > > > > > > > like if we just make the post-condition of the delete command be > > > that > > > > > the > > > > > > > topic is deleted that will get rid of the need for this right? > > And > > > it > > > > > > will > > > > > > > be much more intuitive. > > > > > > > > > > > > > > A: Fixed in the latest patch - removed topics marked for deletion > > > in > > > > > > > ListTopicsRequest. > > > > > > > > > > > > > > 9. Should we consider batching these requests? We have generally > > > > tried > > > > > to > > > > > > > allow multiple operations to be batched. My suspicion is that > > > without > > > > > > this > > > > > > > we will get a lot of code that does something like > > > > > > > for(topic: adminClient.listTopics()) > > > > > > > adminClient.describeTopic(topic) > > > > > > > this code will work great when you test on 5 topics but not do as > > > > well > > > > > if > > > > > > > you have 50k. > > > > > > > > > > > > > > A: Updated the KIP - please check "Topic Admin Schema" section. > > > > > > > > > > > > > > 10. I think we should also discuss how we want to expose a > > > > programmatic > > > > > > JVM > > > > > > > client api for these operations. Currently people rely on > > > AdminUtils > > > > > > which > > > > > > > is totally sketchy. I think we probably need another client under > > > > > > clients/ > > > > > > > that exposes administrative functionality. We will need this just > > > to > > > > > > > properly test the new apis, I suspect. We should figure out that > > > API. > > > > > > > > > > > > > > A: Updated the KIP - please check "Admin Client" section with an > > > > > initial > > > > > > > API proposal. > > > > > > > > > > > > > > 11. The other information that would be really useful to get > > would > > > be > > > > > > > information about partitions--how much data is in the partition, > > > what > > > > > are > > > > > > > the segment offsets, what is the log-end offset (i.e. last > > offset), > > > > > what > > > > > > is > > > > > > > the compaction point, etc. I think that done right this would be > > > the > > > > > > > successor to the very awkward OffsetRequest we have today. > > > > > > > > > > > > > > A: I removed ConsumerGroupOffsetsRequest in the latest patch. I > > > > believe > > > > > > > this should > > > > > > > be resolved in a separate KIP / jira ticket. > > > > > > > > > > > > > > 12. Generally we can do good error handling without needing > > custom > > > > > > > server-side > > > > > > > messages. I.e. generally the client has the context to know that > > if > > > > it > > > > > > got > > > > > > > an error that the topic doesn't exist to say "Topic X doesn't > > > exist" > > > > > > rather > > > > > > > than "error code 14" (or whatever). Maybe there are specific > > cases > > > > > where > > > > > > > this is hard? If we want to add server-side error messages we > > > really > > > > do > > > > > > > need to do this in a consistent way across the protocol. > > > > > > > > > > > > > > A: Updated the KIP - please check "Protocol Errors" section. I > > > added > > > > > the > > > > > > > comprehensive, fine-grained list of error codes. > > > > > > > > > > > > > > Comments from Guozhang: > > > > > > > 13. Describe topic request: it would be great to go beyond just > > > > > batching > > > > > > on > > > > > > > topic name regex for this request. For example, a very common use > > > > case > > > > > of > > > > > > > the topic command is to list all topics whose config A's value is > > > B. > > > > > With > > > > > > > topic name regex then we have to first retrieve __all__ topics's > > > > > > > description info and then filter at the client end, which will > > be a > > > > > huge > > > > > > > burden on ZK. > > > > > > > AND > > > > > > > 14. Config K-Vs in create topic: this is related to the previous > > > > point; > > > > > > > maybe we can add another metadata K-V or just a metadata string > > > along > > > > > > side > > > > > > > with config K-V in create topic like we did for offset commit > > > > request. > > > > > > This > > > > > > > field can be quite useful in storing information like "owner" of > > > the > > > > > > topic > > > > > > > who issue the create command, etc, which is quite important for a > > > > > > > multi-tenant setting. Then in the describe topic request we can > > > also > > > > > > batch > > > > > > > on regex of the metadata field. > > > > > > > > > > > > > > A: As discussed it is very interesting but can be implemented > > later > > > > > after > > > > > > > we have some basic functionality there. > > > > > > > > > > > > > > 15. Today all the admin operations are async in the sense that > > > > command > > > > > > will > > > > > > > return once it is written in ZK, and that is why we need extra > > > > > > verification > > > > > > > like testUtil.waitForTopicCreated() / verify partition > > reassignment > > > > > > > request, etc. With admin requests we could add a flag to enable / > > > > > disable > > > > > > > synchronous requests; when it is turned on, the response will not > > > > > return > > > > > > > until the request has been completed. And for async requests we > > can > > > > > add a > > > > > > > "token" field in the response, and then only need a general > > "admin > > > > > > > verification request" with the given token to check if the async > > > > > request > > > > > > > has been completed. > > > > > > > > > > > > > > A: I see your point. My idea was to provide specific > > > Verify...Request > > > > > per > > > > > > > each > > > > > > > long running request, where needed. We can do it the way you > > > suggest. > > > > > The > > > > > > > only > > > > > > > concern is that introducing a token we again will make schema > > > > > "dynamic". > > > > > > We > > > > > > > wanted > > > > > > > to do similar thing introducing single AdminRequest for all topic > > > > > > commands > > > > > > > but rejected > > > > > > > this idea because we wanted to have schema defined. So this is > > > more a > > > > > > > choice between: > > > > > > > a) have fixed schema but introduce each time new Verify...Request > > > for > > > > > > > long-running requests > > > > > > > b) use one request for verification but generalize it with token > > > > > > > I'm fine with whatever decision community come to. Just let me > > know > > > > > your > > > > > > > thoughts. > > > > > > > > > > > > > > Comment from Gwen: > > > > > > > 16. Specifically for ownership, I think the plan is to add ACL > > (it > > > > > sounds > > > > > > > like you are describing ACL) via an external system (Argus, > > > Sentry). > > > > > > > I remember KIP-11 described this, but I can't find the KIP any > > > > longer. > > > > > > > > > > > > > > A: Okay, no problem. Not sure though how we are going to handle > > it. > > > > > Wait > > > > > > > which KIP > > > > > > > will be committed first and include changes to TopicMetadata from > > > the > > > > > > later > > > > > > > one? > > > > > > > Anyway, I added this note to "Open Questions" section so we don't > > > > miss > > > > > > this > > > > > > > piece. > > > > > > > > > > > > > > Thanks, > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > On Fri, Mar 13, 2015 at 12:34 AM, Andrii Biletskyi < > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > Today I uploaded the patch that covers some of the discussed > > and > > > > > agreed > > > > > > > > items: > > > > > > > > - removed MaybeOf optional type > > > > > > > > - switched to java protocol definitions > > > > > > > > - simplified messages (normalized configs, removed topic marked > > > for > > > > > > > > deletion) > > > > > > > > > > > > > > > > I also updated the KIP-4 with respective changes and wrote down > > > my > > > > > > > > proposal for > > > > > > > > pending items: > > > > > > > > - Batch Admin Operations -> updated Wire Protocol schema > > proposal > > > > > > > > - Remove ClusterMetadata -> changed to extend > > > TopicMetadataRequest > > > > > > > > - Admin Client -> updated my initial proposal to reflect > > batching > > > > > > > > - Error codes -> proposed fine-grained error code instead of > > > > > > > > AdminRequestFailed > > > > > > > > > > > > > > > > I will also send a separate email to cover all comments from > > this > > > > > > thread. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 12, 2015 at 9:26 PM, Gwen Shapira < > > > > gshap...@cloudera.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >> Found KIP-11 ( > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface > > > > > > > >> ) > > > > > > > >> It actually specifies changes to the Metadata protocol, so > > > making > > > > > sure > > > > > > > >> both KIPs are consistent in this regard will be good. > > > > > > > >> > > > > > > > >> On Thu, Mar 12, 2015 at 12:21 PM, Gwen Shapira < > > > > > gshap...@cloudera.com > > > > > > > > > > > > > > >> wrote: > > > > > > > >> > Specifically for ownership, I think the plan is to add ACL > > (it > > > > > > sounds > > > > > > > >> > like you are describing ACL) via an external system (Argus, > > > > > Sentry). > > > > > > > >> > I remember KIP-11 described this, but I can't find the KIP > > any > > > > > > longer. > > > > > > > >> > > > > > > > > >> > Regardless, I think KIP-4 focuses on getting information > > that > > > > > > already > > > > > > > >> > exists from Kafka brokers, not on adding information that > > > > perhaps > > > > > > > >> > should exist but doesn't yet? > > > > > > > >> > > > > > > > > >> > Gwen > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > On Thu, Mar 12, 2015 at 6:37 AM, Guozhang Wang < > > > > > wangg...@gmail.com> > > > > > > > >> wrote: > > > > > > > >> >> Folks, > > > > > > > >> >> > > > > > > > >> >> Just want to elaborate a bit more on the create-topic > > > metadata > > > > > and > > > > > > > >> batching > > > > > > > >> >> describe-topic based on config / metadata in my previous > > > email > > > > as > > > > > > we > > > > > > > >> work > > > > > > > >> >> on KAFKA-1694. The main motivation is to have some sort of > > > > topic > > > > > > > >> management > > > > > > > >> >> mechanisms, which I think is quite important in a > > > multi-tenant > > > > / > > > > > > > cloud > > > > > > > >> >> architecture: today anyone can create topics in a shared > > > Kafka > > > > > > > >> cluster, but > > > > > > > >> >> there is no concept or "ownership" of topics that are > > created > > > > by > > > > > > > >> different > > > > > > > >> >> users. For example, at LinkedIn we basically distinguish > > > topic > > > > > > owners > > > > > > > >> via > > > > > > > >> >> some casual topic name prefix, which is a bit awkward and > > > does > > > > > not > > > > > > > fly > > > > > > > >> as > > > > > > > >> >> we scale our customers. It would be great to use > > > > describe-topics > > > > > > such > > > > > > > >> as: > > > > > > > >> >> > > > > > > > >> >> Describe all topics that is created by me. > > > > > > > >> >> > > > > > > > >> >> Describe all topics whose retention time is overriden to X. > > > > > > > >> >> > > > > > > > >> >> Describe all topics whose writable group include user Y > > (this > > > > is > > > > > > > >> related to > > > > > > > >> >> authorization), etc.. > > > > > > > >> >> > > > > > > > >> >> One possible way to achieve this is to add a metadata file > > in > > > > the > > > > > > > >> >> create-topic request, whose value will also be written ZK > > as > > > we > > > > > > > create > > > > > > > >> the > > > > > > > >> >> topic; then describe-topics can choose to batch topics > > based > > > on > > > > > 1) > > > > > > > name > > > > > > > >> >> regex, 2) config K-V matching, 3) metadata regex, etc. > > > > > > > >> >> > > > > > > > >> >> Thoughts? > > > > > > > >> >> > > > > > > > >> >> Guozhang > > > > > > > >> >> > > > > > > > >> >> On Thu, Mar 5, 2015 at 4:37 PM, Guozhang Wang < > > > > > wangg...@gmail.com> > > > > > > > >> wrote: > > > > > > > >> >> > > > > > > > >> >>> Thanks for the updated wiki. A few comments below: > > > > > > > >> >>> > > > > > > > >> >>> 1. Error description in response: I think if some > > errorCode > > > > > could > > > > > > > >> indicate > > > > > > > >> >>> several different error cases then we should really change > > > it > > > > to > > > > > > > >> multiple > > > > > > > >> >>> codes. In general the errorCode itself would be precise > > and > > > > > > > >> sufficient for > > > > > > > >> >>> describing the server side errors. > > > > > > > >> >>> > > > > > > > >> >>> 2. Describe topic request: it would be great to go beyond > > > just > > > > > > > >> batching on > > > > > > > >> >>> topic name regex for this request. For example, a very > > > common > > > > > use > > > > > > > >> case of > > > > > > > >> >>> the topic command is to list all topics whose config A's > > > value > > > > > is > > > > > > B. > > > > > > > >> With > > > > > > > >> >>> topic name regex then we have to first retrieve __all__ > > > > topics's > > > > > > > >> >>> description info and then filter at the client end, which > > > will > > > > > be > > > > > > a > > > > > > > >> huge > > > > > > > >> >>> burden on ZK. > > > > > > > >> >>> > > > > > > > >> >>> 3. Config K-Vs in create topic: this is related to the > > > > previous > > > > > > > point; > > > > > > > >> >>> maybe we can add another metadata K-V or just a metadata > > > > string > > > > > > > along > > > > > > > >> side > > > > > > > >> >>> with config K-V in create topic like we did for offset > > > commit > > > > > > > >> request. This > > > > > > > >> >>> field can be quite useful in storing information like > > > "owner" > > > > of > > > > > > the > > > > > > > >> topic > > > > > > > >> >>> who issue the create command, etc, which is quite > > important > > > > for > > > > > a > > > > > > > >> >>> multi-tenant setting. Then in the describe topic request > > we > > > > can > > > > > > also > > > > > > > >> batch > > > > > > > >> >>> on regex of the metadata field. > > > > > > > >> >>> > > > > > > > >> >>> 4. Today all the admin operations are async in the sense > > > that > > > > > > > command > > > > > > > >> will > > > > > > > >> >>> return once it is written in ZK, and that is why we need > > > extra > > > > > > > >> verification > > > > > > > >> >>> like testUtil.waitForTopicCreated() / verify partition > > > > > > reassignment > > > > > > > >> >>> request, etc. With admin requests we could add a flag to > > > > enable > > > > > / > > > > > > > >> disable > > > > > > > >> >>> synchronous requests; when it is turned on, the response > > > will > > > > > not > > > > > > > >> return > > > > > > > >> >>> until the request has been completed. And for async > > requests > > > > we > > > > > > can > > > > > > > >> add a > > > > > > > >> >>> "token" field in the response, and then only need a > > general > > > > > "admin > > > > > > > >> >>> verification request" with the given token to check if the > > > > async > > > > > > > >> request > > > > > > > >> >>> has been completed. > > > > > > > >> >>> > > > > > > > >> >>> 5. +1 for extending Metadata request to include > > controller / > > > > > > > >> coordinator > > > > > > > >> >>> information, and then we can remove the ConsumerMetadata / > > > > > > > >> ClusterMetadata > > > > > > > >> >>> requests. > > > > > > > >> >>> > > > > > > > >> >>> Guozhang > > > > > > > >> >>> > > > > > > > >> >>> On Tue, Mar 3, 2015 at 10:23 AM, Joel Koshy < > > > > > jjkosh...@gmail.com> > > > > > > > >> wrote: > > > > > > > >> >>> > > > > > > > >> >>>> Thanks for sending that out Joe - I don't think I will be > > > > able > > > > > to > > > > > > > >> make > > > > > > > >> >>>> it today, so if notes can be sent out afterward that > > would > > > be > > > > > > > great. > > > > > > > >> >>>> > > > > > > > >> >>>> On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen Shapira > > > wrote: > > > > > > > >> >>>> > Thanks for sending this out Joe. Looking forward to > > > > chatting > > > > > > with > > > > > > > >> >>>> everyone :) > > > > > > > >> >>>> > > > > > > > > >> >>>> > On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein < > > > > > > joe.st...@stealth.ly> > > > > > > > >> wrote: > > > > > > > >> >>>> > > Hey, I just sent out a google hangout invite to all > > > pmc, > > > > > > > >> committers > > > > > > > >> >>>> and > > > > > > > >> >>>> > > everyone I found working on a KIP. If I missed anyone > > > in > > > > > the > > > > > > > >> invite > > > > > > > >> >>>> please > > > > > > > >> >>>> > > let me know and can update it, np. > > > > > > > >> >>>> > > > > > > > > > >> >>>> > > We should do this every Tuesday @ 2pm Eastern Time. > > > Maybe > > > > > we > > > > > > > can > > > > > > > >> get > > > > > > > >> >>>> INFRA > > > > > > > >> >>>> > > help to make a google account so we can manage > > better? > > > > > > > >> >>>> > > > > > > > > > >> >>>> > > To discuss > > > > > > > >> >>>> > > > > > > > > > >> >>>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > >> >>>> > > in progress and related JIRA that are interdependent > > > and > > > > > > common > > > > > > > >> work. > > > > > > > >> >>>> > > > > > > > > > >> >>>> > > ~ Joe Stein > > > > > > > >> >>>> > > > > > > > > > >> >>>> > > On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps < > > > > > > > jay.kr...@gmail.com> > > > > > > > >> >>>> wrote: > > > > > > > >> >>>> > > > > > > > > > >> >>>> > >> Let's stay on Google hangouts that will also record > > > and > > > > > make > > > > > > > the > > > > > > > >> >>>> sessions > > > > > > > >> >>>> > >> available on youtube. > > > > > > > >> >>>> > >> > > > > > > > >> >>>> > >> -Jay > > > > > > > >> >>>> > >> > > > > > > > >> >>>> > >> On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman < > > > > > > > >> >>>> jholo...@cloudera.com> > > > > > > > >> >>>> > >> wrote: > > > > > > > >> >>>> > >> > > > > > > > >> >>>> > >> > Jay / Joe > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > We're happy to send out a Webex for this purpose. > > We > > > > > could > > > > > > > >> record > > > > > > > >> >>>> the > > > > > > > >> >>>> > >> > sessions if there is interest and publish them > > out. > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > Thanks > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > Jeff > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps < > > > > > > > >> jay.kr...@gmail.com> > > > > > > > >> >>>> wrote: > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > Let's try to get the technical hang-ups sorted > > > out, > > > > > > > though. > > > > > > > >> I > > > > > > > >> >>>> really > > > > > > > >> >>>> > >> > think > > > > > > > >> >>>> > >> > > there is some benefit to live discussion vs > > > > writing. I > > > > > > am > > > > > > > >> >>>> hopeful that > > > > > > > >> >>>> > >> if > > > > > > > >> >>>> > >> > > we post instructions and give ourselves a few > > > > attempts > > > > > > we > > > > > > > >> can > > > > > > > >> >>>> get it > > > > > > > >> >>>> > >> > > working. > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > Tuesday at that time would work for me...any > > > > > objections? > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > -Jay > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein < > > > > > > > >> joe.st...@stealth.ly > > > > > > > >> >>>> > > > > > > > > >> >>>> > >> wrote: > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > Weekly would be great maybe like every > > Tuesday ~ > > > > 1pm > > > > > > ET > > > > > > > / > > > > > > > >> 10am > > > > > > > >> >>>> PT > > > > > > > >> >>>> > >> ???? > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > I don't mind google hangout but there is > > always > > > > some > > > > > > > >> issue or > > > > > > > >> >>>> > >> whatever > > > > > > > >> >>>> > >> > so > > > > > > > >> >>>> > >> > > > we know the apache irc channel works. We can > > > start > > > > > > there > > > > > > > >> and > > > > > > > >> >>>> see how > > > > > > > >> >>>> > >> it > > > > > > > >> >>>> > >> > > > goes? We can pull transcripts too and > > associate > > > to > > > > > > > >> tickets if > > > > > > > >> >>>> need be > > > > > > > >> >>>> > >> > > makes > > > > > > > >> >>>> > >> > > > it helpful for things. > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > ~ Joestein > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > On Tue, Feb 24, 2015 at 11:10 AM, Jay Kreps < > > > > > > > >> >>>> jay.kr...@gmail.com> > > > > > > > >> >>>> > >> > wrote: > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > > We'd talked about doing a Google Hangout to > > > chat > > > > > > about > > > > > > > >> this. > > > > > > > >> >>>> What > > > > > > > >> >>>> > >> > about > > > > > > > >> >>>> > >> > > > > generalizing that a little further...I > > > actually > > > > > > think > > > > > > > it > > > > > > > >> >>>> would be > > > > > > > >> >>>> > >> > good > > > > > > > >> >>>> > >> > > > for > > > > > > > >> >>>> > >> > > > > everyone spending a reasonable chunk of > > their > > > > week > > > > > > on > > > > > > > >> Kafka > > > > > > > >> >>>> stuff > > > > > > > >> >>>> > >> to > > > > > > > >> >>>> > >> > > > maybe > > > > > > > >> >>>> > >> > > > > sync up once a week. I think we could use > > time > > > > to > > > > > > talk > > > > > > > >> >>>> through > > > > > > > >> >>>> > >> design > > > > > > > >> >>>> > >> > > > > stuff, make sure we are on top of code > > > reviews, > > > > > talk > > > > > > > >> through > > > > > > > >> >>>> any > > > > > > > >> >>>> > >> > tricky > > > > > > > >> >>>> > >> > > > > issues, etc. > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > We can make it publicly available so that > > any > > > > one > > > > > > can > > > > > > > >> follow > > > > > > > >> >>>> along > > > > > > > >> >>>> > >> > who > > > > > > > >> >>>> > >> > > > > likes. > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > Any interest in doing this? If so I'll try > > to > > > > set > > > > > it > > > > > > > up > > > > > > > >> >>>> starting > > > > > > > >> >>>> > >> next > > > > > > > >> >>>> > >> > > > week. > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > -Jay > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > On Tue, Feb 24, 2015 at 3:57 AM, Andrii > > > > Biletskyi > > > > > < > > > > > > > >> >>>> > >> > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > > Hi all, > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > I've updated KIP page, fixed / aligned > > > > document > > > > > > > >> structure. > > > > > > > >> >>>> Also I > > > > > > > >> >>>> > >> > > added > > > > > > > >> >>>> > >> > > > > > some > > > > > > > >> >>>> > >> > > > > > very initial proposal for AdminClient so > > we > > > > have > > > > > > > >> something > > > > > > > >> >>>> to > > > > > > > >> >>>> > >> start > > > > > > > >> >>>> > >> > > > from > > > > > > > >> >>>> > >> > > > > > while > > > > > > > >> >>>> > >> > > > > > discussing the KIP. > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > >> >>>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > Thanks, > > > > > > > >> >>>> > >> > > > > > Andrii Biletskyi > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > On Wed, Feb 18, 2015 at 9:01 PM, Andrii > > > > > Biletskyi > > > > > > < > > > > > > > >> >>>> > >> > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > > Jay, > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > Re error messages: you are right, in > > most > > > > > cases > > > > > > > >> client > > > > > > > >> >>>> will > > > > > > > >> >>>> > >> have > > > > > > > >> >>>> > >> > > > enough > > > > > > > >> >>>> > >> > > > > > > context to show descriptive error > > message. > > > > My > > > > > > > >> concern is > > > > > > > >> >>>> that > > > > > > > >> >>>> > >> we > > > > > > > >> >>>> > >> > > will > > > > > > > >> >>>> > >> > > > > > have > > > > > > > >> >>>> > >> > > > > > > to > > > > > > > >> >>>> > >> > > > > > > add lots of new error codes for each > > > > possible > > > > > > > >> error. Of > > > > > > > >> >>>> course, > > > > > > > >> >>>> > >> > we > > > > > > > >> >>>> > >> > > > > could > > > > > > > >> >>>> > >> > > > > > > reuse > > > > > > > >> >>>> > >> > > > > > > some of existing like > > > > > > UknownTopicOrPartitionCode, > > > > > > > >> but we > > > > > > > >> >>>> will > > > > > > > >> >>>> > >> > also > > > > > > > >> >>>> > >> > > > need > > > > > > > >> >>>> > >> > > > > > to > > > > > > > >> >>>> > >> > > > > > > add smth like: TopicAlreadyExistsCode, > > > > > > > >> >>>> TopicConfigInvalid (both > > > > > > > >> >>>> > >> > for > > > > > > > >> >>>> > >> > > > > topic > > > > > > > >> >>>> > >> > > > > > > name and config, and probably user would > > > > like > > > > > to > > > > > > > >> know > > > > > > > >> >>>> what > > > > > > > >> >>>> > >> > exactly > > > > > > > >> >>>> > >> > > > > > > is wrong in his config), > > > > > > InvalidReplicaAssignment, > > > > > > > >> >>>> > >> InternalError > > > > > > > >> >>>> > >> > > > (e.g. > > > > > > > >> >>>> > >> > > > > > > zookeeper failure) etc. > > > > > > > >> >>>> > >> > > > > > > And this is only for TopicCommand, we > > will > > > > > also > > > > > > > >> need to > > > > > > > >> >>>> add > > > > > > > >> >>>> > >> > similar > > > > > > > >> >>>> > >> > > > > stuff > > > > > > > >> >>>> > >> > > > > > > for > > > > > > > >> >>>> > >> > > > > > > ReassignPartitions, PreferredReplica. So > > > > we'll > > > > > > end > > > > > > > >> up > > > > > > > >> >>>> with a > > > > > > > >> >>>> > >> > large > > > > > > > >> >>>> > >> > > > list > > > > > > > >> >>>> > >> > > > > > of > > > > > > > >> >>>> > >> > > > > > > error codes, used only in Admin > > protocol. > > > > > > > >> >>>> > >> > > > > > > Having said that, I agree my proposal is > > > not > > > > > > > >> consistent > > > > > > > >> >>>> with > > > > > > > >> >>>> > >> > other > > > > > > > >> >>>> > >> > > > > cases. > > > > > > > >> >>>> > >> > > > > > > Maybe we can find better solution or > > > > something > > > > > > > >> >>>> in-between. > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > Re Hangout chat: I think it is a great > > > idea. > > > > > > This > > > > > > > >> way we > > > > > > > >> >>>> can > > > > > > > >> >>>> > >> move > > > > > > > >> >>>> > >> > > on > > > > > > > >> >>>> > >> > > > > > > faster. > > > > > > > >> >>>> > >> > > > > > > Let's agree somehow on date/time so > > people > > > > can > > > > > > > join. > > > > > > > >> >>>> Will work > > > > > > > >> >>>> > >> > for > > > > > > > >> >>>> > >> > > me > > > > > > > >> >>>> > >> > > > > > this > > > > > > > >> >>>> > >> > > > > > > and > > > > > > > >> >>>> > >> > > > > > > next week almost anytime if agreed in > > > > advance. > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > Thanks, > > > > > > > >> >>>> > >> > > > > > > Andrii > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > On Wed, Feb 18, 2015 at 7:09 PM, Jay > > > Kreps < > > > > > > > >> >>>> > >> jay.kr...@gmail.com> > > > > > > > >> >>>> > >> > > > > wrote: > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> Hey Andrii, > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> Generally we can do good error handling > > > > > without > > > > > > > >> needing > > > > > > > >> >>>> custom > > > > > > > >> >>>> > >> > > > > > server-side > > > > > > > >> >>>> > >> > > > > > >> messages. I.e. generally the client has > > > the > > > > > > > >> context to > > > > > > > >> >>>> know > > > > > > > >> >>>> > >> that > > > > > > > >> >>>> > >> > > if > > > > > > > >> >>>> > >> > > > it > > > > > > > >> >>>> > >> > > > > > got > > > > > > > >> >>>> > >> > > > > > >> an error that the topic doesn't exist > > to > > > > say > > > > > > > >> "Topic X > > > > > > > >> >>>> doesn't > > > > > > > >> >>>> > >> > > exist" > > > > > > > >> >>>> > >> > > > > > >> rather > > > > > > > >> >>>> > >> > > > > > >> than "error code 14" (or whatever). > > Maybe > > > > > there > > > > > > > are > > > > > > > >> >>>> specific > > > > > > > >> >>>> > >> > cases > > > > > > > >> >>>> > >> > > > > where > > > > > > > >> >>>> > >> > > > > > >> this is hard? If we want to add > > > server-side > > > > > > error > > > > > > > >> >>>> messages we > > > > > > > >> >>>> > >> > > really > > > > > > > >> >>>> > >> > > > > do > > > > > > > >> >>>> > >> > > > > > >> need to do this in a consistent way > > > across > > > > > the > > > > > > > >> protocol. > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> I still have a bunch of open questions > > > here > > > > > > from > > > > > > > my > > > > > > > >> >>>> previous > > > > > > > >> >>>> > >> > > list. I > > > > > > > >> >>>> > >> > > > > > will > > > > > > > >> >>>> > >> > > > > > >> be out for the next few days for Strata > > > > > though. > > > > > > > >> Maybe > > > > > > > >> >>>> we could > > > > > > > >> >>>> > >> > do > > > > > > > >> >>>> > >> > > a > > > > > > > >> >>>> > >> > > > > > Google > > > > > > > >> >>>> > >> > > > > > >> Hangout chat on any open issues some > > time > > > > > > towards > > > > > > > >> the > > > > > > > >> >>>> end of > > > > > > > >> >>>> > >> > next > > > > > > > >> >>>> > >> > > > week > > > > > > > >> >>>> > >> > > > > > for > > > > > > > >> >>>> > >> > > > > > >> anyone interested in this ticket? I > > have > > > a > > > > > > > feeling > > > > > > > >> that > > > > > > > >> >>>> might > > > > > > > >> >>>> > >> > > > progress > > > > > > > >> >>>> > >> > > > > > >> things a little faster than email--I > > > think > > > > we > > > > > > > >> could talk > > > > > > > >> >>>> > >> through > > > > > > > >> >>>> > >> > > > those > > > > > > > >> >>>> > >> > > > > > >> issues I brought up fairly quickly... > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> -Jay > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> On Wed, Feb 18, 2015 at 7:27 AM, Andrii > > > > > > > Biletskyi < > > > > > > > >> >>>> > >> > > > > > >> andrii.bilets...@stealth.ly> wrote: > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > Hi all, > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > I'm trying to address some of the > > > issues > > > > > > which > > > > > > > >> were > > > > > > > >> >>>> > >> mentioned > > > > > > > >> >>>> > >> > > > > earlier > > > > > > > >> >>>> > >> > > > > > >> about > > > > > > > >> >>>> > >> > > > > > >> > Admin RQ/RP format. One of those was > > > > about > > > > > > > >> batching > > > > > > > >> >>>> > >> > operations. > > > > > > > >> >>>> > >> > > > What > > > > > > > >> >>>> > >> > > > > > if > > > > > > > >> >>>> > >> > > > > > >> we > > > > > > > >> >>>> > >> > > > > > >> > follow TopicCommand approach and let > > > > people > > > > > > > >> specify > > > > > > > >> >>>> > >> topic-name > > > > > > > >> >>>> > >> > > by > > > > > > > >> >>>> > >> > > > > > >> regexp - > > > > > > > >> >>>> > >> > > > > > >> > would that cover most of the use > > cases? > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > Secondly, is what information should > > we > > > > > > > generally > > > > > > > >> >>>> provide in > > > > > > > >> >>>> > >> > > Admin > > > > > > > >> >>>> > >> > > > > > >> > responses. > > > > > > > >> >>>> > >> > > > > > >> > I realize that Admin commands don't > > > imply > > > > > > they > > > > > > > >> will > > > > > > > >> >>>> be used > > > > > > > >> >>>> > >> > only > > > > > > > >> >>>> > >> > > > in > > > > > > > >> >>>> > >> > > > > > CLI > > > > > > > >> >>>> > >> > > > > > >> > but, > > > > > > > >> >>>> > >> > > > > > >> > it seems to me, CLI is a very > > important > > > > > > client > > > > > > > >> of this > > > > > > > >> >>>> > >> > feature. > > > > > > > >> >>>> > >> > > In > > > > > > > >> >>>> > >> > > > > > this > > > > > > > >> >>>> > >> > > > > > >> > case, > > > > > > > >> >>>> > >> > > > > > >> > seems logical, we would like to > > provide > > > > > users > > > > > > > >> with > > > > > > > >> >>>> rich > > > > > > > >> >>>> > >> > > experience > > > > > > > >> >>>> > >> > > > > in > > > > > > > >> >>>> > >> > > > > > >> terms > > > > > > > >> >>>> > >> > > > > > >> > of > > > > > > > >> >>>> > >> > > > > > >> > getting results / errors of the > > > executed > > > > > > > >> commands. > > > > > > > >> >>>> Usually > > > > > > > >> >>>> > >> we > > > > > > > >> >>>> > >> > > > supply > > > > > > > >> >>>> > >> > > > > > >> with > > > > > > > >> >>>> > >> > > > > > >> > responses only errorCode, which looks > > > > very > > > > > > > >> limiting, > > > > > > > >> >>>> in case > > > > > > > >> >>>> > >> > of > > > > > > > >> >>>> > >> > > > CLI > > > > > > > >> >>>> > >> > > > > we > > > > > > > >> >>>> > >> > > > > > >> may > > > > > > > >> >>>> > >> > > > > > >> > want to print human readable error > > > > > > description. > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > So, taking into account previous item > > > > about > > > > > > > >> batching, > > > > > > > >> >>>> what > > > > > > > >> >>>> > >> do > > > > > > > >> >>>> > >> > > you > > > > > > > >> >>>> > >> > > > > > think > > > > > > > >> >>>> > >> > > > > > >> > about > > > > > > > >> >>>> > >> > > > > > >> > having smth like: > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > ('create' doesn't support regexp) > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicRequest => TopicName > > > > Partitions > > > > > > > >> Replicas > > > > > > > >> >>>> > >> > > > > ReplicaAssignment > > > > > > > >> >>>> > >> > > > > > >> > [Config] > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicResponse => ErrorCode > > > > > > > ErrorDescription > > > > > > > >> >>>> > >> > > > > > >> > ErrorCode => int16 > > > > > > > >> >>>> > >> > > > > > >> > ErrorDescription => string (empty > > if > > > > > > > >> successful) > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > AlterTopicRequest -> TopicNameRegexp > > > > > > Partitions > > > > > > > >> >>>> > >> > > ReplicaAssignment > > > > > > > >> >>>> > >> > > > > > >> > [AddedConfig] [DeletedConfig] > > > > > > > >> >>>> > >> > > > > > >> > AlterTopicResponse -> [TopicName > > > > ErrorCode > > > > > > > >> >>>> ErrorDescription] > > > > > > > >> >>>> > >> > > > > > >> > CommandErrorCode > > > CommandErrorDescription > > > > > > > >> >>>> > >> > > > > > >> > CommandErrorCode => int16 > > > > > > > >> >>>> > >> > > > > > >> > CommandErrorDescription => string > > > > > (nonempty > > > > > > > in > > > > > > > >> case > > > > > > > >> >>>> of > > > > > > > >> >>>> > >> fatal > > > > > > > >> >>>> > >> > > > > error, > > > > > > > >> >>>> > >> > > > > > >> e.g. > > > > > > > >> >>>> > >> > > > > > >> > we couldn't get topics by regexp) > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > DescribeTopicRequest -> > > TopicNameRegexp > > > > > > > >> >>>> > >> > > > > > >> > DescribeTopicResponse -> [TopicName > > > > > > > >> TopicDescription > > > > > > > >> >>>> > >> ErrorCode > > > > > > > >> >>>> > >> > > > > > >> > ErrorDescription] CommandErrorCode > > > > > > > >> >>>> CommandErrorDescription > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > Also, any thoughts about our > > discussion > > > > > > > regarding > > > > > > > >> >>>> re-routing > > > > > > > >> >>>> > >> > > > > facility? > > > > > > > >> >>>> > >> > > > > > >> In > > > > > > > >> >>>> > >> > > > > > >> > my > > > > > > > >> >>>> > >> > > > > > >> > understanding, it is like between > > > > > augmenting > > > > > > > >> >>>> > >> > > TopicMetadataRequest > > > > > > > >> >>>> > >> > > > > > >> > (to include at least controllerId) > > and > > > > > > > >> implementing > > > > > > > >> >>>> new > > > > > > > >> >>>> > >> > generic > > > > > > > >> >>>> > >> > > > > > >> re-routing > > > > > > > >> >>>> > >> > > > > > >> > facility so sending messages to > > > > controller > > > > > > will > > > > > > > >> be > > > > > > > >> >>>> handled > > > > > > > >> >>>> > >> by > > > > > > > >> >>>> > >> > > it. > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > Thanks, > > > > > > > >> >>>> > >> > > > > > >> > Andrii Biletskyi > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > On Mon, Feb 16, 2015 at 5:26 PM, > > Andrii > > > > > > > >> Biletskyi < > > > > > > > >> >>>> > >> > > > > > >> > andrii.bilets...@stealth.ly> wrote: > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > > @Guozhang: > > > > > > > >> >>>> > >> > > > > > >> > > Thanks for your comments, I've > > > answered > > > > > > some > > > > > > > of > > > > > > > >> >>>> those. The > > > > > > > >> >>>> > >> > > main > > > > > > > >> >>>> > >> > > > > > thing > > > > > > > >> >>>> > >> > > > > > >> is > > > > > > > >> >>>> > >> > > > > > >> > > having merged request for > > > > > > > >> >>>> create-alter-delete-describe - I > > > > > > > >> >>>> > >> > > have > > > > > > > >> >>>> > >> > > > > some > > > > > > > >> >>>> > >> > > > > > >> > > concerns about this approach. > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > > @*Jay*: > > > > > > > >> >>>> > >> > > > > > >> > > I see that introduced > > > > > ClusterMetadaRequest > > > > > > is > > > > > > > >> also > > > > > > > >> >>>> one of > > > > > > > >> >>>> > >> > the > > > > > > > >> >>>> > >> > > > > > >> concerns. > > > > > > > >> >>>> > >> > > > > > >> > We > > > > > > > >> >>>> > >> > > > > > >> > > can solve it if we implement > > > re-routing > > > > > > > >> facility. > > > > > > > >> >>>> But I > > > > > > > >> >>>> > >> > agree > > > > > > > >> >>>> > >> > > > with > > > > > > > >> >>>> > >> > > > > > >> > > Guozhang - it will make clients' > > > > > internals > > > > > > a > > > > > > > >> little > > > > > > > >> >>>> bit > > > > > > > >> >>>> > >> > easier > > > > > > > >> >>>> > >> > > > but > > > > > > > >> >>>> > >> > > > > > >> this > > > > > > > >> >>>> > >> > > > > > >> > > seems to be a complex logic to > > > > implement > > > > > > and > > > > > > > >> >>>> support then. > > > > > > > >> >>>> > >> > > > > > Especially > > > > > > > >> >>>> > >> > > > > > >> for > > > > > > > >> >>>> > >> > > > > > >> > > Fetch and Produce (even if we add > > > > > > re-routing > > > > > > > >> later > > > > > > > >> >>>> for > > > > > > > >> >>>> > >> these > > > > > > > >> >>>> > >> > > > > > >> requests). > > > > > > > >> >>>> > >> > > > > > >> > > Also people will tend to avoid this > > > > > > > re-routing > > > > > > > >> >>>> facility > > > > > > > >> >>>> > >> and > > > > > > > >> >>>> > >> > > hold > > > > > > > >> >>>> > >> > > > > > local > > > > > > > >> >>>> > >> > > > > > >> > > cluster cache to ensure their > > > > > high-priority > > > > > > > >> requests > > > > > > > >> >>>> > >> (which > > > > > > > >> >>>> > >> > > some > > > > > > > >> >>>> > >> > > > > of > > > > > > > >> >>>> > >> > > > > > >> the > > > > > > > >> >>>> > >> > > > > > >> > > admin requests are) not sent to > > some > > > > busy > > > > > > > >> broker > > > > > > > >> >>>> where > > > > > > > >> >>>> > >> they > > > > > > > >> >>>> > >> > > wait > > > > > > > >> >>>> > >> > > > > to > > > > > > > >> >>>> > >> > > > > > be > > > > > > > >> >>>> > >> > > > > > >> > > routed to the correct one. > > > > > > > >> >>>> > >> > > > > > >> > > As pointed out by Jun here ( > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > >> >>>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1772?focusedCommentId=14234530&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234530 > > > > > > > >> >>>> > >> > > > > > >> > ) > > > > > > > >> >>>> > >> > > > > > >> > > to solve the issue we might > > > introduce a > > > > > > > message > > > > > > > >> >>>> type to > > > > > > > >> >>>> > >> get > > > > > > > >> >>>> > >> > > > > cluster > > > > > > > >> >>>> > >> > > > > > >> > state. > > > > > > > >> >>>> > >> > > > > > >> > > But I agree we can just update > > > > > > > >> >>>> TopicMetadataResponse to > > > > > > > >> >>>> > >> > > include > > > > > > > >> >>>> > >> > > > > > >> > > controllerId (and probably smth > > > else). > > > > > > > >> >>>> > >> > > > > > >> > > What are you thougths? > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > > Thanks, > > > > > > > >> >>>> > >> > > > > > >> > > Andrii > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > > On Thu, Feb 12, 2015 at 8:31 AM, > > > > Guozhang > > > > > > > Wang > > > > > > > >> < > > > > > > > >> >>>> > >> > > > > wangg...@gmail.com> > > > > > > > >> >>>> > >> > > > > > >> > wrote: > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> I think for the topics commands we > > > can > > > > > > > >> actually > > > > > > > >> >>>> merge > > > > > > > >> >>>> > >> > > > > > >> > >> create/alter/delete/describe as > > one > > > > > > request > > > > > > > >> type > > > > > > > >> >>>> since > > > > > > > >> >>>> > >> > their > > > > > > > >> >>>> > >> > > > > > formats > > > > > > > >> >>>> > >> > > > > > >> are > > > > > > > >> >>>> > >> > > > > > >> > >> very much similar, and keep > > > > list-topics > > > > > > and > > > > > > > >> others > > > > > > > >> >>>> like > > > > > > > >> >>>> > >> > > > > > >> > >> partition-reassignment / > > > > > > > >> preferred-leader-election > > > > > > > >> >>>> as > > > > > > > >> >>>> > >> > > separate > > > > > > > >> >>>> > >> > > > > > >> request > > > > > > > >> >>>> > >> > > > > > >> > >> types, I also left some other > > > comments > > > > > on > > > > > > > the > > > > > > > >> RB ( > > > > > > > >> >>>> > >> > > > > > >> > >> > > https://reviews.apache.org/r/29301/ > > > ). > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> On Wed, Feb 11, 2015 at 2:04 PM, > > Jay > > > > > > Kreps < > > > > > > > >> >>>> > >> > > > jay.kr...@gmail.com> > > > > > > > >> >>>> > >> > > > > > >> wrote: > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > Yeah I totally agree that we > > don't > > > > > want > > > > > > to > > > > > > > >> just > > > > > > > >> >>>> have > > > > > > > >> >>>> > >> one > > > > > > > >> >>>> > >> > > "do > > > > > > > >> >>>> > >> > > > > > admin > > > > > > > >> >>>> > >> > > > > > >> > >> stuff" > > > > > > > >> >>>> > >> > > > > > >> > >> > command that has the union of > > all > > > > > > > >> parameters. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > What I am saying is that command > > > > line > > > > > > > tools > > > > > > > >> are > > > > > > > >> >>>> one > > > > > > > >> >>>> > >> > client > > > > > > > >> >>>> > >> > > of > > > > > > > >> >>>> > >> > > > > the > > > > > > > >> >>>> > >> > > > > > >> > >> > administrative apis, but these > > > will > > > > be > > > > > > > used > > > > > > > >> in a > > > > > > > >> >>>> number > > > > > > > >> >>>> > >> > of > > > > > > > >> >>>> > >> > > > > > >> scenarios > > > > > > > >> >>>> > >> > > > > > >> > so > > > > > > > >> >>>> > >> > > > > > >> > >> > they should make logical sense > > > even > > > > in > > > > > > the > > > > > > > >> >>>> absence of > > > > > > > >> >>>> > >> the > > > > > > > >> >>>> > >> > > > > command > > > > > > > >> >>>> > >> > > > > > >> line > > > > > > > >> >>>> > >> > > > > > >> > >> > tool. Hence comments like trying > > > to > > > > > > > clarify > > > > > > > >> the > > > > > > > >> >>>> > >> > > relationship > > > > > > > >> >>>> > >> > > > > > >> between > > > > > > > >> >>>> > >> > > > > > >> > >> > ClusterMetadata and > > > > > > TopicMetadata...these > > > > > > > >> kinds > > > > > > > >> >>>> of > > > > > > > >> >>>> > >> things > > > > > > > >> >>>> > >> > > > > really > > > > > > > >> >>>> > >> > > > > > >> need > > > > > > > >> >>>> > >> > > > > > >> > >> to be > > > > > > > >> >>>> > >> > > > > > >> > >> > thought through. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > Hope that makes sense. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > -Jay > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > On Wed, Feb 11, 2015 at 1:41 PM, > > > > > Andrii > > > > > > > >> >>>> Biletskyi < > > > > > > > >> >>>> > >> > > > > > >> > >> > andrii.bilets...@stealth.ly> > > > wrote: > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > Jay, > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > Thanks for answering. You > > > > understood > > > > > > > >> >>>> correctly, most > > > > > > > >> >>>> > >> of > > > > > > > >> >>>> > >> > > my > > > > > > > >> >>>> > >> > > > > > >> comments > > > > > > > >> >>>> > >> > > > > > >> > >> were > > > > > > > >> >>>> > >> > > > > > >> > >> > > related to your point 1) - > > about > > > > > "well > > > > > > > >> >>>> thought-out" > > > > > > > >> >>>> > >> > apis. > > > > > > > >> >>>> > >> > > > > Also, > > > > > > > >> >>>> > >> > > > > > >> yes, > > > > > > > >> >>>> > >> > > > > > >> > >> as I > > > > > > > >> >>>> > >> > > > > > >> > >> > > understood we would like to > > > > > introduce > > > > > > a > > > > > > > >> single > > > > > > > >> >>>> > >> unified > > > > > > > >> >>>> > >> > > CLI > > > > > > > >> >>>> > >> > > > > tool > > > > > > > >> >>>> > >> > > > > > >> with > > > > > > > >> >>>> > >> > > > > > >> > >> > > centralized server-side > > request > > > > > > handling > > > > > > > >> for > > > > > > > >> >>>> lots of > > > > > > > >> >>>> > >> > > > existing > > > > > > > >> >>>> > >> > > > > > >> ones > > > > > > > >> >>>> > >> > > > > > >> > >> (incl. > > > > > > > >> >>>> > >> > > > > > >> > >> > > TopicCommand, > > > CommitOffsetChecker, > > > > > > > >> >>>> > >> ReassignPartitions, > > > > > > > >> >>>> > >> > > smth > > > > > > > >> >>>> > >> > > > > > else > > > > > > > >> >>>> > >> > > > > > >> if > > > > > > > >> >>>> > >> > > > > > >> > >> added > > > > > > > >> >>>> > >> > > > > > >> > >> > > in future). In our previous > > > > > > discussion ( > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> https://issues.apache.org/jira/browse/KAFKA-1694) > > > > > > > >> >>>> > >> > people > > > > > > > >> >>>> > >> > > > > said > > > > > > > >> >>>> > >> > > > > > >> > they'd > > > > > > > >> >>>> > >> > > > > > >> > >> > > rather > > > > > > > >> >>>> > >> > > > > > >> > >> > > have a separate message for > > each > > > > > > > command, > > > > > > > >> so, > > > > > > > >> >>>> yes, > > > > > > > >> >>>> > >> this > > > > > > > >> >>>> > >> > > > way I > > > > > > > >> >>>> > >> > > > > > >> came > > > > > > > >> >>>> > >> > > > > > >> > to > > > > > > > >> >>>> > >> > > > > > >> > >> 1-1 > > > > > > > >> >>>> > >> > > > > > >> > >> > > mapping between commands in > > the > > > > tool > > > > > > and > > > > > > > >> >>>> protocol > > > > > > > >> >>>> > >> > > > additions. > > > > > > > >> >>>> > >> > > > > > But > > > > > > > >> >>>> > >> > > > > > >> I > > > > > > > >> >>>> > >> > > > > > >> > >> might > > > > > > > >> >>>> > >> > > > > > >> > >> > be > > > > > > > >> >>>> > >> > > > > > >> > >> > > wrong. > > > > > > > >> >>>> > >> > > > > > >> > >> > > At the end I just try to start > > > > > > > discussion > > > > > > > >> how > > > > > > > >> >>>> at > > > > > > > >> >>>> > >> least > > > > > > > >> >>>> > >> > > > > > generally > > > > > > > >> >>>> > >> > > > > > >> > this > > > > > > > >> >>>> > >> > > > > > >> > >> > > protocol should look like. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > Thanks, > > > > > > > >> >>>> > >> > > > > > >> > >> > > Andrii > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > On Wed, Feb 11, 2015 at 11:10 > > > PM, > > > > > Jay > > > > > > > >> Kreps < > > > > > > > >> >>>> > >> > > > > > jay.kr...@gmail.com > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> wrote: > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > Hey Andrii, > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > To answer your earlier > > > question > > > > we > > > > > > > just > > > > > > > >> >>>> really > > > > > > > >> >>>> > >> can't > > > > > > > >> >>>> > >> > be > > > > > > > >> >>>> > >> > > > > > adding > > > > > > > >> >>>> > >> > > > > > >> any > > > > > > > >> >>>> > >> > > > > > >> > >> more > > > > > > > >> >>>> > >> > > > > > >> > >> > > > scala protocol objects. > > These > > > > > things > > > > > > > are > > > > > > > >> >>>> super hard > > > > > > > >> >>>> > >> > to > > > > > > > >> >>>> > >> > > > > > maintain > > > > > > > >> >>>> > >> > > > > > >> > >> because > > > > > > > >> >>>> > >> > > > > > >> > >> > > > they hand code the byte > > > parsing > > > > > and > > > > > > > >> don't > > > > > > > >> >>>> have good > > > > > > > >> >>>> > >> > > > > > versioning > > > > > > > >> >>>> > >> > > > > > >> > >> support. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > Since we are already > > planning > > > on > > > > > > > >> converting > > > > > > > >> >>>> we > > > > > > > >> >>>> > >> > > definitely > > > > > > > >> >>>> > >> > > > > > don't > > > > > > > >> >>>> > >> > > > > > >> > >> want to > > > > > > > >> >>>> > >> > > > > > >> > >> > > add > > > > > > > >> >>>> > >> > > > > > >> > >> > > > a ton more of these--they > > are > > > > > total > > > > > > > tech > > > > > > > >> >>>> debt. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > What does it mean that the > > > > changes > > > > > > are > > > > > > > >> >>>> isolated > > > > > > > >> >>>> > >> from > > > > > > > >> >>>> > >> > > the > > > > > > > >> >>>> > >> > > > > > >> current > > > > > > > >> >>>> > >> > > > > > >> > >> code > > > > > > > >> >>>> > >> > > > > > >> > >> > > base? > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > I actually didn't understand > > > the > > > > > > > >> remaining > > > > > > > >> >>>> > >> comments, > > > > > > > >> >>>> > >> > > > which > > > > > > > >> >>>> > >> > > > > of > > > > > > > >> >>>> > >> > > > > > >> the > > > > > > > >> >>>> > >> > > > > > >> > >> > points > > > > > > > >> >>>> > >> > > > > > >> > >> > > > are you responding to? > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > Maybe one sticking point > > here > > > is > > > > > > that > > > > > > > it > > > > > > > >> >>>> seems like > > > > > > > >> >>>> > >> > you > > > > > > > >> >>>> > >> > > > > want > > > > > > > >> >>>> > >> > > > > > to > > > > > > > >> >>>> > >> > > > > > >> > make > > > > > > > >> >>>> > >> > > > > > >> > >> > some > > > > > > > >> >>>> > >> > > > > > >> > >> > > > kind of tool, and you have > > > made > > > > a > > > > > > 1-1 > > > > > > > >> mapping > > > > > > > >> >>>> > >> between > > > > > > > >> >>>> > >> > > > > > commands > > > > > > > >> >>>> > >> > > > > > >> you > > > > > > > >> >>>> > >> > > > > > >> > >> > > imagine > > > > > > > >> >>>> > >> > > > > > >> > >> > > > in the tool and protocol > > > > > additions. > > > > > > I > > > > > > > >> want > > > > > > > >> >>>> to make > > > > > > > >> >>>> > >> > sure > > > > > > > >> >>>> > >> > > > we > > > > > > > >> >>>> > >> > > > > > >> don't > > > > > > > >> >>>> > >> > > > > > >> > do > > > > > > > >> >>>> > >> > > > > > >> > >> > that. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > The protocol needs to be > > > really > > > > > > really > > > > > > > >> well > > > > > > > >> >>>> thought > > > > > > > >> >>>> > >> > out > > > > > > > >> >>>> > >> > > > > > against > > > > > > > >> >>>> > >> > > > > > >> > many > > > > > > > >> >>>> > >> > > > > > >> > >> > use > > > > > > > >> >>>> > >> > > > > > >> > >> > > > cases so it should make > > > perfect > > > > > > > logical > > > > > > > >> >>>> sense in > > > > > > > >> >>>> > >> the > > > > > > > >> >>>> > >> > > > > absence > > > > > > > >> >>>> > >> > > > > > of > > > > > > > >> >>>> > >> > > > > > >> > >> knowing > > > > > > > >> >>>> > >> > > > > > >> > >> > > the > > > > > > > >> >>>> > >> > > > > > >> > >> > > > command line tool, right? > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > -Jay > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > On Wed, Feb 11, 2015 at > > 11:57 > > > > AM, > > > > > > > Andrii > > > > > > > >> >>>> Biletskyi > > > > > > > >> >>>> > >> < > > > > > > > >> >>>> > >> > > > > > >> > >> > > > andrii.bilets...@stealth.ly > > > > > > > > wrote: > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Hey Jay, > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > I would like to continue > > > this > > > > > > > >> discussion > > > > > > > >> >>>> as it > > > > > > > >> >>>> > >> seem > > > > > > > >> >>>> > >> > > > there > > > > > > > >> >>>> > >> > > > > > is > > > > > > > >> >>>> > >> > > > > > >> no > > > > > > > >> >>>> > >> > > > > > >> > >> > > progress > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > here. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > First of all, could you > > > please > > > > > > > explain > > > > > > > >> >>>> what did > > > > > > > >> >>>> > >> you > > > > > > > >> >>>> > >> > > > mean > > > > > > > >> >>>> > >> > > > > in > > > > > > > >> >>>> > >> > > > > > >> 2? > > > > > > > >> >>>> > >> > > > > > >> > How > > > > > > > >> >>>> > >> > > > > > >> > >> > > > exactly > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > are we going to migrate to > > > the > > > > > new > > > > > > > >> java > > > > > > > >> >>>> protocol > > > > > > > >> >>>> > >> > > > > > definitions. > > > > > > > >> >>>> > >> > > > > > >> > And > > > > > > > >> >>>> > >> > > > > > >> > >> why > > > > > > > >> >>>> > >> > > > > > >> > >> > > > it's > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > a blocker for centralized > > > CLI? > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > I agree with you, this > > > feature > > > > > > > >> includes > > > > > > > >> >>>> lots of > > > > > > > >> >>>> > >> > > stuff, > > > > > > > >> >>>> > >> > > > > but > > > > > > > >> >>>> > >> > > > > > >> > >> thankfully > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > almost all changes are > > > > isolated > > > > > > from > > > > > > > >> the > > > > > > > >> >>>> current > > > > > > > >> >>>> > >> > code > > > > > > > >> >>>> > >> > > > > base, > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > so the main thing, I > > think, > > > we > > > > > > need > > > > > > > to > > > > > > > >> >>>> agree is > > > > > > > >> >>>> > >> > RQ/RP > > > > > > > >> >>>> > >> > > > > > format. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > So how can we start > > > discussion > > > > > > about > > > > > > > >> the > > > > > > > >> >>>> concrete > > > > > > > >> >>>> > >> > > > > messages > > > > > > > >> >>>> > >> > > > > > >> > format? > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Can we take ( > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > >> >>>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-ProposedRQ/RPFormat > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > ) > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > as starting point? > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > We had some doubts earlier > > > > > whether > > > > > > > it > > > > > > > >> worth > > > > > > > >> >>>> > >> > > introducing > > > > > > > >> >>>> > >> > > > > one > > > > > > > >> >>>> > >> > > > > > >> > >> generic > > > > > > > >> >>>> > >> > > > > > >> > >> > > Admin > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Request for all commands ( > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> https://issues.apache.org/jira/browse/KAFKA-1694 > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > ) > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > but then everybody agreed > > it > > > > > would > > > > > > > be > > > > > > > >> >>>> better to > > > > > > > >> >>>> > >> > have > > > > > > > >> >>>> > >> > > > > > separate > > > > > > > >> >>>> > >> > > > > > >> > >> message > > > > > > > >> >>>> > >> > > > > > >> > >> > > for > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > each admin command. The > > > > Request > > > > > > part > > > > > > > >> is > > > > > > > >> >>>> really > > > > > > > >> >>>> > >> > > dictated > > > > > > > >> >>>> > >> > > > > > from > > > > > > > >> >>>> > >> > > > > > >> the > > > > > > > >> >>>> > >> > > > > > >> > >> > > command > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > (e.g. TopicCommand) > > > arguments > > > > > > > itself, > > > > > > > >> so > > > > > > > >> >>>> the > > > > > > > >> >>>> > >> > proposed > > > > > > > >> >>>> > >> > > > > > version > > > > > > > >> >>>> > >> > > > > > >> > >> should > > > > > > > >> >>>> > >> > > > > > >> > >> > be > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > fine (let's put aside for > > > now > > > > > > > remarks > > > > > > > >> about > > > > > > > >> >>>> > >> > Optional > > > > > > > >> >>>> > >> > > > > type, > > > > > > > >> >>>> > >> > > > > > >> > >> batching, > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > configs normalization - I > > > > agree > > > > > > with > > > > > > > >> all of > > > > > > > >> >>>> > >> them). > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > So the second part is > > > > Response. > > > > > I > > > > > > > see > > > > > > > >> >>>> there are > > > > > > > >> >>>> > >> two > > > > > > > >> >>>> > >> > > > cases > > > > > > > >> >>>> > >> > > > > > >> here. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > a) "Mutate" requests - > > > > > > > >> Create/Alter/... ; > > > > > > > >> >>>> b) > > > > > > > >> >>>> > >> "Get" > > > > > > > >> >>>> > >> > > > > > requests - > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > List/Describe... > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > a) should only hold > > request > > > > > result > > > > > > > >> >>>> (regardless > > > > > > > >> >>>> > >> what > > > > > > > >> >>>> > >> > > we > > > > > > > >> >>>> > >> > > > > > decide > > > > > > > >> >>>> > >> > > > > > >> > >> about > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > blocking/non-blocking > > > commands > > > > > > > >> execution). > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Usually we provide error > > > code > > > > in > > > > > > > >> response > > > > > > > >> >>>> but > > > > > > > >> >>>> > >> since > > > > > > > >> >>>> > >> > > we > > > > > > > >> >>>> > >> > > > > will > > > > > > > >> >>>> > >> > > > > > >> use > > > > > > > >> >>>> > >> > > > > > >> > >> this > > > > > > > >> >>>> > >> > > > > > >> > >> > in > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > interactive shell we need > > > some > > > > > > human > > > > > > > >> >>>> readable > > > > > > > >> >>>> > >> error > > > > > > > >> >>>> > >> > > > > > >> description > > > > > > > >> >>>> > >> > > > > > >> > - > > > > > > > >> >>>> > >> > > > > > >> > >> so > > > > > > > >> >>>> > >> > > > > > >> > >> > I > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > added errorDesription > > field > > > > > where > > > > > > > you > > > > > > > >> can > > > > > > > >> >>>> at > > > > > > > >> >>>> > >> least > > > > > > > >> >>>> > >> > > > leave > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > exception.getMessage. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > b) in addition to previous > > > > item > > > > > > > >> message > > > > > > > >> >>>> should > > > > > > > >> >>>> > >> hold > > > > > > > >> >>>> > >> > > > > command > > > > > > > >> >>>> > >> > > > > > >> > >> specific > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > response data. We can > > > discuss > > > > in > > > > > > > >> detail > > > > > > > >> >>>> each of > > > > > > > >> >>>> > >> > them > > > > > > > >> >>>> > >> > > > but > > > > > > > >> >>>> > >> > > > > > >> let's > > > > > > > >> >>>> > >> > > > > > >> > for > > > > > > > >> >>>> > >> > > > > > >> > >> > now > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > agree about the overall > > > > pattern. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Thanks, > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Andrii Biletskyi > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > On Fri, Jan 23, 2015 at > > 6:59 > > > > AM, > > > > > > Jay > > > > > > > >> Kreps > > > > > > > >> >>>> < > > > > > > > >> >>>> > >> > > > > > >> jay.kr...@gmail.com > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > wrote: > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > Hey Joe, > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > This is great. A few > > > > comments > > > > > on > > > > > > > >> KIP-4 > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 1. This is much needed > > > > > > > >> functionality, > > > > > > > >> >>>> but there > > > > > > > >> >>>> > >> > > are a > > > > > > > >> >>>> > >> > > > > lot > > > > > > > >> >>>> > >> > > > > > >> of > > > > > > > >> >>>> > >> > > > > > >> > >> the so > > > > > > > >> >>>> > >> > > > > > >> > >> > > > let's > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > really think these > > > protocols > > > > > > > >> through. We > > > > > > > >> >>>> really > > > > > > > >> >>>> > >> > > want > > > > > > > >> >>>> > >> > > > to > > > > > > > >> >>>> > >> > > > > > >> end up > > > > > > > >> >>>> > >> > > > > > >> > >> > with a > > > > > > > >> >>>> > >> > > > > > >> > >> > > > set > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > of well thought-out, > > > > > orthoganol > > > > > > > >> apis. > > > > > > > >> >>>> For this > > > > > > > >> >>>> > >> > > > reason I > > > > > > > >> >>>> > >> > > > > > >> think > > > > > > > >> >>>> > >> > > > > > >> > >> it is > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > really > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > important to think > > through > > > > the > > > > > > end > > > > > > > >> state > > > > > > > >> >>>> even > > > > > > > >> >>>> > >> if > > > > > > > >> >>>> > >> > > that > > > > > > > >> >>>> > >> > > > > > >> includes > > > > > > > >> >>>> > >> > > > > > >> > >> APIs > > > > > > > >> >>>> > >> > > > > > >> > >> > > we > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > won't implement in the > > > first > > > > > > > phase. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 2. Let's please please > > > > please > > > > > > wait > > > > > > > >> until > > > > > > > >> >>>> we > > > > > > > >> >>>> > >> have > > > > > > > >> >>>> > >> > > > > switched > > > > > > > >> >>>> > >> > > > > > >> the > > > > > > > >> >>>> > >> > > > > > >> > >> > server > > > > > > > >> >>>> > >> > > > > > >> > >> > > > over > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > to the new java protocol > > > > > > > >> definitions. If > > > > > > > >> >>>> we add > > > > > > > >> >>>> > >> > > > upteen > > > > > > > >> >>>> > >> > > > > > >> more ad > > > > > > > >> >>>> > >> > > > > > >> > >> hoc > > > > > > > >> >>>> > >> > > > > > >> > >> > > > scala > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > objects that is just > > > > > generating > > > > > > > more > > > > > > > >> >>>> work for > > > > > > > >> >>>> > >> the > > > > > > > >> >>>> > >> > > > > > >> conversion > > > > > > > >> >>>> > >> > > > > > >> > we > > > > > > > >> >>>> > >> > > > > > >> > >> > know > > > > > > > >> >>>> > >> > > > > > >> > >> > > we > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > have to do. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 3. This proposal > > > introduces > > > > a > > > > > > new > > > > > > > >> type of > > > > > > > >> >>>> > >> > optional > > > > > > > >> >>>> > >> > > > > > >> parameter. > > > > > > > >> >>>> > >> > > > > > >> > >> This > > > > > > > >> >>>> > >> > > > > > >> > >> > is > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > inconsistent with > > > everything > > > > > > else > > > > > > > >> in the > > > > > > > >> >>>> > >> protocol > > > > > > > >> >>>> > >> > > > where > > > > > > > >> >>>> > >> > > > > > we > > > > > > > >> >>>> > >> > > > > > >> use > > > > > > > >> >>>> > >> > > > > > >> > >> -1 > > > > > > > >> >>>> > >> > > > > > >> > >> > or > > > > > > > >> >>>> > >> > > > > > >> > >> > > > some > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > other marker value. You > > > > could > > > > > > > argue > > > > > > > >> >>>> either way > > > > > > > >> >>>> > >> > but > > > > > > > >> >>>> > >> > > > > let's > > > > > > > >> >>>> > >> > > > > > >> stick > > > > > > > >> >>>> > >> > > > > > >> > >> with > > > > > > > >> >>>> > >> > > > > > >> > >> > > > that > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > for consistency. For > > > clients > > > > > > that > > > > > > > >> >>>> implemented > > > > > > > >> >>>> > >> the > > > > > > > >> >>>> > >> > > > > > protocol > > > > > > > >> >>>> > >> > > > > > >> in > > > > > > > >> >>>> > >> > > > > > >> > a > > > > > > > >> >>>> > >> > > > > > >> > >> > > better > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > way > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > than our scala code > > these > > > > > basic > > > > > > > >> >>>> primitives are > > > > > > > >> >>>> > >> > hard > > > > > > > >> >>>> > >> > > > to > > > > > > > >> >>>> > >> > > > > > >> change. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 4. ClusterMetadata: This > > > > seems > > > > > > to > > > > > > > >> >>>> duplicate > > > > > > > >> >>>> > >> > > > > > >> > TopicMetadataRequest > > > > > > > >> >>>> > >> > > > > > >> > >> > > which > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > has > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > brokers, topics, and > > > > > > partitions. I > > > > > > > >> think > > > > > > > >> >>>> we > > > > > > > >> >>>> > >> > should > > > > > > > >> >>>> > >> > > > > rename > > > > > > > >> >>>> > >> > > > > > >> that > > > > > > > >> >>>> > >> > > > > > >> > >> > > request > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > ClusterMetadataRequest > > (or > > > > > just > > > > > > > >> >>>> > >> MetadataRequest) > > > > > > > >> >>>> > >> > > and > > > > > > > >> >>>> > >> > > > > > >> include > > > > > > > >> >>>> > >> > > > > > >> > >> the id > > > > > > > >> >>>> > >> > > > > > >> > >> > > of > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > the > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > controller. Or are there > > > > other > > > > > > > >> things we > > > > > > > >> >>>> could > > > > > > > >> >>>> > >> > add > > > > > > > >> >>>> > >> > > > > here? > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 5. We have a tendency to > > > try > > > > > to > > > > > > > >> make a > > > > > > > >> >>>> lot of > > > > > > > >> >>>> > >> > > > requests > > > > > > > >> >>>> > >> > > > > > that > > > > > > > >> >>>> > >> > > > > > >> > can > > > > > > > >> >>>> > >> > > > > > >> > >> > only > > > > > > > >> >>>> > >> > > > > > >> > >> > > go > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > to > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > particular nodes. This > > > adds > > > > a > > > > > > lot > > > > > > > of > > > > > > > >> >>>> burden for > > > > > > > >> >>>> > >> > > > client > > > > > > > >> >>>> > >> > > > > > >> > >> > > implementations > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > (it > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > sounds easy but each > > > > discovery > > > > > > can > > > > > > > >> fail > > > > > > > >> >>>> in many > > > > > > > >> >>>> > >> > > parts > > > > > > > >> >>>> > >> > > > > so > > > > > > > >> >>>> > >> > > > > > it > > > > > > > >> >>>> > >> > > > > > >> > >> ends up > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > being a > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > full state machine to do > > > > > > right). I > > > > > > > >> think > > > > > > > >> >>>> we > > > > > > > >> >>>> > >> > should > > > > > > > >> >>>> > >> > > > > > consider > > > > > > > >> >>>> > >> > > > > > >> > >> making > > > > > > > >> >>>> > >> > > > > > >> > >> > > > admin > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > commands and ideally as > > > many > > > > > of > > > > > > > the > > > > > > > >> >>>> other apis > > > > > > > >> >>>> > >> as > > > > > > > >> >>>> > >> > > > > > possible > > > > > > > >> >>>> > >> > > > > > >> > >> > available > > > > > > > >> >>>> > >> > > > > > >> > >> > > on > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > all > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > brokers and just > > redirect > > > to > > > > > the > > > > > > > >> >>>> controller on > > > > > > > >> >>>> > >> > the > > > > > > > >> >>>> > >> > > > > broker > > > > > > > >> >>>> > >> > > > > > >> > side. > > > > > > > >> >>>> > >> > > > > > >> > >> > > Perhaps > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > there would be a general > > > way > > > > > to > > > > > > > >> >>>> encapsulate > > > > > > > >> >>>> > >> this > > > > > > > >> >>>> > >> > > > > > re-routing > > > > > > > >> >>>> > >> > > > > > >> > >> > behavior. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 6. We should probably > > > > > normalize > > > > > > > the > > > > > > > >> key > > > > > > > >> >>>> value > > > > > > > >> >>>> > >> > pairs > > > > > > > >> >>>> > >> > > > > used > > > > > > > >> >>>> > >> > > > > > >> for > > > > > > > >> >>>> > >> > > > > > >> > >> > configs > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > rather > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > than embedding a new > > > > > formatting. > > > > > > > So > > > > > > > >> two > > > > > > > >> >>>> strings > > > > > > > >> >>>> > >> > > > rather > > > > > > > >> >>>> > >> > > > > > than > > > > > > > >> >>>> > >> > > > > > >> > one > > > > > > > >> >>>> > >> > > > > > >> > >> > with > > > > > > > >> >>>> > >> > > > > > >> > >> > > an > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > internal equals sign. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 7. Is the postcondition > > of > > > > > these > > > > > > > >> APIs > > > > > > > >> >>>> that the > > > > > > > >> >>>> > >> > > > command > > > > > > > >> >>>> > >> > > > > > has > > > > > > > >> >>>> > >> > > > > > >> > >> begun or > > > > > > > >> >>>> > >> > > > > > >> > >> > > > that > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > the command has been > > > > > completed? > > > > > > It > > > > > > > >> is a > > > > > > > >> >>>> lot > > > > > > > >> >>>> > >> more > > > > > > > >> >>>> > >> > > > usable > > > > > > > >> >>>> > >> > > > > > if > > > > > > > >> >>>> > >> > > > > > >> the > > > > > > > >> >>>> > >> > > > > > >> > >> > > command > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > has > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > been completed so you > > know > > > > > that > > > > > > if > > > > > > > >> you > > > > > > > >> >>>> create a > > > > > > > >> >>>> > >> > > topic > > > > > > > >> >>>> > >> > > > > and > > > > > > > >> >>>> > >> > > > > > >> then > > > > > > > >> >>>> > >> > > > > > >> > >> > > publish > > > > > > > >> >>>> > >> > > > > > >> > >> > > > to > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > it you won't get an > > > > exception > > > > > > > about > > > > > > > >> >>>> there being > > > > > > > >> >>>> > >> > no > > > > > > > >> >>>> > >> > > > such > > > > > > > >> >>>> > >> > > > > > >> topic. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 8. Describe topic and > > list > > > > > > topics > > > > > > > >> >>>> duplicate a > > > > > > > >> >>>> > >> lot > > > > > > > >> >>>> > >> > > of > > > > > > > >> >>>> > >> > > > > > stuff > > > > > > > >> >>>> > >> > > > > > >> in > > > > > > > >> >>>> > >> > > > > > >> > >> the > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > metadata > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > request. Is there a > > reason > > > > to > > > > > > give > > > > > > > >> back > > > > > > > >> >>>> topics > > > > > > > >> >>>> > >> > > marked > > > > > > > >> >>>> > >> > > > > for > > > > > > > >> >>>> > >> > > > > > >> > >> > deletion? I > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > feel > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > like if we just make the > > > > > > > >> post-condition > > > > > > > >> >>>> of the > > > > > > > >> >>>> > >> > > delete > > > > > > > >> >>>> > >> > > > > > >> command > > > > > > > >> >>>> > >> > > > > > >> > be > > > > > > > >> >>>> > >> > > > > > >> > >> > that > > > > > > > >> >>>> > >> > > > > > >> > >> > > > the > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > topic is deleted that > > will > > > > get > > > > > > rid > > > > > > > >> of > > > > > > > >> >>>> the need > > > > > > > >> >>>> > >> > for > > > > > > > >> >>>> > >> > > > this > > > > > > > >> >>>> > >> > > > > > >> right? > > > > > > > >> >>>> > >> > > > > > >> > >> And > > > > > > > >> >>>> > >> > > > > > >> > >> > it > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > will > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > be much more intuitive. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 9. Should we consider > > > > batching > > > > > > > these > > > > > > > >> >>>> requests? > > > > > > > >> >>>> > >> We > > > > > > > >> >>>> > >> > > > have > > > > > > > >> >>>> > >> > > > > > >> > generally > > > > > > > >> >>>> > >> > > > > > >> > >> > > tried > > > > > > > >> >>>> > >> > > > > > >> > >> > > > to > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > allow multiple > > operations > > > to > > > > > be > > > > > > > >> batched. > > > > > > > >> >>>> My > > > > > > > >> >>>> > >> > > suspicion > > > > > > > >> >>>> > >> > > > > is > > > > > > > >> >>>> > >> > > > > > >> that > > > > > > > >> >>>> > >> > > > > > >> > >> > without > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > this > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > we will get a lot of > > code > > > > that > > > > > > > does > > > > > > > >> >>>> something > > > > > > > >> >>>> > >> > like > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > for(topic: > > > > > > > >> adminClient.listTopics()) > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> adminClient.describeTopic(topic) > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > this code will work > > great > > > > when > > > > > > you > > > > > > > >> test > > > > > > > >> >>>> on 5 > > > > > > > >> >>>> > >> > topics > > > > > > > >> >>>> > >> > > > but > > > > > > > >> >>>> > >> > > > > > >> not do > > > > > > > >> >>>> > >> > > > > > >> > >> as > > > > > > > >> >>>> > >> > > > > > >> > >> > > well > > > > > > > >> >>>> > >> > > > > > >> > >> > > > if > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > you have 50k. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 10. I think we should > > also > > > > > > discuss > > > > > > > >> how > > > > > > > >> >>>> we want > > > > > > > >> >>>> > >> to > > > > > > > >> >>>> > >> > > > > expose > > > > > > > >> >>>> > >> > > > > > a > > > > > > > >> >>>> > >> > > > > > >> > >> > > programmatic > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > JVM > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > client api for these > > > > > operations. > > > > > > > >> >>>> Currently > > > > > > > >> >>>> > >> people > > > > > > > >> >>>> > >> > > > rely > > > > > > > >> >>>> > >> > > > > on > > > > > > > >> >>>> > >> > > > > > >> > >> > AdminUtils > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > which > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > is totally sketchy. I > > > think > > > > we > > > > > > > >> probably > > > > > > > >> >>>> need > > > > > > > >> >>>> > >> > > another > > > > > > > >> >>>> > >> > > > > > client > > > > > > > >> >>>> > >> > > > > > >> > >> under > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > clients/ > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > that exposes > > > administrative > > > > > > > >> >>>> functionality. We > > > > > > > >> >>>> > >> > will > > > > > > > >> >>>> > >> > > > need > > > > > > > >> >>>> > >> > > > > > >> this > > > > > > > >> >>>> > >> > > > > > >> > >> just > > > > > > > >> >>>> > >> > > > > > >> > >> > to > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > properly test the new > > > apis, > > > > I > > > > > > > >> suspect. We > > > > > > > >> >>>> > >> should > > > > > > > >> >>>> > >> > > > figure > > > > > > > >> >>>> > >> > > > > > out > > > > > > > >> >>>> > >> > > > > > >> > that > > > > > > > >> >>>> > >> > > > > > >> > >> > API. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 11. The other > > information > > > > that > > > > > > > >> would be > > > > > > > >> >>>> really > > > > > > > >> >>>> > >> > > useful > > > > > > > >> >>>> > >> > > > > to > > > > > > > >> >>>> > >> > > > > > >> get > > > > > > > >> >>>> > >> > > > > > >> > >> would > > > > > > > >> >>>> > >> > > > > > >> > >> > be > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > information about > > > > > > partitions--how > > > > > > > >> much > > > > > > > >> >>>> data is > > > > > > > >> >>>> > >> in > > > > > > > >> >>>> > >> > > the > > > > > > > >> >>>> > >> > > > > > >> > partition, > > > > > > > >> >>>> > >> > > > > > >> > >> > what > > > > > > > >> >>>> > >> > > > > > >> > >> > > > are > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > the segment offsets, > > what > > > is > > > > > the > > > > > > > >> log-end > > > > > > > >> >>>> offset > > > > > > > >> >>>> > >> > > (i.e. > > > > > > > >> >>>> > >> > > > > > last > > > > > > > >> >>>> > >> > > > > > >> > >> offset), > > > > > > > >> >>>> > >> > > > > > >> > >> > > > what > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > is > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > the compaction point, > > > etc. I > > > > > > think > > > > > > > >> that > > > > > > > >> >>>> done > > > > > > > >> >>>> > >> > right > > > > > > > >> >>>> > >> > > > this > > > > > > > >> >>>> > >> > > > > > >> would > > > > > > > >> >>>> > >> > > > > > >> > be > > > > > > > >> >>>> > >> > > > > > >> > >> > the > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > successor to the very > > > > awkward > > > > > > > >> >>>> OffsetRequest we > > > > > > > >> >>>> > >> > have > > > > > > > >> >>>> > >> > > > > > today. > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > -Jay > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > On Wed, Jan 21, 2015 at > > > > 10:27 > > > > > > PM, > > > > > > > >> Joe > > > > > > > >> >>>> Stein < > > > > > > > >> >>>> > >> > > > > > >> > >> joe.st...@stealth.ly> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > wrote: > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > Hi, created a KIP > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > >> >>>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > JIRA > > > > > > > >> >>>> > >> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1694 > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> /******************************************* > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > Joe Stein > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > Founder, Principal > > > > > Consultant > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > Big Data Open Source > > > > > Security > > > > > > > LLC > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > http://www.stealth.ly > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > Twitter: > > > > @allthingshadoop < > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > http://www.twitter.com/allthingshadoop > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> ********************************************/ > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> -- > > > > > > > >> >>>> > >> > > > > > >> > >> -- Guozhang > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > -- > > > > > > > >> >>>> > >> > Jeff Holoman > > > > > > > >> >>>> > >> > Systems Engineer > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > >> >>>> > > > > > > > >> >>>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> -- > > > > > > > >> >>> -- Guozhang > > > > > > > >> >>> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> -- > > > > > > > >> >> -- Guozhang > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
