Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-17 Thread David Jacot
Hey Jose, Yes, we have considered it. Check "Put clientName and clientVersion in the RequestHeader" in the rejected alternatives. Best, David On Tue, Sep 17, 2019 at 12:57 AM Jose Armando Garcia Sancio < jsan...@confluent.io> wrote: > On Mon, Sep 9, 2019 at 4:00 PM Colin McCabe wrote: > > > >

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-16 Thread Jose Armando Garcia Sancio
On Mon, Sep 9, 2019 at 4:00 PM Colin McCabe wrote: > > One solution to consider is adding the clientVersion and clientType to the > request header as optional (tagged) fields. This would let us skip the > extra round trip. I don't think it's that much more messy than having a > separate request

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-16 Thread David Jacot
Hi all, I have updated the KIP to reflect the offline discussion that we have had. It seems that we have finally reached a consensus on the approach. Therefore, I will start a vote. Best, David On Wed, Sep 11, 2019 at 3:53 PM David Jacot wrote: > Hi all, > > I have discussed with Magnus about

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-11 Thread David Jacot
Hi all, I have discussed with Magnus about the various options to get his view from a librdkafka perspective and he has suggested a good alternative. It seems we could continue with the idea to use the ApiVersionsRequest/Response but we a different failing back strategy. When a broker get an unkn

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-09 Thread Colin McCabe
Hi all, I agree that freezing the request header is not very appealing. We might want to add something there later. Having a separate request type is very clean, but as David mentioned, it does add an extra request/response cycle to the time required to get a connection into a usable state.

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-09 Thread David Jacot
Hi Gwen, The reasoning behind having the information before the Sasl authentication was to have the information for troubleshooting purposes. For instance, when there are errors in the handshake, it would be great to know if it comes from a specific buggy clients. I think we could live without tho

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-09 Thread David Jacot
Hey, I have made a couple ot scenarios to better understand the situation, the options and their tradeoffs. Please, find (raw) my notes below. Assumptions - - AK 2.4 is released with RequestHeader v1 (KIP-482) and ApiVersionsRequest v3 (KIP-511); - Broker fails back to the latest

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-08 Thread Gwen Shapira
Hey, Since modifying ApiVersionsRequest seems to be quite involved, do we want to re-examine the rejected option of adding another request/response pair? It will add another roundtrip, but Kafka already expects client connections to be long-lived, so the overhead is probably negligible. In additi

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-06 Thread Colin McCabe
Hi David, Yeah, this is kind of difficult. >From the high level, we either need forward compatibility (old brokers able to >read new ApiVersionsRequests) or some kind of challenge/response system. I >think you're on the right track to be thinking about forward compatibility... >a challenge/r

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-03 Thread David Jacot
Hi Colin, Thanks for your input. Please, find my comments below: >> Currently, we don't parse the contents of ApiVersionsRequest at all, since it's an empty message. KIP-511 proposes adding some fields here, which will clearly change that situation. In the future, any changes to ApiVersionsRequ

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-03 Thread Colin McCabe
Hi David, Thanks again for the KIP. Currently, we don't parse the contents of ApiVersionsRequest at all, since it's an empty message. KIP-511 proposes adding some fields here, which will clearly change that situation. In the future, any changes to ApiVersionsRequest will have to only add stu

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-03 Thread David Jacot
Hi all, I have updated the KIP to address the various comments. I have also added a section about the handling of the ApiVersionsRequest/Response in the broker. Please, let me know what you think. I would like to make it for the next release if possible. Best, David On Fri, Aug 30, 2019 at 10:3

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-08-30 Thread David Jacot
Hi Magnus, Thank you for your feedback. Please, find my comments below. 1. I thought that the clientId was meant for this purpose (providing information about the application). Is there a gap I don't see? 2. I have put two fields to avoid requiring deep validation and parsing on the broker. I be

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-08-29 Thread Colin McCabe
On Fri, Aug 23, 2019, at 00:07, Magnus Edenhill wrote: > Great proposal, this feature is well overdue! > > 1) > From an operator's perspective I don't think the kafka client > implementation name and version are sufficient, > I also believe the application name and version are of interest. > You c

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-08-23 Thread Magnus Edenhill
Great proposal, this feature is well overdue! 1) >From an operator's perspective I don't think the kafka client implementation name and version are sufficient, I also believe the application name and version are of interest. You could have all applications in your cluster run the same kafka client

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-08-22 Thread David Jacot
Hi Satish, Thank you for your feedback! Please find my answers below. >> Did you consider taking version property by loading “kafka/kafka-version.properties” as a resource while java client is initialized? “kafka/kafka-version.properties” is shipped with kafka-clients jar. I wasn't aware of th

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-08-21 Thread Satish Duggana
Hi David, Thanks for the KIP. I have a couple of questions. >> For the Java client, the idea is to define two constants in the code to >> store its name and its version. If possible, the version will be set >> automatically based on metadata coming from gradle (or the repo itself) to >> avoid h

[DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-08-21 Thread David Jacot
Hi all, I would like to start a discussion for KIP-511: https://cwiki.apache.org/confluence/display/KAFKA/KIP-511%3A+Collect+and+Expose+Client%27s+Name+and+Version+in+the+Brokers Let me know what you think. Best, David