Re: [DISCUSS] KIP-35 - Retrieve protocol version

Done. Thanks! On Thu, Apr 21, 2016 at 4:32 PM, Gwen Shapira wrote: > Lets start a vote immediately? We are short of time toward the release. > > On Thu, Apr 21, 2016 at 2:57 PM, Ashish Singh wrote: > > Hey Guys, > > > > KIP-35 > > < >

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Lets start a vote immediately? We are short of time toward the release. On Thu, Apr 21, 2016 at 2:57 PM, Ashish Singh wrote: > Hey Guys, > > KIP-35 > > has been updated based on latest

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hey Guys, KIP-35 has been updated based on latest discussions and following PRs have also been updated. 1. KAFKA-3307: Add ApiVersion request/response and server side handling.

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hi, Ismael, The SASL engine that we used is the SASL library, right? How did the C client generate those SASL tokens? Once a SASL mechanism is chosen, the subsequent tokens are determined, right? So, my feeling is that those tokens are part of SaslHandshakeRequest and are just extended across

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hi Rajini, Yes, I agree that it's not ideal. However, doing this once at the broker is more manageable than pushing the additional complexity to the clients. Between the two options you outlined, the second one seemed the least bad (we do something similar for controlled shutdown because it was

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Ismael, My only concern about wrapping SASL tokens in Kafka headers is backward compatibility. We would either have a different format for GSSAPI alone to match 0.9.0.x or we would need to support two different wire protocols for GSSAPI. Neither sounds ideal. On Tue, Apr 12, 2016 at 9:18 AM,

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hi Jun, I understand the point about the SASL tokens being similar to the SSL handshake in a way. However, is there any SASL library that handles the network communication for these tokens? I couldn't find any and without that, there isn't much benefit in deviating from Kafka's protocol (we

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Jun's tweaked proposal sounds good to me. In terms of completing KIP-43, this changes the format of the request-response for exchanging mechanisms, but not the overall logic. Since the request format in KIP-43 is worth changing anyway, I will update the KIP and the PR. On Tue, Apr 12, 2016 at

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Magnus, That sounds reasonable. To reduce the changes on the server side, I'd suggest the following minor tweaks on the proposal. 1. Continue supporting the separate SASL and SASL_SSL port. On SASL port, we support the new sequence ApiVersionRequest (optional), SaslHandshakeRequest, SASL

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hey Jun, see inline 2016-04-11 19:19 GMT+02:00 Jun Rao : > Hi, Magnus, > > Let me understand your proposal in more details just from the client's > perspective. My understanding of your proposal is the following. > > On plaintext port, the client will send the following bytes

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hi, Magnus, Let me understand your proposal in more details just from the client's perspective. My understanding of your proposal is the following. On plaintext port, the client will send the following bytes in order. ApiVersionRequest, SaslHandshakeRequest, SASL tokens (if SASL is enabled),

Re: [DISCUSS] KIP-35 - Retrieve protocol version

2016-04-11 3:01 GMT+02:00 Jun Rao : > Thinking about ApiVersionRequest a bit more. There are quite a few things > special about it. In the ideal case, (1) its version should never change; > The only thing we know of the future is that we dont know anything, we can't think of

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hi Jun, the "send empty response instead of disconnect on unsupported protocol" was actually a sub-proposal and was never meant as an alternative to the proper feature detection proposed by KIP-35. I've added it back to the list of rejected alternatives on the wiki page, thanks for pointing

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Thinking about ApiVersionRequest a bit more. There are quite a few things special about it. In the ideal case, (1) its version should never change; (2) it needs to be done before authentication (either SSL/SASL); (3) it is required to be issued at the beginning of each connection but never needs

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Magnus, A while back, we had another proposal for the broker to just send the correlation id and an empty payload if it receives an unsupported version of the request. I didn't see that in the rejected section. It seems simpler than the current proposal where the client has to issue an

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Fri, Apr 1, 2016 at 1:32 AM, Ismael Juma wrote: > Two more things: > > 3. We talk about backporting of new request versions to stable branches in > the KIP. In practice, we can't do that until the Java client is changed so > that it doesn't blindly use the latest protocol

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Ismael, thanks for the review. On Fri, Apr 1, 2016 at 1:22 AM, Ismael Juma wrote: > A couple of questions: > > 1. The KIP says "Specific version may be deprecated through protocol > documentation but must still be supported (although it is fair to return an > error code if

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hey, people have had concerns about the complexity of the client of mapping API versions to features, so I implemented this in librdkafka and it is rather straight forward. See here: https://github.com/edenhill/librdkafka/blob/KIP-35/src/rdkafka_feature.c#L52 Consider it a proof of concept at

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Ashish, thanks, didnt know that. For ApiVersionRequest requesting no Apis to be returned doesnt make sense so the distinction isn't necessary, but I'm fine with adding Null to be more in line with future protocol requests, as long as it doesn't delay this KIP any longer! :) 2016-04-05 19:38

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Magnus, it is proposed to be changed in version 1, https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-MetadataSchema . On Tue, Apr 5, 2016 at 10:23 AM, Magnus Edenhill

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Empty arrays are already used in MetadataRequest to retrieve all topics in the cluster, ApiVersion request will have the same standard semantics. 2016-04-05 19:01 GMT+02:00 Ewen Cheslack-Postava : > Also, just a thought but is empty list the sentinel we really want to >

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Sounds fair. I am OK with putting down, permanent support of ApiVersion api versions, as a limitation in KIP. On Tue, Apr 5, 2016 at 9:23 AM, Ewen Cheslack-Postava wrote: > On Mon, Apr 4, 2016 at 11:24 AM, Gwen Shapira wrote: > > > > > > > > > >In case

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Null array sounds good to me as well. On Tue, Apr 5, 2016 at 10:06 AM, Ismael Juma wrote: > Yeah, we should use nullable arrays (which have been introduced in > KIP-4-Metadata) instead of empty list to indicate all versions. > > Ismael > On 5 Apr 2016 18:01, "Ewen

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Yeah, we should use nullable arrays (which have been introduced in KIP-4-Metadata) instead of empty list to indicate all versions. Ismael On 5 Apr 2016 18:01, "Ewen Cheslack-Postava" wrote: > Also, just a thought but is empty list the sentinel we really want to > indicate we

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Also, just a thought but is empty list the sentinel we really want to indicate we want all API versions? We've got nullable string and nullable bytes in the protocol. Should we have nullable array support as well and use that to indicate we want everything? I can't think of a use case for sending

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Mon, Apr 4, 2016 at 11:24 AM, Gwen Shapira wrote: > > > > > >In case of connection closures, the KIP recommends that clients should > >use some other method of determining the apiRequest version to use, > > like, > >probing. For instance, client will send V0

Re: [DISCUSS] KIP-35 - Retrieve protocol version

> > >In case of connection closures, the KIP recommends that clients should >use some other method of determining the apiRequest version to use, > like, >probing. For instance, client will send V0 version of apiVersion request >and will try higher versions incrementally. In case b,

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Sorry for jumping over some feedback, but not sure the best way to provide this feedback. Somewhat indirectly targeting feedback from Jay: > "I actually do think good is the enemy of perfect (not sure if that makes sense but I think you get what I mean)." Yes I get it, but I don't agree (so

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Gwen, I do not think it will break clients. Just to be clear we have following options. 1. Versioned apiVersion request and response. KIP-35 aware clients will send apiRequest, version Vn, to get api versions from a broker. If a broker supports Vn of apiVersion request, it will

Re: [DISCUSS] KIP-35 - Retrieve protocol version

It isn't a guarantee, it is a limitation. It doesn't even matter what we do (have versions, no versions, deprecate or not) - clients will never be able to use version other than v0 (how will they know it is there?) and if we ever remove v0, we will immediately break all clients. If we are ok

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Never ever changing the version of VersionReq/resp will be ideal, but it is hard to guarantee that we will never need a change. We should definitely try to avoid it and honestly I do not see a reason, as of now, why it will change. I would say, we should keep it versioned and document any risk of

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I'm sympathetic to Jay's argument, but I feel the version API can be justified on its own without requiring a big change in the way we manage compatibility. For the official client compatibility story (i.e. what the Java client implements), we can use this API to verify compatibility with the

Re: [DISCUSS] KIP-35 - Retrieve protocol version

My understanding is that we are not adding the client portion to the KIP is because we believe it will require a lot of discussion (read: will be really hard to get right). Maybe it is a hint that this protocol is too difficult for clients to implement? I can't see why is it easy in C and Python

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hey Ewen, with protocol design I actually do think good is the enemy of perfect (not sure if that makes sense but I think you get what I mean). Your comment seemed to be along the lines of "this isn't very good, but let's do something". Can you elaborate on what you think we should be doing? The

Re: [DISCUSS] KIP-35 - Retrieve protocol version

That is a fair concern and I think eventually we might want to have java clients backwards compatible. However, blocking KIP-35 on that might not be the best idea. The reason I say so is due to following reasons. 1. Backwards compatibility in java clients is a larger discussion, as we have

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira wrote: > I have two concerns with the proposal as it is: > > 1. Having an API that publishes protocol is useless for clients if > developers don't bump the API when they should. > I would like to see good documentation on when

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Two more things: 3. We talk about backporting of new request versions to stable branches in the KIP. In practice, we can't do that until the Java client is changed so that it doesn't blindly use the latest protocol version. Otherwise, if new request versions were added to 0.9.0.2, the client

Re: [DISCUSS] KIP-35 - Retrieve protocol version

A couple of questions: 1. The KIP says "Specific version may be deprecated through protocol documentation but must still be supported (although it is fair to return an error code if the specific API supports it).". It may be worth expanding this a little more. For example, what does it mean to

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I agree in a "perfect is the enemy of good" sense. I don't think what's expressed in the API is necessarily the *ideal* thing for client developers, which is who I think this API should be optimizing for. That said, this is a practical solution to the problem and while finer-grained than I think

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I agree with Jason. The blocking factor has been how to use the proposed changes in java client without making it super complicated. With Jason's suggestion we can get past this blocker, while keeping the core of the proposal intact. On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Bumping this thread. I talked with Ashish and Magnus about this KIP offline and I'm gradually coming over. The new API actually stands by itself outside of the discussion about whether the client should support backwards compatibility or not. For the Java client, we could continue to support the

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I like it. I think the revised KIP-35 approach is simple, easy to understand, and workable. I think it will solve the main problem that I have had, and I assume other 3rd party client devs have had as well, and should open the door to more backwards-compatible drivers and less fracturing of client

Re: [DISCUSS] KIP-35 - Retrieve protocol version

We have proposed and discussed majorly three approaches so far, there were many minor versions with small variations. Comparing them really requires a side by side proposal and their pros/cons, and I agree with others that this has been lacking in the KIP. We just updated the KIP with following

Re: [DISCUSS] KIP-35 - Retrieve protocol version

"I think I would make this approach work by looking at the released server version documentation for each version that I am trying to support and test against*, manually identify the expected "protocol vectors" each supports, store that as a map of vectors to "broker versions", check each vector

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira wrote: > "I think I would make this approach work by looking at the released server > version documentation for each version that I am trying to support and test > against*, manually identify the expected "protocol vectors" each

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Jay, I think you are correct. Once a client gets the metadata from a broker, it does not really need to get brokers and topics info from other brokers again. However, with the current suggestion the client will still have to send metadata request to get broker's supported protocols. Client can

Re: [DISCUSS] KIP-35 - Retrieve protocol version

My problem with per-api is that not all combinations are supported. So the client will need to figure out which "safe" combination to revert to - which is a lot like global numbers, but less linear... On Thu, Mar 17, 2016 at 11:29 AM, Joel Koshy wrote: > @Gwen - yes it did

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I dont really see how first checking if the broker supports a certain set of API versions is so much more complex than calling the same set of API versions during operation? The only difference is that without the check we might only make it halfway through, i.e., being able to join a group,

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I was a bit surprised to discover that the latest KIP includes an array of versions per request. Especially since we spent an hour discussing global APIs and their tradeoffs. I'm pointing this out in case anyone else missed that - to make sure we are all on the same page in this discussion. Gwen

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Yeah, Magnus, I'm not arguing for either approach (1) or approach (2), I'm saying the only reason it would make sense to combine the version check and the metadata request would be if the version check was cluster wide like the other metadata. That was where that idea originated and if you make

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I'm a fan of separating feature support/versions into a separate request from metadata -- they're different requests and I don't see how the current model of metadata can properly map to the request version info we need. The complexity of support differing across brokers seems like something

Re: [DISCUSS] KIP-35 - Retrieve protocol version

> > > Anyway, I'm not particularly strong in favor of either approach, but I > think what's concerning Gwen (and myself) is that we're going down to the > wire with a significant decision and the KIP itself seems different every > time we look at it. And still a little incomplete. It would be

Re: [DISCUSS] KIP-35 - Retrieve protocol version

@Gwen - yes it did come back to per-API versions - I think that happened toward the end of the KIP hangout. I already said this, but personally I'm also in favor of per-API versions. A global/aggregate API version also works but that is just a proxy to a vector of per-API versions and at the end

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Thu, Mar 17, 2016 at 2:07 PM, Jason Gustafson wrote: > It would also be nice to discuss the way the current client > versioning works and why it is inadequate for third-party clients. > My understanding of the java client versioning is that it is not backwards-compatible.

Re: [DISCUSS] KIP-35 - Retrieve protocol version

My point was only that the way people would probably end up using the request for all versions as a proxy for determining the broker version. For example, if the JoinGroup v0 is present, then it must be 0.9. Or if you do it in terms of features, then you need to check 5 or 6 different request

Re: [DISCUSS] KIP-35 - Retrieve protocol version

1. But there is no way of conveying version mismatch in the protocol, that's one of the reasons for this KIP in the first place :) Even if there was (e.g., the empty response hack) it makes the client implementation more complex since the cached cluster-level version support returned by the broker

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Sent incomplete message by mistake. I do not have a preference for (1) or (2). (1) basically makes us view supported versions for all brokers as a part of cluster wide state, however (2) lets client deal with individual brokers. From usage perspective, clients would need to choose a certain

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I do not have a preference for 1 or 2. 1 basically makes us view supported versions for all brokers On Tuesday, March 15, 2016, Jay Kreps wrote: > Yeah I think there are two possible approaches: > 1. You get the versions in the metadata request and cache them and > invalidate

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Yeah I think there are two possible approaches: 1. You get the versions in the metadata request and cache them and invalidate that cache if you get a version mismatch error (basically as we do with leadership information). 2. You check each connection I think combining metadata request and

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hey Jay, as discussed earlier it is not safe to cache/relay a broker's version or its supported API versions, by the time the client connects the broker might have upgraded to another version which effectively makes this information useless in a cached form. The complexity of querying for

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hello Jay, On Tue, Mar 15, 2016 at 3:35 PM, Jay Kreps wrote: > Hey Ashish, > > Can you expand in the proposal on how this would be used by clients? > Will add some info along this line. This proposal only has one slot for api versions, though in fact there > is potentially

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hey Ashish, Can you expand in the proposal on how this would be used by clients? This proposal only has one slot for api versions, though in fact there is potentially a different version on each broker. I think the proposal is that every single time the client establishes a connection it would

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Magnus and I had a brief discussion following the KIP call. KIP-35 wiki has been updated accordingly. Please review the KIP and vote on the corresponding vote thread. On Mon, Mar 14, 2016 at 11:27 PM, Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I think there is a bit of misunderstanding going on here regarding protocol documentation and its versioning. It could be that I am the one who misunderstood it, please correct me if so. Taking Gwen's example. 1. 0.10.0 (protocol v4) is released with current KIP-35 2. On trunk, modify produce

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Yeah, Gwen's example is a good one. And it doesn't even have to be thought of in terms of the implementation -- you can think of the protocol itself as effectively being possible to branch and have changes cherry-picked. Given the way some changes interact and that only some may be feasible to

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I have been following this thread with some interest over the past couple of days especially since we run off trunk for the most part. I have been on the fence on the approach of a global API version (which we did come up in the original KIP discussion). I think what everyone felt at the time was

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Again I think that implies a world view where the code is the source of truth. What I'm proposing, though is that there are only protocol versions 5 and version 6 as defined by the protocol spec and you either support 6 or you don't, supporting part of 6 is just being an incorrect implementation

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Jay, Ewen had a good example: 1. 0.10.0 (protocol v4) is released with current KIP-35 2. On trunk, modify produce requests and bump to v5 3. On trunk, we modify metadata requests and bump to v6 4. Now we decide that the metadata change fixes a super critical issue and want to backport the

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hey Dana, I am actually thinking about it differently. Basically I think you are imagining a world in which the Kafka code is the source of truth, and the Kafka developers make random changes that inflict pain on you at will. The protocol documentation is basically just some semi-accurate

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Thats a great point, Dana. Thanks for bringing this up. Ewen raised the same concern and suggested something called "feature detection" where the broker would advertise support for specific features that clients may need and by that avoid a linear model (it also allows for advertising features

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Is a linear protocol int consistent with the current release model? It seems like that would break down w/ the multiple release branches that are all simultaneously maintained? Or is it implicit that no patch release can ever bump the protocol int? Or maybe the protocol int gets some extra

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Yeah I think that is the point--we have a proposal for a new protocol versioning scheme and a vote on it but it doesn't actually describe how versioning will work yet! I gave my vague impression based on this thread, but I want to make sure that is correct and get it written down before we adopt

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps wrote: > Couple of missing things: > > This KIP doesn't have a proposal on versioning it just gives different > options, it'd be good to get a concrete proposal in the KIP. Here is my > understanding of what we are proposing (can

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Mon, Mar 14, 2016 at 4:51 PM, Jason Gustafson wrote: > > > > Are you suggesting this as a solution for the problem where a KIP-35 > aware > > client sends a higher version of metadata req, say v2, to a KIP-35 aware > > broker that only supports up to v1. > > > Yes, that's

Re: [DISCUSS] KIP-35 - Retrieve protocol version

> > Are you suggesting this as a solution for the problem where a KIP-35 aware > client sends a higher version of metadata req, say v2, to a KIP-35 aware > broker that only supports up to v1. Yes, that's right. In that case, the client first sends v1, finds out that the broker supports v2, and

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hi Jason, On Mon, Mar 14, 2016 at 4:04 PM, Jason Gustafson wrote: > Perhaps clients should always send the oldest version of the metadata > request which supports KIP-35 when initially connecting to the cluster. > Depending on the versions in the response, it can upgrade to

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Perhaps clients should always send the oldest version of the metadata request which supports KIP-35 when initially connecting to the cluster. Depending on the versions in the response, it can upgrade to a more recent version. Then maybe we don't need the empty response hack? One thing that's not

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I have updated the KIP based on the discussions so far. Will initiate a VOTE thread soon. There are some minor details that are still under discussion, but nothing major to stop us from voting. On Mon, Mar 14, 2016 at 3:39 PM, Ashish Singh wrote: > > > On Mon, Mar 14, 2016

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Mon, Mar 14, 2016 at 3:37 PM, Ismael Juma wrote: > On Mon, Mar 14, 2016 at 10:35 PM, Ashish Singh > wrote: > > > > Also, I think it's a bit odd to say a `single null topic with size -1`. > > Do > > > we mean an array of topics with size -1 and no

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Mon, Mar 14, 2016 at 10:35 PM, Ashish Singh wrote: > > Also, I think it's a bit odd to say a `single null topic with size -1`. > Do > > we mean an array of topics with size -1 and no elements? That would imply > > introducing a NULLABLE_ARRAY type (we currently have

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma wrote: > On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira wrote: > > > > > I don't see how it helps. If the client is communicating with a broker > > that > > > does not support KIP-35, that broker will simply close

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Agree. Regardless of which one of us understood the KIP correctly, this kind of thing should be crystal clear in the KIP. On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma wrote: > On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira wrote: > > > > > I don't see how

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira wrote: > > > I don't see how it helps. If the client is communicating with a broker > that > > does not support KIP-35, that broker will simply close the connection. If > > the broker supports KIP-35, then it will provide the broker

Re: [DISCUSS] KIP-35 - Retrieve protocol version

> > > I don't see how it helps. If the client is communicating with a broker that > does not support KIP-35, that broker will simply close the connection. If > the broker supports KIP-35, then it will provide the broker version. I > don't envisage a scenario where a broker does not support KIP-35,

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hi Ashish, Comments inline. On Mon, Mar 14, 2016 at 6:15 PM, Ashish Singh wrote: > I think ApiVersion aims to identify protocol version changes, more than > release change, > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L30 >

Re: [DISCUSS] KIP-35 - Retrieve protocol version

There seems to be a lot of tension between support for release-deploys vs. trunk-deploys. Is there a better way to handle this? In my experience the vast majority of third-party client code is written managing compatibility on the first case (release-deploys). I would prefer a simple approach

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Mon, Mar 14, 2016 at 9:37 AM, Gwen Shapira wrote: > On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma wrote: > > Hi Ashish, > > > > A few comments below. > > > > On Fri, Mar 11, 2016 at 9:59 PM, Ashish Singh > wrote: > > > >> Sounds like

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hello Gwen/ Ismael, On Mon, Mar 14, 2016 at 9:55 AM, Ismael Juma wrote: > Hi Gwen, > > On Mon, Mar 14, 2016 at 4:37 PM, Gwen Shapira wrote: > > > On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma wrote:> > > >>1. Every time a protocol

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hi Gwen, On Mon, Mar 14, 2016 at 4:37 PM, Gwen Shapira wrote: > On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma wrote:> > >>1. Every time a protocol version changes, for any request/response, > >>broker version, ApiVersion, will be bumped up. > >> > >

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma wrote: > Hi Ashish, > > A few comments below. > > On Fri, Mar 11, 2016 at 9:59 PM, Ashish Singh wrote: > >> Sounds like we are mostly in agreement. Following are the key points. >> >>1. Every time a protocol

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hi Ashish, A few comments below. On Fri, Mar 11, 2016 at 9:59 PM, Ashish Singh wrote: > Sounds like we are mostly in agreement. Following are the key points. > >1. Every time a protocol version changes, for any request/response, >broker version, ApiVersion, will be

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Good summary! Some comments inline: 2016-03-11 22:59 GMT+01:00 Ashish Singh : > Sounds like we are mostly in agreement. Following are the key points. > >1. Every time a protocol version changes, for any request/response, >broker version, ApiVersion, will be bumped

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Yeah I agree with that characterization of the tradeoff. I think what that would imply would be that evolution of the metadata request (or the protocol version request) would remain server-first, whereas other apis would be independent. Not sure if I've fully thought it through, though. -Jay On

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I'm not sure supporting specific interim versions between releases are really that big of a concern, for a start the protocol changes may be in flux and not settle until the formal release, secondly the 3rd party clients typically lag behind at least until the formal release before they implement

Re: [DISCUSS] KIP-35 - Retrieve protocol version

We introduced a way to bump the API version in between releases as part of the KIP-31/KIP-32 by the way. Extending that could maybe work. Take a look at the ApiVersion class and its documentation. Ismael On 11 Mar 2016 19:06, "Gwen Shapira" wrote: > Magnus, > > If we go with

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Magnus, If we go with release version as protocol version (which I agree is much more user-friendly) - what will be the release version on trunk? 0.10.0-SNAPSHOT? How will clients handle the fact that some 0.10.0-SNAPSHOT will have different protocol than others (because we modify the protocol

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hey Jay, some thoughts in-line: 2016-03-04 18:52 GMT+01:00 Jay Kreps : > Yeah here is my summary of my take: > > 1. Negotiating a per-connection protocol actually does add a lot of > complexity to clients (many more failure states to get right). > I'm not sure about this. A

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hi all, sorry for joining late in the game, the carribean got in the way. My thoughts: There is no way around the chicken problem, so the sooner we can add protocol versioning functionality the better and we'll add heuristics in clients to handle the migration period (e.g, what Dana has done in

Re: [DISCUSS] KIP-35 - Retrieve protocol version

@Magnus, Does the latest suggestion sound OK to you. I am planning to update PR based on latest suggestion. On Mon, Mar 7, 2016 at 10:58 AM, Ashish Singh wrote: > > > On Fri, Mar 4, 2016 at 5:46 PM, Jay Kreps wrote: > >> Hey Ashish, >> >> Both good

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Fri, Mar 4, 2016 at 5:46 PM, Jay Kreps wrote: > Hey Ashish, > > Both good points. > > I think the issue with the general metadata request is the same as the > issue with a version-specific metadata request from the other > proposal--basically it's a chicken and egg problem,

  1   2   >