It isn't a guarantee, it is a limitation. It doesn't even matter what we do (have versions, no versions, deprecate or not) - clients will never be able to use version other than v0 (how will they know it is there?) and if we ever remove v0, we will immediately break all clients.
If we are ok making changes that immediately break all clients, we don't need this KIP. I thought the entire point was to fix this. So, by having this KIP we agree to never ever remove VersionRequest v0 (as it is in the KIP). I agree it could be hard, but no one promised that implementing a server that is backward and forward compatible is easy. Everyone who votes on this KIP must be aware that whatever we do here is here to stay. Which is exactly why this discussion is so important. Gwen On Fri, Apr 1, 2016 at 1:52 PM, Ashish Singh <asi...@cloudera.com> wrote: > Never ever changing the version of VersionReq/resp will be ideal, but it is > hard to guarantee that we will never need a change. We should definitely > try to avoid it and honestly I do not see a reason, as of now, why it will > change. I would say, we should keep it versioned and document any risk of > changing it. Does that sound OK? > > On Fri, Apr 1, 2016 at 1:12 PM, Gwen Shapira <g...@confluent.io> wrote: > > > I am happy with: > > 1. Adding only validation support in the client for now, deferring > backward > > compatibility for later. With the odd side-effect of having the C and > > Python clients actually better than Java for a bit. > > 2. Never ever changing the version of VersionRequest and VersionResponse. > > > > I'd also like to see: > > 1. At least one client implement backward compatibility (so we can use it > > in testing) > > 2. Backward compatibility tests using that client (including an upgrade > > test) > > 3. Checklist for reviewers on when protocol bumps are required > > > > I am strongly against releasing an API that no one is using as > intended... > > so either we do it in our client or use a 3rd party client. > > > > Gwen > > > > On Fri, Apr 1, 2016 at 1:05 PM, Jason Gustafson <ja...@confluent.io> > > wrote: > > > > > I'm sympathetic to Jay's argument, but I feel the version API can be > > > justified on its own without requiring a big change in the way we > manage > > > compatibility. For the official client compatibility story (i.e. what > the > > > Java client implements), we can use this API to verify compatibility > with > > > the broker, which is a significant gap at the moment (we can do this > now > > > before the release if we all agree without much delay). It also happens > > to > > > help clients trying to maintain backwards compatibility, which is > great. > > > > > > Maybe the question is whether this API will restrict later efforts to > > > change the way we manage compatibility? I don't see much risk in terms > of > > > maintenance of the API, but there is some risk that it could become > > > superfluous at some point. For example, if we changed the response > header > > > to include an error code for an unknown request version, then maybe the > > > Java client would no longer need the initial call to the version API > > > (though it might arguably still be a good idea to do so to detect > > > incompatibility sooner). Maybe there is some risk also that it makes > the > > > compatibility story more confusing? I actually think it fits pretty > well > > > with the versioned API approach generally, though it's admittedly a > > little > > > odd to have the version API versioned itself. Practically speaking, I > > think > > > this means that we shouldn't increment the version. Otherwise the > client > > > will need another round trip (after always starting with v0), which > might > > > be doable even if it seems a little messy. > > > > > > Thoughts? > > > > > > -Jason > > > > > > On Fri, Apr 1, 2016 at 11:36 AM, Gwen Shapira <g...@confluent.io> > wrote: > > > > > > > My understanding is that we are not adding the client portion to the > > KIP > > > is > > > > because we believe it will require a lot of discussion (read: will be > > > > really hard to get right). > > > > > > > > Maybe it is a hint that this protocol is too difficult for clients to > > > > implement? > > > > I can't see why is it easy in C and Python and super difficult in > Java. > > > > > > > > Even if it is too hard to implement just in Java for some reason (god > > > knows > > > > the language has issues), isn't it a good reason to come up with > > > something > > > > we can implement in a reasonable way? > > > > > > > > Gwen > > > > > > > > On Fri, Apr 1, 2016 at 10:58 AM, Ashish Singh <asi...@cloudera.com> > > > wrote: > > > > > > > > > That is a fair concern and I think eventually we might want to have > > > java > > > > > clients backwards compatible. However, blocking KIP-35 on that > might > > > not > > > > be > > > > > the best idea. The reason I say so is due to following reasons. > > > > > > > > > > 1. Backwards compatibility in java clients is a larger discussion, > as > > > we > > > > > have already seen. Having a separate KIP focussing exactly on that > > will > > > > > help in reducing moving pieces in one KIP. > > > > > 2. It probably will fall out of 0.10 due to tight timeline, I am > > > assuming > > > > > we do not want to delay 0.10 a lot. > > > > > 3. Even if we make java clients backward compatible in 0.10, I do > not > > > > think > > > > > it will be able to work with older releases as older broker > versions > > > > still > > > > > won't provide info on supported api versions. If we add api > versions > > > > > req/resp in 0.10, and add backwards compatibility in java clients > in > > > > later > > > > > versions, they will probably work with 0.10 and we will be able to > > test > > > > > that. > > > > > > > > > > On Fri, Apr 1, 2016 at 10:28 AM, Ewen Cheslack-Postava < > > > > e...@confluent.io> > > > > > wrote: > > > > > > > > > > > On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira <g...@confluent.io > > > > > > wrote: > > > > > > > > > > > > > I have two concerns with the proposal as it is: > > > > > > > > > > > > > > 1. Having an API that publishes protocol is useless for clients > > if > > > > > > > developers don't bump the API when they should. > > > > > > > I would like to see good documentation on when protocols are > > bumped > > > > > and a > > > > > > > proposal on how this will be automatically validated to the > > extent > > > > > > possible > > > > > > > (and to what extent are we still at risk of accidental > breakage). > > > > > > > Even if we don't implement all of this at first pass, I want to > > > know > > > > > > which > > > > > > > direction we are going in order to solve the client > compatibility > > > > > issue. > > > > > > > > > > > > > > 2. In addition to third-party clients, there are stream > > processing > > > > > > > frameworks who use the Java client we publish as their client > and > > > > would > > > > > > > also like to enjoy the same compatibility benefits C and Python > > > > clients > > > > > > > will enjoy. It will be very silly if Apache Kafka clients are > the > > > > worst > > > > > > > clients out there from compatibility POV. The KIP can be > > > implemented > > > > in > > > > > > > parts, but I really want to see an effort to build the Java > > client > > > > > > > compatibility into the KIP and if possible into the release > too. > > > > > > > > > > > > > > > > > > > I wouldn't conflate those two things. Changing the compatibility > > > > approach > > > > > > of the Java clients could easily be another KIP (and probably a > > large > > > > > one, > > > > > > too). There are already high quality, well maintained clients > > trying > > > to > > > > > use > > > > > > a different approach and this addresses their needs. Blocking the > > > > entire > > > > > > ecosystem on the Java client seems problematic. > > > > > > > > > > > > That said, agreed that it would be bad for the Java client to > have > > > the > > > > > > worst compatibility story. > > > > > > > > > > > > -Ewen > > > > > > > > > > > > > > > > > > > > > > > > > > Gwen > > > > > > > > > > > > > > On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson < > > > ja...@confluent.io > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Bumping this thread. I talked with Ashish and Magnus about > this > > > KIP > > > > > > > offline > > > > > > > > and I'm gradually coming over. The new API actually stands by > > > > itself > > > > > > > > outside of the discussion about whether the client should > > support > > > > > > > backwards > > > > > > > > compatibility or not. For the Java client, we could continue > to > > > > > support > > > > > > > the > > > > > > > > current compatibility approach in which the client supports > > only > > > > > > brokers > > > > > > > > with the same version or greater. In that case, we would use > > this > > > > API > > > > > > > only > > > > > > > > to assert that the current API versions are all supported, > and > > > > raise > > > > > an > > > > > > > > exception if they are not. This gives us the capability going > > > > forward > > > > > > to > > > > > > > > detect when the client is talking to an older broker, which > we > > > > don't > > > > > > have > > > > > > > > right now. This check should be straightforward, so we could > do > > > it > > > > > now, > > > > > > > > which would resolve some of the uneasiness about having an > > unused > > > > > > feature > > > > > > > > which we depended on other clients to test for us. Does that > > make > > > > > sense > > > > > > > or > > > > > > > > not? > > > > > > > > > > > > > > > > -Jason > > > > > > > > > > > > > > > > On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh < > > > asi...@cloudera.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > We have proposed and discussed majorly three approaches so > > far, > > > > > there > > > > > > > > were > > > > > > > > > many minor versions with small variations. Comparing them > > > really > > > > > > > > requires a > > > > > > > > > side by side proposal and their pros/cons, and I agree with > > > > others > > > > > > that > > > > > > > > > this has been lacking in the KIP. We just updated the KIP > > with > > > > > > > following > > > > > > > > > details. > > > > > > > > > > > > > > > > > > 1. Provide proposed changes in all the three proposals we > > have > > > > > > > discussed > > > > > > > > so > > > > > > > > > far. Except the current proposal, these proposals are in > > > rejected > > > > > > > > > alternatives. > > > > > > > > > 2. Provide reasoning on why the rejected proposals were > > > rejected. > > > > > > > > > 3. Add scenarios for all of these proposals from a client > > > > developer > > > > > > and > > > > > > > > > core Kafka developer point of view. > > > > > > > > > > > > > > > > > > As we are really close to 0.10 deadline, a quick round of > > > voting > > > > > will > > > > > > > > > really help. If you really do not like the idea, please > feel > > > free > > > > > to > > > > > > > say > > > > > > > > > so. If the vote fails for the current proposal, it can at > > lease > > > > > > provide > > > > > > > > > recommendations that we should consider for next version of > > > > > proposal > > > > > > > and > > > > > > > > > put it up for vote again for next release. However, as > stated > > > > > earlier > > > > > > > by > > > > > > > > > multiple people having this ASAP will be awesome. > > > > > > > > > > > > > > > > > > On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers < > > > > > dana.pow...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira < > > > > g...@confluent.io > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > "I think I would make this approach work by looking at > > the > > > > > > released > > > > > > > > > > server > > > > > > > > > > > version documentation for each version that I am trying > > to > > > > > > support > > > > > > > > and > > > > > > > > > > test > > > > > > > > > > > against*, manually identify the expected "protocol > > vectors" > > > > > each > > > > > > > > > > supports, > > > > > > > > > > > store that as a map of vectors to "broker versions", > > check > > > > each > > > > > > > > vector > > > > > > > > > at > > > > > > > > > > > runtime until I find a match, and write code > > compatibility > > > > > checks > > > > > > > > from > > > > > > > > > > > there." > > > > > > > > > > > > > > > > > > > > > > How is this better than a global version ID? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As a client developer, it seems roughly the same. I think > > it > > > > > > probably > > > > > > > > > > avoids the server development workflow issues, and > possibly > > > the > > > > > > need > > > > > > > to > > > > > > > > > > agree on semantics of the global version ID? But others > > > surely > > > > > are > > > > > > > more > > > > > > > > > > qualified than me to comment on that part. > > > > > > > > > > > > > > > > > > > > -Dana > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Ashish > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Thanks, > > > > > > Ewen > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Regards, > > > > > Ashish > > > > > > > > > > > > > > > > > > -- > > Regards, > Ashish >
