I'm sympathetic to Jay's argument, but I feel the version API can be justified on its own without requiring a big change in the way we manage compatibility. For the official client compatibility story (i.e. what the Java client implements), we can use this API to verify compatibility with the broker, which is a significant gap at the moment (we can do this now before the release if we all agree without much delay). It also happens to help clients trying to maintain backwards compatibility, which is great.
Maybe the question is whether this API will restrict later efforts to change the way we manage compatibility? I don't see much risk in terms of maintenance of the API, but there is some risk that it could become superfluous at some point. For example, if we changed the response header to include an error code for an unknown request version, then maybe the Java client would no longer need the initial call to the version API (though it might arguably still be a good idea to do so to detect incompatibility sooner). Maybe there is some risk also that it makes the compatibility story more confusing? I actually think it fits pretty well with the versioned API approach generally, though it's admittedly a little odd to have the version API versioned itself. Practically speaking, I think this means that we shouldn't increment the version. Otherwise the client will need another round trip (after always starting with v0), which might be doable even if it seems a little messy. Thoughts? -Jason On Fri, Apr 1, 2016 at 11:36 AM, Gwen Shapira <g...@confluent.io> wrote: > My understanding is that we are not adding the client portion to the KIP is > because we believe it will require a lot of discussion (read: will be > really hard to get right). > > Maybe it is a hint that this protocol is too difficult for clients to > implement? > I can't see why is it easy in C and Python and super difficult in Java. > > Even if it is too hard to implement just in Java for some reason (god knows > the language has issues), isn't it a good reason to come up with something > we can implement in a reasonable way? > > Gwen > > On Fri, Apr 1, 2016 at 10:58 AM, Ashish Singh <asi...@cloudera.com> wrote: > > > That is a fair concern and I think eventually we might want to have java > > clients backwards compatible. However, blocking KIP-35 on that might not > be > > the best idea. The reason I say so is due to following reasons. > > > > 1. Backwards compatibility in java clients is a larger discussion, as we > > have already seen. Having a separate KIP focussing exactly on that will > > help in reducing moving pieces in one KIP. > > 2. It probably will fall out of 0.10 due to tight timeline, I am assuming > > we do not want to delay 0.10 a lot. > > 3. Even if we make java clients backward compatible in 0.10, I do not > think > > it will be able to work with older releases as older broker versions > still > > won't provide info on supported api versions. If we add api versions > > req/resp in 0.10, and add backwards compatibility in java clients in > later > > versions, they will probably work with 0.10 and we will be able to test > > that. > > > > On Fri, Apr 1, 2016 at 10:28 AM, Ewen Cheslack-Postava < > e...@confluent.io> > > wrote: > > > > > On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira <g...@confluent.io> > wrote: > > > > > > > I have two concerns with the proposal as it is: > > > > > > > > 1. Having an API that publishes protocol is useless for clients if > > > > developers don't bump the API when they should. > > > > I would like to see good documentation on when protocols are bumped > > and a > > > > proposal on how this will be automatically validated to the extent > > > possible > > > > (and to what extent are we still at risk of accidental breakage). > > > > Even if we don't implement all of this at first pass, I want to know > > > which > > > > direction we are going in order to solve the client compatibility > > issue. > > > > > > > > 2. In addition to third-party clients, there are stream processing > > > > frameworks who use the Java client we publish as their client and > would > > > > also like to enjoy the same compatibility benefits C and Python > clients > > > > will enjoy. It will be very silly if Apache Kafka clients are the > worst > > > > clients out there from compatibility POV. The KIP can be implemented > in > > > > parts, but I really want to see an effort to build the Java client > > > > compatibility into the KIP and if possible into the release too. > > > > > > > > > > I wouldn't conflate those two things. Changing the compatibility > approach > > > of the Java clients could easily be another KIP (and probably a large > > one, > > > too). There are already high quality, well maintained clients trying to > > use > > > a different approach and this addresses their needs. Blocking the > entire > > > ecosystem on the Java client seems problematic. > > > > > > That said, agreed that it would be bad for the Java client to have the > > > worst compatibility story. > > > > > > -Ewen > > > > > > > > > > > > > > Gwen > > > > > > > > On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson <ja...@confluent.io > > > > > > wrote: > > > > > > > > > Bumping this thread. I talked with Ashish and Magnus about this KIP > > > > offline > > > > > and I'm gradually coming over. The new API actually stands by > itself > > > > > outside of the discussion about whether the client should support > > > > backwards > > > > > compatibility or not. For the Java client, we could continue to > > support > > > > the > > > > > current compatibility approach in which the client supports only > > > brokers > > > > > with the same version or greater. In that case, we would use this > API > > > > only > > > > > to assert that the current API versions are all supported, and > raise > > an > > > > > exception if they are not. This gives us the capability going > forward > > > to > > > > > detect when the client is talking to an older broker, which we > don't > > > have > > > > > right now. This check should be straightforward, so we could do it > > now, > > > > > which would resolve some of the uneasiness about having an unused > > > feature > > > > > which we depended on other clients to test for us. Does that make > > sense > > > > or > > > > > not? > > > > > > > > > > -Jason > > > > > > > > > > On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh <asi...@cloudera.com > > > > > > wrote: > > > > > > > > > > > We have proposed and discussed majorly three approaches so far, > > there > > > > > were > > > > > > many minor versions with small variations. Comparing them really > > > > > requires a > > > > > > side by side proposal and their pros/cons, and I agree with > others > > > that > > > > > > this has been lacking in the KIP. We just updated the KIP with > > > > following > > > > > > details. > > > > > > > > > > > > 1. Provide proposed changes in all the three proposals we have > > > > discussed > > > > > so > > > > > > far. Except the current proposal, these proposals are in rejected > > > > > > alternatives. > > > > > > 2. Provide reasoning on why the rejected proposals were rejected. > > > > > > 3. Add scenarios for all of these proposals from a client > developer > > > and > > > > > > core Kafka developer point of view. > > > > > > > > > > > > As we are really close to 0.10 deadline, a quick round of voting > > will > > > > > > really help. If you really do not like the idea, please feel free > > to > > > > say > > > > > > so. If the vote fails for the current proposal, it can at lease > > > provide > > > > > > recommendations that we should consider for next version of > > proposal > > > > and > > > > > > put it up for vote again for next release. However, as stated > > earlier > > > > by > > > > > > multiple people having this ASAP will be awesome. > > > > > > > > > > > > On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers < > > dana.pow...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira < > g...@confluent.io > > > > > > > > wrote: > > > > > > > > > > > > > > > "I think I would make this approach work by looking at the > > > released > > > > > > > server > > > > > > > > version documentation for each version that I am trying to > > > support > > > > > and > > > > > > > test > > > > > > > > against*, manually identify the expected "protocol vectors" > > each > > > > > > > supports, > > > > > > > > store that as a map of vectors to "broker versions", check > each > > > > > vector > > > > > > at > > > > > > > > runtime until I find a match, and write code compatibility > > checks > > > > > from > > > > > > > > there." > > > > > > > > > > > > > > > > How is this better than a global version ID? > > > > > > > > > > > > > > > > > > > > > As a client developer, it seems roughly the same. I think it > > > probably > > > > > > > avoids the server development workflow issues, and possibly the > > > need > > > > to > > > > > > > agree on semantics of the global version ID? But others surely > > are > > > > more > > > > > > > qualified than me to comment on that part. > > > > > > > > > > > > > > -Dana > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Regards, > > > > > > Ashish > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Thanks, > > > Ewen > > > > > > > > > > > -- > > > > Regards, > > Ashish > > >
