Hey Ewen, with protocol design I actually do think good is the enemy of perfect (not sure if that makes sense but I think you get what I mean). Your comment seemed to be along the lines of "this isn't very good, but let's do something". Can you elaborate on what you think we should be doing?
The only two things I'd add to the comments so far: 1. We will probably live with this scheme for five years if we make this change. Let's for the love of god not do something half assed. It would be far far better not to do anything yet and come at this when we have the time to think it through, then to do something that doesn't make sense. 2. As Gwen says unless/until someone actually sees this through--gets the docs updated, implements the java client support, etc. we will be in a worse situation than we currently are. Is anyone actually signing up for that? I do think these things are coupled in actually getting us to a better place though they are separate chunks of work. I get that people want to see forward motion here but I really think this is an area where thoughtful beats fast. If we do something half-assed we first do the work of converting the world to that, then the work of undoing it and doing the thing we should have originally done so it really doesn't speed things up. -Jay On Fri, Apr 1, 2016 at 10:58 AM, Ashish Singh <asi...@cloudera.com> wrote: > That is a fair concern and I think eventually we might want to have java > clients backwards compatible. However, blocking KIP-35 on that might not be > the best idea. The reason I say so is due to following reasons. > > 1. Backwards compatibility in java clients is a larger discussion, as we > have already seen. Having a separate KIP focussing exactly on that will > help in reducing moving pieces in one KIP. > 2. It probably will fall out of 0.10 due to tight timeline, I am assuming > we do not want to delay 0.10 a lot. > 3. Even if we make java clients backward compatible in 0.10, I do not think > it will be able to work with older releases as older broker versions still > won't provide info on supported api versions. If we add api versions > req/resp in 0.10, and add backwards compatibility in java clients in later > versions, they will probably work with 0.10 and we will be able to test > that. > > On Fri, Apr 1, 2016 at 10:28 AM, Ewen Cheslack-Postava <e...@confluent.io> > wrote: > > > On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira <g...@confluent.io> wrote: > > > > > I have two concerns with the proposal as it is: > > > > > > 1. Having an API that publishes protocol is useless for clients if > > > developers don't bump the API when they should. > > > I would like to see good documentation on when protocols are bumped > and a > > > proposal on how this will be automatically validated to the extent > > possible > > > (and to what extent are we still at risk of accidental breakage). > > > Even if we don't implement all of this at first pass, I want to know > > which > > > direction we are going in order to solve the client compatibility > issue. > > > > > > 2. In addition to third-party clients, there are stream processing > > > frameworks who use the Java client we publish as their client and would > > > also like to enjoy the same compatibility benefits C and Python clients > > > will enjoy. It will be very silly if Apache Kafka clients are the worst > > > clients out there from compatibility POV. The KIP can be implemented in > > > parts, but I really want to see an effort to build the Java client > > > compatibility into the KIP and if possible into the release too. > > > > > > > I wouldn't conflate those two things. Changing the compatibility approach > > of the Java clients could easily be another KIP (and probably a large > one, > > too). There are already high quality, well maintained clients trying to > use > > a different approach and this addresses their needs. Blocking the entire > > ecosystem on the Java client seems problematic. > > > > That said, agreed that it would be bad for the Java client to have the > > worst compatibility story. > > > > -Ewen > > > > > > > > > > Gwen > > > > > > On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson <ja...@confluent.io> > > > wrote: > > > > > > > Bumping this thread. I talked with Ashish and Magnus about this KIP > > > offline > > > > and I'm gradually coming over. The new API actually stands by itself > > > > outside of the discussion about whether the client should support > > > backwards > > > > compatibility or not. For the Java client, we could continue to > support > > > the > > > > current compatibility approach in which the client supports only > > brokers > > > > with the same version or greater. In that case, we would use this API > > > only > > > > to assert that the current API versions are all supported, and raise > an > > > > exception if they are not. This gives us the capability going forward > > to > > > > detect when the client is talking to an older broker, which we don't > > have > > > > right now. This check should be straightforward, so we could do it > now, > > > > which would resolve some of the uneasiness about having an unused > > feature > > > > which we depended on other clients to test for us. Does that make > sense > > > or > > > > not? > > > > > > > > -Jason > > > > > > > > On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh <asi...@cloudera.com> > > > wrote: > > > > > > > > > We have proposed and discussed majorly three approaches so far, > there > > > > were > > > > > many minor versions with small variations. Comparing them really > > > > requires a > > > > > side by side proposal and their pros/cons, and I agree with others > > that > > > > > this has been lacking in the KIP. We just updated the KIP with > > > following > > > > > details. > > > > > > > > > > 1. Provide proposed changes in all the three proposals we have > > > discussed > > > > so > > > > > far. Except the current proposal, these proposals are in rejected > > > > > alternatives. > > > > > 2. Provide reasoning on why the rejected proposals were rejected. > > > > > 3. Add scenarios for all of these proposals from a client developer > > and > > > > > core Kafka developer point of view. > > > > > > > > > > As we are really close to 0.10 deadline, a quick round of voting > will > > > > > really help. If you really do not like the idea, please feel free > to > > > say > > > > > so. If the vote fails for the current proposal, it can at lease > > provide > > > > > recommendations that we should consider for next version of > proposal > > > and > > > > > put it up for vote again for next release. However, as stated > earlier > > > by > > > > > multiple people having this ASAP will be awesome. > > > > > > > > > > On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers < > dana.pow...@gmail.com> > > > > > wrote: > > > > > > > > > > > On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira <g...@confluent.io > > > > > > wrote: > > > > > > > > > > > > > "I think I would make this approach work by looking at the > > released > > > > > > server > > > > > > > version documentation for each version that I am trying to > > support > > > > and > > > > > > test > > > > > > > against*, manually identify the expected "protocol vectors" > each > > > > > > supports, > > > > > > > store that as a map of vectors to "broker versions", check each > > > > vector > > > > > at > > > > > > > runtime until I find a match, and write code compatibility > checks > > > > from > > > > > > > there." > > > > > > > > > > > > > > How is this better than a global version ID? > > > > > > > > > > > > > > > > > > As a client developer, it seems roughly the same. I think it > > probably > > > > > > avoids the server development workflow issues, and possibly the > > need > > > to > > > > > > agree on semantics of the global version ID? But others surely > are > > > more > > > > > > qualified than me to comment on that part. > > > > > > > > > > > > -Dana > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Regards, > > > > > Ashish > > > > > > > > > > > > > > > > > > > > -- > > Thanks, > > Ewen > > > > > > -- > > Regards, > Ashish >
