Great questions. But I wonder if we're expanding the scope of this KIP too much? The questions you've raised relate to java client development, and broker testing. Issues #2 and #3 are not related to this KIP at all: how does the server test and validate api protocol changes and compatibility? That is a fundamental question regardless whether clients can get version metadata or not. It is the basis for client forwards compatibility (and broker backwards compatibility).
While these are great questions that need good solutions, I dont think the KIP was intended to solve them. Rather, it is aimed at kafka clients that are attempting to be backwards compatible, namely librdkafka and kafka-python. It also seems that the Java client dev team doesn't have backwards compatible clients as high on the priority list as we do. That's ok! But lets not let that delay or prevent the *server* dev team from adding support for this simple API to help other client teams. Recall that you were both (Jay and Gwen) in favor of this approach a year ago: http://mail-archives.apache.org/mod_mbox/kafka-dev/201501.mbox/%3ccaoejijgnidpnvr-tpcpkzqfkuwdj+rtymvgkkkdwmkczfqm...@mail.gmail.com%3E KIP-35 is exactly option #2 from that thread. This doesn't seem like a controversial API at all. It's a bit frustrating that something this simple, and which is seemingly unopposed, is taking so long to get approval. If there's anything I can do to help facilitate, please let me know. -Dana We (Jay + me) had some extra information we wanted to see in the KIP before we are comfortable voting: * Where does the Java client fits in. Hopefully we can use this KIP to standardize behavior and guarantees between Java and non-Java clients, so when we reason about the Java clients, which most Kafka developers are familiar with, we will make the right decisions for all clients. * When do we bump the protocol? I think 90% of the issue is not that the version got bumped but rather that we changed behavior without bumping versions. For the new VersionRequest to be useful, we need to all know when to get new versions... * How do we test / validate - I think our recent experience shows that our protocol tests and compatibility tests are still inadequate. Having VersionRequest is useless if we can't validate that Kafka actually implements the protocol it says it does (and we caught this breaks twice in the last two weeks) * Error handling of protocol mismatches Ashish kindly agreed to think about this and improve the KIP. We'll resume the vote as soon as he's back :) Gwen On Wed, Mar 23, 2016 at 5:55 PM, Dana Powers <dana.pow...@gmail.com> wrote: > speaking of pending KIPs, what's the status on this one? > > > On Fri, Mar 18, 2016 at 9:47 PM, Ashish Singh <asi...@cloudera.com> wrote: > > > Hey Jay, > > > > Answers inline. > > > > On Fri, Mar 18, 2016 at 10:45 AM, Jay Kreps <j...@confluent.io> wrote: > > > > Hey Ashish, > > > > > > Couple quick things: > > > > > > 1. You list as a rejected alternative "making the documentation the > > > source of truth for the protocol", but I think what you actually > > > describe in that section is global versioning, which of those two > > > things are we voting to reject? I think this is a philosophical point > > > but an important one... > > > > > One of the major differences between Option 3 and other options discussed > > on KIP is that Option 3 is documentation oriented and it is that what I > > wanted to capture in the title. I am happy to change it to global > > versioning. > > > > > > > 2. Can you describe the changes necessary and classes we'd have to > > > update in the java clients to make use of this feature? What would > > > that look like? One concern I have is just the complexity necessary to > > > do the per-connection protocol version check and really handle all the > > > cases. I assume you've thought through what that looks like, can you > > > sketch that out for people? > > > > > I would imagine any client, even Java client, would follow the steps > > mentioned here > > < > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-Aclientdeveloperwantstoaddsupportforanewfeature.1 > > >. > > Below are my thoughts on how java client can maintain api versions > > supported by various brokers in cluster. > > > > 1. ClusterConnectionStates can provide info on whether api versions > have > > been retrieved for a connection or not. > > 2. NetworkClient.handleConnections can send ApiVersionQueryRequest to > > newly connected nodes. > > 3. NetworkClient can be enhanced to handle ApiVersionQueryResponse and > > set ClusterConnectionStates to indicate api versions have been > retrieved > > for the node. > > 4. NetworkClient maintains mapping Node -> [(api_key, min_ver, > > max_ver)], brokerApiVersions, cached. > > 5. NetworkClient.processDisconnection can remove entry for a node from > > brokerApiVersions cache. > > 6. NetworkClient.canSendRequest can have an added condition on node to > > have api versions available. > > > > With the above changes, at any given point of time NetworkClient will be > > aware of api versions supported by each of the connected nodes. I am not > > sure if the above changes are the best way to do it, people are welcome > to > > pitch in. Does it help? > > > > > > > -Jay > > > > > > On Mon, Mar 14, 2016 at 3:54 PM, Ashish Singh <asi...@cloudera.com> > > wrote: > > > > Hey Guys, > > > > > > > > I would like to start voting process for *KIP-35: Retrieving protocol > > > > version*. The KIP is available here > > > > < > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version > > > >. > > > > Here > > > > < > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-SummaryofthechangesproposedaspartofthisKIP > > > > > > > > is a brief summary of the KIP. > > > > > > > > The vote will run for 72 hours. > > > > > > > > -- > > > > > > > > Regards, > > > > Ashish > > > > > > > -- > > > > Regards, > > Ashish > > >
