Re: [VOTE] KIP-35: Retrieving protocol version

[Dana Powers]

Great questions.

But I wonder if we're expanding the scope of this KIP too much? The
questions you've raised relate to java client development, and broker
testing. Issues #2 and #3 are not related to this KIP at all: how does the
server test and validate api protocol changes and compatibility? That is a
fundamental question regardless whether clients can get version metadata or
not. It is the basis for client forwards compatibility (and broker
backwards compatibility).

While these are great questions that need good solutions, I dont think the
KIP was intended to solve them. Rather, it is aimed at kafka clients that
are attempting to be backwards compatible, namely librdkafka and
kafka-python. It also seems that the Java client dev team doesn't have
backwards compatible clients as high on the priority list as we do. That's
ok! But lets not let that delay or prevent the *server* dev team from
adding support for this simple API to help other client teams.

Recall that you were both (Jay and Gwen) in favor of this approach a year
ago:
http://mail-archives.apache.org/mod_mbox/kafka-dev/201501.mbox/%3ccaoejijgnidpnvr-tpcpkzqfkuwdj+rtymvgkkkdwmkczfqm...@mail.gmail.com%3E

KIP-35 is exactly option #2 from that thread. This doesn't seem like a
controversial API at all.

It's a bit frustrating that something this simple, and which is seemingly
unopposed, is taking so long to get approval. If there's anything I can do
to help facilitate, please let me know.

-Dana
We (Jay + me) had some extra information we wanted to see in the KIP before
we are comfortable voting:

* Where does the Java client fits in. Hopefully we can use this KIP to
standardize behavior and guarantees between Java and non-Java clients, so
when we reason about the Java clients, which most Kafka developers are
familiar with, we will make the right decisions for all clients.
* When do we bump the protocol? I think 90% of the issue is not that the
version got bumped but rather that we changed behavior without bumping
versions. For the new VersionRequest to be useful, we need to all know when
to get new versions...
* How do we test / validate - I think our recent experience shows that our
protocol tests and compatibility tests are still inadequate. Having
VersionRequest is useless if we can't validate that Kafka actually
implements the protocol it says it does (and we caught this breaks twice in
the last two weeks)
* Error handling of protocol mismatches

Ashish kindly agreed to think about this and improve the KIP.
We'll resume the vote as soon as he's back :)

Gwen


On Wed, Mar 23, 2016 at 5:55 PM, Dana Powers  wrote:

> speaking of pending KIPs, what's the status on this one?
>
>
> On Fri, Mar 18, 2016 at 9:47 PM, Ashish Singh  wrote:
>
> > Hey Jay,
> >
> > Answers inline.
> >
> > On Fri, Mar 18, 2016 at 10:45 AM, Jay Kreps  wrote:
> >
> > Hey Ashish,
> > >
> > > Couple quick things:
> > >
> > > 1. You list as a rejected alternative "making the documentation the
> > > source of truth for the protocol", but I think what you actually
> > > describe in that section is global versioning, which of those two
> > > things are we voting to reject? I think this is a philosophical point
> > > but an important one...
> > >
> > One of the major differences between Option 3 and other options
discussed
> > on KIP is that Option 3 is documentation oriented and it is that what I
> > wanted to capture in the title. I am happy to change it to global
> > versioning.
> >
> >
> > > 2. Can you describe the changes necessary and classes we'd have to
> > > update in the java clients to make use of this feature? What would
> > > that look like? One concern I have is just the complexity necessary to
> > > do the per-connection protocol version check and really handle all the
> > > cases. I assume you've thought through what that looks like, can you
> > > sketch that out for people?
> > >
> > I would imagine any client, even Java client, would follow the steps
> > mentioned here
> > <
> >
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-Aclientdeveloperwantstoaddsupportforanewfeature.1
> > >.
> > Below are my thoughts on how java client can maintain api versions
> > supported by various brokers in cluster.
> >
> >1. ClusterConnectionStates can provide info on whether api versions
> have
> >been retrieved for a connection or not.
> >2. NetworkClient.handleConnections can send ApiVersionQueryRequest to
> >newly connected nodes.
> >3. NetworkClient can be enhanced to handle ApiVersionQueryResponse
and
> >set ClusterConnectionStates to indicate api versions have been
> retrieved
> >for the node.
> >4. NetworkClient maintains mapping Node -> [(api_key, min_ver,
> >max_ver)], brokerApiVersions, cached.
> >5. NetworkClient.processDisconnection can remove entry for a node
from
> >brokerApiVersions 

Re: [VOTE] KIP-35: Retrieving protocol version

[Gwen Shapira]

We (Jay + me) had some extra information we wanted to see in the KIP before
we are comfortable voting:

* Where does the Java client fits in. Hopefully we can use this KIP to
standardize behavior and guarantees between Java and non-Java clients, so
when we reason about the Java clients, which most Kafka developers are
familiar with, we will make the right decisions for all clients.
* When do we bump the protocol? I think 90% of the issue is not that the
version got bumped but rather that we changed behavior without bumping
versions. For the new VersionRequest to be useful, we need to all know when
to get new versions...
* How do we test / validate - I think our recent experience shows that our
protocol tests and compatibility tests are still inadequate. Having
VersionRequest is useless if we can't validate that Kafka actually
implements the protocol it says it does (and we caught this breaks twice in
the last two weeks)
* Error handling of protocol mismatches

Ashish kindly agreed to think about this and improve the KIP.
We'll resume the vote as soon as he's back :)

Gwen


On Wed, Mar 23, 2016 at 5:55 PM, Dana Powers  wrote:

> speaking of pending KIPs, what's the status on this one?
>
>
> On Fri, Mar 18, 2016 at 9:47 PM, Ashish Singh  wrote:
>
> > Hey Jay,
> >
> > Answers inline.
> >
> > On Fri, Mar 18, 2016 at 10:45 AM, Jay Kreps  wrote:
> >
> > Hey Ashish,
> > >
> > > Couple quick things:
> > >
> > > 1. You list as a rejected alternative "making the documentation the
> > > source of truth for the protocol", but I think what you actually
> > > describe in that section is global versioning, which of those two
> > > things are we voting to reject? I think this is a philosophical point
> > > but an important one...
> > >
> > One of the major differences between Option 3 and other options discussed
> > on KIP is that Option 3 is documentation oriented and it is that what I
> > wanted to capture in the title. I am happy to change it to global
> > versioning.
> >
> >
> > > 2. Can you describe the changes necessary and classes we'd have to
> > > update in the java clients to make use of this feature? What would
> > > that look like? One concern I have is just the complexity necessary to
> > > do the per-connection protocol version check and really handle all the
> > > cases. I assume you've thought through what that looks like, can you
> > > sketch that out for people?
> > >
> > I would imagine any client, even Java client, would follow the steps
> > mentioned here
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-Aclientdeveloperwantstoaddsupportforanewfeature.1
> > >.
> > Below are my thoughts on how java client can maintain api versions
> > supported by various brokers in cluster.
> >
> >1. ClusterConnectionStates can provide info on whether api versions
> have
> >been retrieved for a connection or not.
> >2. NetworkClient.handleConnections can send ApiVersionQueryRequest to
> >newly connected nodes.
> >3. NetworkClient can be enhanced to handle ApiVersionQueryResponse and
> >set ClusterConnectionStates to indicate api versions have been
> retrieved
> >for the node.
> >4. NetworkClient maintains mapping Node -> [(api_key, min_ver,
> >max_ver)], brokerApiVersions, cached.
> >5. NetworkClient.processDisconnection can remove entry for a node from
> >brokerApiVersions cache.
> >6. NetworkClient.canSendRequest can have an added condition on node to
> >have api versions available.
> >
> > With the above changes, at any given point of time NetworkClient will be
> > aware of api versions supported by each of the connected nodes. I am not
> > sure if the above changes are the best way to do it, people are welcome
> to
> > pitch in. Does it help?
> >
> >
> > > -Jay
> > >
> > > On Mon, Mar 14, 2016 at 3:54 PM, Ashish Singh 
> > wrote:
> > > > Hey Guys,
> > > >
> > > > I would like to start voting process for *KIP-35: Retrieving protocol
> > > > version*. The KIP is available here
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> > > >.
> > > > Here
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-SummaryofthechangesproposedaspartofthisKIP
> > > >
> > > > is a brief summary of the KIP.
> > > >
> > > > The vote will run for 72 hours.
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashish
> > >
> > ​
> > --
> >
> > Regards,
> > Ashish
> >
>

Re: [VOTE] KIP-35: Retrieving protocol version

[Dana Powers]

speaking of pending KIPs, what's the status on this one?


On Fri, Mar 18, 2016 at 9:47 PM, Ashish Singh  wrote:

> Hey Jay,
>
> Answers inline.
>
> On Fri, Mar 18, 2016 at 10:45 AM, Jay Kreps  wrote:
>
> Hey Ashish,
> >
> > Couple quick things:
> >
> > 1. You list as a rejected alternative "making the documentation the
> > source of truth for the protocol", but I think what you actually
> > describe in that section is global versioning, which of those two
> > things are we voting to reject? I think this is a philosophical point
> > but an important one...
> >
> One of the major differences between Option 3 and other options discussed
> on KIP is that Option 3 is documentation oriented and it is that what I
> wanted to capture in the title. I am happy to change it to global
> versioning.
>
>
> > 2. Can you describe the changes necessary and classes we'd have to
> > update in the java clients to make use of this feature? What would
> > that look like? One concern I have is just the complexity necessary to
> > do the per-connection protocol version check and really handle all the
> > cases. I assume you've thought through what that looks like, can you
> > sketch that out for people?
> >
> I would imagine any client, even Java client, would follow the steps
> mentioned here
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-Aclientdeveloperwantstoaddsupportforanewfeature.1
> >.
> Below are my thoughts on how java client can maintain api versions
> supported by various brokers in cluster.
>
>1. ClusterConnectionStates can provide info on whether api versions have
>been retrieved for a connection or not.
>2. NetworkClient.handleConnections can send ApiVersionQueryRequest to
>newly connected nodes.
>3. NetworkClient can be enhanced to handle ApiVersionQueryResponse and
>set ClusterConnectionStates to indicate api versions have been retrieved
>for the node.
>4. NetworkClient maintains mapping Node -> [(api_key, min_ver,
>max_ver)], brokerApiVersions, cached.
>5. NetworkClient.processDisconnection can remove entry for a node from
>brokerApiVersions cache.
>6. NetworkClient.canSendRequest can have an added condition on node to
>have api versions available.
>
> With the above changes, at any given point of time NetworkClient will be
> aware of api versions supported by each of the connected nodes. I am not
> sure if the above changes are the best way to do it, people are welcome to
> pitch in. Does it help?
>
>
> > -Jay
> >
> > On Mon, Mar 14, 2016 at 3:54 PM, Ashish Singh 
> wrote:
> > > Hey Guys,
> > >
> > > I would like to start voting process for *KIP-35: Retrieving protocol
> > > version*. The KIP is available here
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> > >.
> > > Here
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-SummaryofthechangesproposedaspartofthisKIP
> > >
> > > is a brief summary of the KIP.
> > >
> > > The vote will run for 72 hours.
> > >
> > > --
> > >
> > > Regards,
> > > Ashish
> >
> ​
> --
>
> Regards,
> Ashish
>

Re: [VOTE] KIP-35: Retrieving protocol version

[Jay Kreps]

Hey Ashish,

Couple quick things:

1. You list as a rejected alternative "making the documentation the
source of truth for the protocol", but I think what you actually
describe in that section is global versioning, which of those two
things are we voting to reject? I think this is a philosophical point
but an important one...

2. Can you describe the changes necessary and classes we'd have to
update in the java clients to make use of this feature? What would
that look like? One concern I have is just the complexity necessary to
do the per-connection protocol version check and really handle all the
cases. I assume you've thought through what that looks like, can you
sketch that out for people?

-Jay

On Mon, Mar 14, 2016 at 3:54 PM, Ashish Singh  wrote:
> Hey Guys,
>
> I would like to start voting process for *KIP-35: Retrieving protocol
> version*. The KIP is available here
> .
> Here
> 
> is a brief summary of the KIP.
>
> The vote will run for 72 hours.
>
> --
>
> Regards,
> Ashish

Re: [VOTE] KIP-35: Retrieving protocol version

[Magnus Edenhill]

+1 (nonbinding)

2016-03-18 0:09 GMT+01:00 Ashish Singh :

> As the KIP has been modified since we started this vote, the vote is
> restarted from now.
>
> The updated KIP is available at
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> .
>
> On Mon, Mar 14, 2016 at 3:54 PM, Ashish Singh  wrote:
>
> > Hey Guys,
> >
> > I would like to start voting process for *KIP-35: Retrieving protocol
> > version*. The KIP is available here
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> >.
> > Here
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-SummaryofthechangesproposedaspartofthisKIP
> >
> > is a brief summary of the KIP.
> >
> > The vote will run for 72 hours.
> > ​
> > --
> >
> > Regards,
> > Ashish
> >
>
>
>
> --
>
> Regards,
> Ashish
>

Re: [VOTE] KIP-35: Retrieving protocol version

[Grant Henke]

+1 (non-binding)

On Fri, Mar 18, 2016 at 2:41 AM, Magnus Edenhill  wrote:

> +1 (nonbinding)
>
> 2016-03-18 0:09 GMT+01:00 Ashish Singh :
>
> > As the KIP has been modified since we started this vote, the vote is
> > restarted from now.
> >
> > The updated KIP is available at
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> > .
> >
> > On Mon, Mar 14, 2016 at 3:54 PM, Ashish Singh 
> wrote:
> >
> > > Hey Guys,
> > >
> > > I would like to start voting process for *KIP-35: Retrieving protocol
> > > version*. The KIP is available here
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> > >.
> > > Here
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-SummaryofthechangesproposedaspartofthisKIP
> > >
> > > is a brief summary of the KIP.
> > >
> > > The vote will run for 72 hours.
> > > ​
> > > --
> > >
> > > Regards,
> > > Ashish
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: [VOTE] KIP-35: Retrieving protocol version

[Ashish Singh]

As the KIP has been modified since we started this vote, the vote is
restarted from now.

The updated KIP is available at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
.

On Mon, Mar 14, 2016 at 3:54 PM, Ashish Singh  wrote:

> Hey Guys,
>
> I would like to start voting process for *KIP-35: Retrieving protocol
> version*. The KIP is available here
> .
> Here
> 
> is a brief summary of the KIP.
>
> The vote will run for 72 hours.
> ​
> --
>
> Regards,
> Ashish
>



-- 

Regards,
Ashish

Re: [VOTE] KIP-35: Retrieving protocol version

[Dana Powers]

+1

On Thu, Mar 17, 2016 at 4:09 PM, Ashish Singh  wrote:

> As the KIP has been modified since we started this vote, the vote is
> restarted from now.
>
> The updated KIP is available at
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> .
>
> On Mon, Mar 14, 2016 at 3:54 PM, Ashish Singh  wrote:
>
> > Hey Guys,
> >
> > I would like to start voting process for *KIP-35: Retrieving protocol
> > version*. The KIP is available here
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> >.
> > Here
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-SummaryofthechangesproposedaspartofthisKIP
> >
> > is a brief summary of the KIP.
> >
> > The vote will run for 72 hours.
> > ​
> > --
> >
> > Regards,
> > Ashish
> >
>
>
>
> --
>
> Regards,
> Ashish
>

Re: [VOTE] KIP-35: Retrieving protocol version

[Ashish Singh]

Hey Jay,

Answers inline.

On Fri, Mar 18, 2016 at 10:45 AM, Jay Kreps  wrote:

Hey Ashish,
>
> Couple quick things:
>
> 1. You list as a rejected alternative "making the documentation the
> source of truth for the protocol", but I think what you actually
> describe in that section is global versioning, which of those two
> things are we voting to reject? I think this is a philosophical point
> but an important one...
>
One of the major differences between Option 3 and other options discussed
on KIP is that Option 3 is documentation oriented and it is that what I
wanted to capture in the title. I am happy to change it to global
versioning.


> 2. Can you describe the changes necessary and classes we'd have to
> update in the java clients to make use of this feature? What would
> that look like? One concern I have is just the complexity necessary to
> do the per-connection protocol version check and really handle all the
> cases. I assume you've thought through what that looks like, can you
> sketch that out for people?
>
I would imagine any client, even Java client, would follow the steps
mentioned here
.
Below are my thoughts on how java client can maintain api versions
supported by various brokers in cluster.

   1. ClusterConnectionStates can provide info on whether api versions have
   been retrieved for a connection or not.
   2. NetworkClient.handleConnections can send ApiVersionQueryRequest to
   newly connected nodes.
   3. NetworkClient can be enhanced to handle ApiVersionQueryResponse and
   set ClusterConnectionStates to indicate api versions have been retrieved
   for the node.
   4. NetworkClient maintains mapping Node -> [(api_key, min_ver,
   max_ver)], brokerApiVersions, cached.
   5. NetworkClient.processDisconnection can remove entry for a node from
   brokerApiVersions cache.
   6. NetworkClient.canSendRequest can have an added condition on node to
   have api versions available.

With the above changes, at any given point of time NetworkClient will be
aware of api versions supported by each of the connected nodes. I am not
sure if the above changes are the best way to do it, people are welcome to
pitch in. Does it help?


> -Jay
>
> On Mon, Mar 14, 2016 at 3:54 PM, Ashish Singh  wrote:
> > Hey Guys,
> >
> > I would like to start voting process for *KIP-35: Retrieving protocol
> > version*. The KIP is available here
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> >.
> > Here
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-SummaryofthechangesproposedaspartofthisKIP
> >
> > is a brief summary of the KIP.
> >
> > The vote will run for 72 hours.
> >
> > --
> >
> > Regards,
> > Ashish
>
​
-- 

Regards,
Ashish

[VOTE] KIP-35: Retrieving protocol version

[Ashish Singh]

Hey Guys,

I would like to start voting process for *KIP-35: Retrieving protocol
version*. The KIP is available here
.
Here

is a brief summary of the KIP.

The vote will run for 72 hours.
​
-- 

Regards,
Ashish