Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Done. Thanks!

On Thu, Apr 21, 2016 at 4:32 PM, Gwen Shapira  wrote:

> Lets start a vote immediately? We are short of time toward the release.
>
> On Thu, Apr 21, 2016 at 2:57 PM, Ashish Singh  wrote:
> > Hey Guys,
> >
> > KIP-35
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> >
> > has been updated based on latest discussions and following PRs have also
> > been updated.
> > 1. KAFKA-3307: Add ApiVersion request/response and server side handling.
> > 
> > 2. KAFKA-3600: Enhance java clients to use ApiVersion Req/Resp to check
> if
> > the broker they are talking to supports required api versions.
> > 
> >
> > If there are no major objections or changes suggested, we can start a
> vote
> > thread in a couple of days.
> >
> > On Tue, Apr 12, 2016 at 8:04 AM, Jun Rao  wrote:
> >
> >> Hi, Ismael,
> >>
> >> The SASL engine that we used is the SASL library, right? How did the C
> >> client generate those SASL tokens? Once a SASL mechanism is chosen, the
> >> subsequent tokens are determined, right? So, my feeling is that those
> >> tokens are part of SaslHandshakeRequest and are just extended across
> >> multiple network packets. So modeling those as independent requests
> feels
> >> weird. When documentation them, we really need to document those as a
> >> sequence, not individual isolated requests that can be issued
> >> in arbitrary order. The version id will only add confusion since we
> can't
> >> version the tokens independently. We could explicitly add the client id
> and
> >> correlation id in the header, but I am not sure if it's worth the
> >> additional complexity.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Tue, Apr 12, 2016 at 1:18 AM, Ismael Juma  wrote:
> >>
> >> > Hi Jun,
> >> >
> >> > I understand the point about the SASL tokens being similar to the SSL
> >> > handshake in a way. However, is there any SASL library that handles
> the
> >> > network communication for these tokens? I couldn't find any and
> without
> >> > that, there isn't much benefit in deviating from Kafka's protocol (we
> >> > basically save the space taken by the request header). It's worth
> >> > mentioning that we are already adding the message size before the
> opaque
> >> > bytes provided by the library, so one could say we are already
> extending
> >> > the protocol.
> >> >
> >> > If we leave versioning aside, adding the standard Kafka request
> header to
> >> > those messages may also help from a debugging perspective as would
> then
> >> > include client id and correlation id along with the message.
> >> >
> >> > Ismael
> >> >
> >> > On Tue, Apr 12, 2016 at 2:13 AM, Jun Rao  wrote:
> >> >
> >> > > Magnus,
> >> > >
> >> > > That sounds reasonable. To reduce the changes on the server side,
> I'd
> >> > > suggest the following minor tweaks on the proposal.
> >> > >
> >> > > 1. Continue supporting the separate SASL and SASL_SSL port.
> >> > >
> >> > > On SASL port, we support the new sequence
> >> > > ApiVersionRequest (optional), SaslHandshakeRequest, SASL tokens,
> >> > > regular
> >> > > requests
> >> > >
> >> > > On SASL_SSL port, we support the new sequence
> >> > > SSL handshake bytes, ApiVersionRequest (optional),
> >> > > SaslHandshakeRequest,
> >> > > SASL tokens, regular requests
> >> > >
> >> > > 2. We don't wrap SASL tokens in Kafka protocol. Similar to your
> >> argument
> >> > > about SSL handshake, those SASL tokens are generated by SASL library
> >> and
> >> > > Kafka doesn't really control its versioning. Kafka only controls the
> >> > > selection of SASL mechanism, which will be versioned in
> >> > > SaslHandshakeRequest.
> >> > >
> >> > > Does that work for you?
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Jun
> >> > >
> >> > >
> >> > > On Mon, Apr 11, 2016 at 11:15 AM, Magnus Edenhill <
> mag...@edenhill.se>
> >> > > wrote:
> >> > >
> >> > > > Hey Jun, see inline
> >> > > >
> >> > > > 2016-04-11 19:19 GMT+02:00 Jun Rao :
> >> > > >
> >> > > > > Hi, Magnus,
> >> > > > >
> >> > > > > Let me understand your proposal in more details just from the
> >> > client's
> >> > > > > perspective. My understanding of your proposal is the following.
> >> > > > >
> >> > > > > On plaintext port, the client will send the following bytes in
> >> order.
> >> > > > > ApiVersionRequest, SaslHandshakeRequest, SASL tokens (if
> SASL
> >> is
> >> > > > > enabled), regular requests
> >> > > > >
> >> > > > > On SSL port, the client will send the following bytes in order.
> >> > > > > SSL handshake bytes, ApiVersionRequest,
> SaslHandshakeRequest,
> >> > SASL
> >> > > > > tokens (if SASL is enabled), regular requests
> >> > > > >
> >> > > >
> >> > > >
> >> > > > Yup!
> >> > > > "SASL tokens" is a series of proper Kafka protocol
> >> > SaslHandshakeRequests
> >> > > > 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

Lets start a vote immediately? We are short of time toward the release.

On Thu, Apr 21, 2016 at 2:57 PM, Ashish Singh  wrote:
> Hey Guys,
>
> KIP-35
> 
> has been updated based on latest discussions and following PRs have also
> been updated.
> 1. KAFKA-3307: Add ApiVersion request/response and server side handling.
> 
> 2. KAFKA-3600: Enhance java clients to use ApiVersion Req/Resp to check if
> the broker they are talking to supports required api versions.
> 
>
> If there are no major objections or changes suggested, we can start a vote
> thread in a couple of days.
>
> On Tue, Apr 12, 2016 at 8:04 AM, Jun Rao  wrote:
>
>> Hi, Ismael,
>>
>> The SASL engine that we used is the SASL library, right? How did the C
>> client generate those SASL tokens? Once a SASL mechanism is chosen, the
>> subsequent tokens are determined, right? So, my feeling is that those
>> tokens are part of SaslHandshakeRequest and are just extended across
>> multiple network packets. So modeling those as independent requests feels
>> weird. When documentation them, we really need to document those as a
>> sequence, not individual isolated requests that can be issued
>> in arbitrary order. The version id will only add confusion since we can't
>> version the tokens independently. We could explicitly add the client id and
>> correlation id in the header, but I am not sure if it's worth the
>> additional complexity.
>>
>> Thanks,
>>
>> Jun
>>
>> On Tue, Apr 12, 2016 at 1:18 AM, Ismael Juma  wrote:
>>
>> > Hi Jun,
>> >
>> > I understand the point about the SASL tokens being similar to the SSL
>> > handshake in a way. However, is there any SASL library that handles the
>> > network communication for these tokens? I couldn't find any and without
>> > that, there isn't much benefit in deviating from Kafka's protocol (we
>> > basically save the space taken by the request header). It's worth
>> > mentioning that we are already adding the message size before the opaque
>> > bytes provided by the library, so one could say we are already extending
>> > the protocol.
>> >
>> > If we leave versioning aside, adding the standard Kafka request header to
>> > those messages may also help from a debugging perspective as would then
>> > include client id and correlation id along with the message.
>> >
>> > Ismael
>> >
>> > On Tue, Apr 12, 2016 at 2:13 AM, Jun Rao  wrote:
>> >
>> > > Magnus,
>> > >
>> > > That sounds reasonable. To reduce the changes on the server side, I'd
>> > > suggest the following minor tweaks on the proposal.
>> > >
>> > > 1. Continue supporting the separate SASL and SASL_SSL port.
>> > >
>> > > On SASL port, we support the new sequence
>> > > ApiVersionRequest (optional), SaslHandshakeRequest, SASL tokens,
>> > > regular
>> > > requests
>> > >
>> > > On SASL_SSL port, we support the new sequence
>> > > SSL handshake bytes, ApiVersionRequest (optional),
>> > > SaslHandshakeRequest,
>> > > SASL tokens, regular requests
>> > >
>> > > 2. We don't wrap SASL tokens in Kafka protocol. Similar to your
>> argument
>> > > about SSL handshake, those SASL tokens are generated by SASL library
>> and
>> > > Kafka doesn't really control its versioning. Kafka only controls the
>> > > selection of SASL mechanism, which will be versioned in
>> > > SaslHandshakeRequest.
>> > >
>> > > Does that work for you?
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > >
>> > > On Mon, Apr 11, 2016 at 11:15 AM, Magnus Edenhill 
>> > > wrote:
>> > >
>> > > > Hey Jun, see inline
>> > > >
>> > > > 2016-04-11 19:19 GMT+02:00 Jun Rao :
>> > > >
>> > > > > Hi, Magnus,
>> > > > >
>> > > > > Let me understand your proposal in more details just from the
>> > client's
>> > > > > perspective. My understanding of your proposal is the following.
>> > > > >
>> > > > > On plaintext port, the client will send the following bytes in
>> order.
>> > > > > ApiVersionRequest, SaslHandshakeRequest, SASL tokens (if SASL
>> is
>> > > > > enabled), regular requests
>> > > > >
>> > > > > On SSL port, the client will send the following bytes in order.
>> > > > > SSL handshake bytes, ApiVersionRequest, SaslHandshakeRequest,
>> > SASL
>> > > > > tokens (if SASL is enabled), regular requests
>> > > > >
>> > > >
>> > > >
>> > > > Yup!
>> > > > "SASL tokens" is a series of proper Kafka protocol
>> > SaslHandshakeRequests
>> > > > until
>> > > > the handshake is done.
>> > > >
>> > > >
>> > > >
>> > > > >
>> > > > > Is that right? Since we can use either SSL or SASL for
>> > authentication,
>> > > > it's
>> > > > > weird that in one case, we require ApiVersionRequest to happen
>> before
>> > > > > authentication and in another case we require the reverse.
>> > > > >
>> > > >
>> > > > Since 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Hey Guys,

KIP-35

has been updated based on latest discussions and following PRs have also
been updated.
1. KAFKA-3307: Add ApiVersion request/response and server side handling.

2. KAFKA-3600: Enhance java clients to use ApiVersion Req/Resp to check if
the broker they are talking to supports required api versions.


If there are no major objections or changes suggested, we can start a vote
thread in a couple of days.

On Tue, Apr 12, 2016 at 8:04 AM, Jun Rao  wrote:

> Hi, Ismael,
>
> The SASL engine that we used is the SASL library, right? How did the C
> client generate those SASL tokens? Once a SASL mechanism is chosen, the
> subsequent tokens are determined, right? So, my feeling is that those
> tokens are part of SaslHandshakeRequest and are just extended across
> multiple network packets. So modeling those as independent requests feels
> weird. When documentation them, we really need to document those as a
> sequence, not individual isolated requests that can be issued
> in arbitrary order. The version id will only add confusion since we can't
> version the tokens independently. We could explicitly add the client id and
> correlation id in the header, but I am not sure if it's worth the
> additional complexity.
>
> Thanks,
>
> Jun
>
> On Tue, Apr 12, 2016 at 1:18 AM, Ismael Juma  wrote:
>
> > Hi Jun,
> >
> > I understand the point about the SASL tokens being similar to the SSL
> > handshake in a way. However, is there any SASL library that handles the
> > network communication for these tokens? I couldn't find any and without
> > that, there isn't much benefit in deviating from Kafka's protocol (we
> > basically save the space taken by the request header). It's worth
> > mentioning that we are already adding the message size before the opaque
> > bytes provided by the library, so one could say we are already extending
> > the protocol.
> >
> > If we leave versioning aside, adding the standard Kafka request header to
> > those messages may also help from a debugging perspective as would then
> > include client id and correlation id along with the message.
> >
> > Ismael
> >
> > On Tue, Apr 12, 2016 at 2:13 AM, Jun Rao  wrote:
> >
> > > Magnus,
> > >
> > > That sounds reasonable. To reduce the changes on the server side, I'd
> > > suggest the following minor tweaks on the proposal.
> > >
> > > 1. Continue supporting the separate SASL and SASL_SSL port.
> > >
> > > On SASL port, we support the new sequence
> > > ApiVersionRequest (optional), SaslHandshakeRequest, SASL tokens,
> > > regular
> > > requests
> > >
> > > On SASL_SSL port, we support the new sequence
> > > SSL handshake bytes, ApiVersionRequest (optional),
> > > SaslHandshakeRequest,
> > > SASL tokens, regular requests
> > >
> > > 2. We don't wrap SASL tokens in Kafka protocol. Similar to your
> argument
> > > about SSL handshake, those SASL tokens are generated by SASL library
> and
> > > Kafka doesn't really control its versioning. Kafka only controls the
> > > selection of SASL mechanism, which will be versioned in
> > > SaslHandshakeRequest.
> > >
> > > Does that work for you?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Apr 11, 2016 at 11:15 AM, Magnus Edenhill 
> > > wrote:
> > >
> > > > Hey Jun, see inline
> > > >
> > > > 2016-04-11 19:19 GMT+02:00 Jun Rao :
> > > >
> > > > > Hi, Magnus,
> > > > >
> > > > > Let me understand your proposal in more details just from the
> > client's
> > > > > perspective. My understanding of your proposal is the following.
> > > > >
> > > > > On plaintext port, the client will send the following bytes in
> order.
> > > > > ApiVersionRequest, SaslHandshakeRequest, SASL tokens (if SASL
> is
> > > > > enabled), regular requests
> > > > >
> > > > > On SSL port, the client will send the following bytes in order.
> > > > > SSL handshake bytes, ApiVersionRequest, SaslHandshakeRequest,
> > SASL
> > > > > tokens (if SASL is enabled), regular requests
> > > > >
> > > >
> > > >
> > > > Yup!
> > > > "SASL tokens" is a series of proper Kafka protocol
> > SaslHandshakeRequests
> > > > until
> > > > the handshake is done.
> > > >
> > > >
> > > >
> > > > >
> > > > > Is that right? Since we can use either SSL or SASL for
> > authentication,
> > > > it's
> > > > > weird that in one case, we require ApiVersionRequest to happen
> before
> > > > > authentication and in another case we require the reverse.
> > > > >
> > > >
> > > > Since the SSL/TLS is standardised and taken care of for us by the SSL
> > > > libraries it
> > > > doesnt make sense to reimplement that on top of Kafka, so it isn't
> > really
> > > > comparable.
> > > > But for SASL there is no standardised handshake protocol so we must
> > > either
> > > > 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jun Rao]

Hi, Ismael,

The SASL engine that we used is the SASL library, right? How did the C
client generate those SASL tokens? Once a SASL mechanism is chosen, the
subsequent tokens are determined, right? So, my feeling is that those
tokens are part of SaslHandshakeRequest and are just extended across
multiple network packets. So modeling those as independent requests feels
weird. When documentation them, we really need to document those as a
sequence, not individual isolated requests that can be issued
in arbitrary order. The version id will only add confusion since we can't
version the tokens independently. We could explicitly add the client id and
correlation id in the header, but I am not sure if it's worth the
additional complexity.

Thanks,

Jun

On Tue, Apr 12, 2016 at 1:18 AM, Ismael Juma  wrote:

> Hi Jun,
>
> I understand the point about the SASL tokens being similar to the SSL
> handshake in a way. However, is there any SASL library that handles the
> network communication for these tokens? I couldn't find any and without
> that, there isn't much benefit in deviating from Kafka's protocol (we
> basically save the space taken by the request header). It's worth
> mentioning that we are already adding the message size before the opaque
> bytes provided by the library, so one could say we are already extending
> the protocol.
>
> If we leave versioning aside, adding the standard Kafka request header to
> those messages may also help from a debugging perspective as would then
> include client id and correlation id along with the message.
>
> Ismael
>
> On Tue, Apr 12, 2016 at 2:13 AM, Jun Rao  wrote:
>
> > Magnus,
> >
> > That sounds reasonable. To reduce the changes on the server side, I'd
> > suggest the following minor tweaks on the proposal.
> >
> > 1. Continue supporting the separate SASL and SASL_SSL port.
> >
> > On SASL port, we support the new sequence
> > ApiVersionRequest (optional), SaslHandshakeRequest, SASL tokens,
> > regular
> > requests
> >
> > On SASL_SSL port, we support the new sequence
> > SSL handshake bytes, ApiVersionRequest (optional),
> > SaslHandshakeRequest,
> > SASL tokens, regular requests
> >
> > 2. We don't wrap SASL tokens in Kafka protocol. Similar to your argument
> > about SSL handshake, those SASL tokens are generated by SASL library and
> > Kafka doesn't really control its versioning. Kafka only controls the
> > selection of SASL mechanism, which will be versioned in
> > SaslHandshakeRequest.
> >
> > Does that work for you?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Apr 11, 2016 at 11:15 AM, Magnus Edenhill 
> > wrote:
> >
> > > Hey Jun, see inline
> > >
> > > 2016-04-11 19:19 GMT+02:00 Jun Rao :
> > >
> > > > Hi, Magnus,
> > > >
> > > > Let me understand your proposal in more details just from the
> client's
> > > > perspective. My understanding of your proposal is the following.
> > > >
> > > > On plaintext port, the client will send the following bytes in order.
> > > > ApiVersionRequest, SaslHandshakeRequest, SASL tokens (if SASL is
> > > > enabled), regular requests
> > > >
> > > > On SSL port, the client will send the following bytes in order.
> > > > SSL handshake bytes, ApiVersionRequest, SaslHandshakeRequest,
> SASL
> > > > tokens (if SASL is enabled), regular requests
> > > >
> > >
> > >
> > > Yup!
> > > "SASL tokens" is a series of proper Kafka protocol
> SaslHandshakeRequests
> > > until
> > > the handshake is done.
> > >
> > >
> > >
> > > >
> > > > Is that right? Since we can use either SSL or SASL for
> authentication,
> > > it's
> > > > weird that in one case, we require ApiVersionRequest to happen before
> > > > authentication and in another case we require the reverse.
> > > >
> > >
> > > Since the SSL/TLS is standardised and taken care of for us by the SSL
> > > libraries it
> > > doesnt make sense to reimplement that on top of Kafka, so it isn't
> really
> > > comparable.
> > > But for SASL there is no standardised handshake protocol so we must
> > either
> > > conceive one from scratch, or use the protocol that we already have
> > > (Kafka).
> > > For the initial SASL implementation in 0.9 the first option was chosen
> > and
> > > while
> > > it required a new protocol implementation in supporting clients and the
> > > broker
> > > it served its purpose. But not for long,  it already needs to evolve,
> > > and this gives us a golden[1] opportunity to make the implementation
> > > reusable, evolvable, less complex
> > > and in line with all our other protocol work, by using the  protocol
> > stack
> > > of Kafka which the
> > > broker and all clients already have in place.
> > >
> > > Not taking this chance and instead diverging the custom SASL handshake
> > > protocol
> > > even further from Kafka seems to me a weird choice.
> > >
> > > The current KIP-43 proposal does not have a clear compatibility story;
> it
> > > doesnt seem to be possible
> > > to 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

Hi Rajini,

Yes, I agree that it's not ideal. However, doing this once at the broker is
more manageable than pushing the additional complexity to the clients.
Between the two options you outlined, the second one seemed the least bad
(we do something similar for controlled shutdown because it was missing a
header before 0.9.0.0).

Ismael

On Tue, Apr 12, 2016 at 9:56 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Ismael,
>
> My only concern about wrapping SASL tokens in Kafka headers is backward
> compatibility. We would either have a different format for GSSAPI alone to
> match 0.9.0.x or we would need to support two different wire protocols for
> GSSAPI. Neither sounds ideal.
>
> On Tue, Apr 12, 2016 at 9:18 AM, Ismael Juma  wrote:
>
> > Hi Jun,
> >
> > I understand the point about the SASL tokens being similar to the SSL
> > handshake in a way. However, is there any SASL library that handles the
> > network communication for these tokens? I couldn't find any and without
> > that, there isn't much benefit in deviating from Kafka's protocol (we
> > basically save the space taken by the request header). It's worth
> > mentioning that we are already adding the message size before the opaque
> > bytes provided by the library, so one could say we are already extending
> > the protocol.
> >
> > If we leave versioning aside, adding the standard Kafka request header to
> > those messages may also help from a debugging perspective as would then
> > include client id and correlation id along with the message.
> >
> > Ismael
> >
> > On Tue, Apr 12, 2016 at 2:13 AM, Jun Rao  wrote:
> >
> > > Magnus,
> > >
> > > That sounds reasonable. To reduce the changes on the server side, I'd
> > > suggest the following minor tweaks on the proposal.
> > >
> > > 1. Continue supporting the separate SASL and SASL_SSL port.
> > >
> > > On SASL port, we support the new sequence
> > > ApiVersionRequest (optional), SaslHandshakeRequest, SASL tokens,
> > > regular
> > > requests
> > >
> > > On SASL_SSL port, we support the new sequence
> > > SSL handshake bytes, ApiVersionRequest (optional),
> > > SaslHandshakeRequest,
> > > SASL tokens, regular requests
> > >
> > > 2. We don't wrap SASL tokens in Kafka protocol. Similar to your
> argument
> > > about SSL handshake, those SASL tokens are generated by SASL library
> and
> > > Kafka doesn't really control its versioning. Kafka only controls the
> > > selection of SASL mechanism, which will be versioned in
> > > SaslHandshakeRequest.
> > >
> > > Does that work for you?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Apr 11, 2016 at 11:15 AM, Magnus Edenhill 
> > > wrote:
> > >
> > > > Hey Jun, see inline
> > > >
> > > > 2016-04-11 19:19 GMT+02:00 Jun Rao :
> > > >
> > > > > Hi, Magnus,
> > > > >
> > > > > Let me understand your proposal in more details just from the
> > client's
> > > > > perspective. My understanding of your proposal is the following.
> > > > >
> > > > > On plaintext port, the client will send the following bytes in
> order.
> > > > > ApiVersionRequest, SaslHandshakeRequest, SASL tokens (if SASL
> is
> > > > > enabled), regular requests
> > > > >
> > > > > On SSL port, the client will send the following bytes in order.
> > > > > SSL handshake bytes, ApiVersionRequest, SaslHandshakeRequest,
> > SASL
> > > > > tokens (if SASL is enabled), regular requests
> > > > >
> > > >
> > > >
> > > > Yup!
> > > > "SASL tokens" is a series of proper Kafka protocol
> > SaslHandshakeRequests
> > > > until
> > > > the handshake is done.
> > > >
> > > >
> > > >
> > > > >
> > > > > Is that right? Since we can use either SSL or SASL for
> > authentication,
> > > > it's
> > > > > weird that in one case, we require ApiVersionRequest to happen
> before
> > > > > authentication and in another case we require the reverse.
> > > > >
> > > >
> > > > Since the SSL/TLS is standardised and taken care of for us by the SSL
> > > > libraries it
> > > > doesnt make sense to reimplement that on top of Kafka, so it isn't
> > really
> > > > comparable.
> > > > But for SASL there is no standardised handshake protocol so we must
> > > either
> > > > conceive one from scratch, or use the protocol that we already have
> > > > (Kafka).
> > > > For the initial SASL implementation in 0.9 the first option was
> chosen
> > > and
> > > > while
> > > > it required a new protocol implementation in supporting clients and
> the
> > > > broker
> > > > it served its purpose. But not for long,  it already needs to evolve,
> > > > and this gives us a golden[1] opportunity to make the implementation
> > > > reusable, evolvable, less complex
> > > > and in line with all our other protocol work, by using the  protocol
> > > stack
> > > > of Kafka which the
> > > > broker and all clients already have in place.
> > > >
> > > > Not taking this chance and instead diverging the custom SASL
> handshake
> > > > 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Rajini Sivaram]

Ismael,

My only concern about wrapping SASL tokens in Kafka headers is backward
compatibility. We would either have a different format for GSSAPI alone to
match 0.9.0.x or we would need to support two different wire protocols for
GSSAPI. Neither sounds ideal.

On Tue, Apr 12, 2016 at 9:18 AM, Ismael Juma  wrote:

> Hi Jun,
>
> I understand the point about the SASL tokens being similar to the SSL
> handshake in a way. However, is there any SASL library that handles the
> network communication for these tokens? I couldn't find any and without
> that, there isn't much benefit in deviating from Kafka's protocol (we
> basically save the space taken by the request header). It's worth
> mentioning that we are already adding the message size before the opaque
> bytes provided by the library, so one could say we are already extending
> the protocol.
>
> If we leave versioning aside, adding the standard Kafka request header to
> those messages may also help from a debugging perspective as would then
> include client id and correlation id along with the message.
>
> Ismael
>
> On Tue, Apr 12, 2016 at 2:13 AM, Jun Rao  wrote:
>
> > Magnus,
> >
> > That sounds reasonable. To reduce the changes on the server side, I'd
> > suggest the following minor tweaks on the proposal.
> >
> > 1. Continue supporting the separate SASL and SASL_SSL port.
> >
> > On SASL port, we support the new sequence
> > ApiVersionRequest (optional), SaslHandshakeRequest, SASL tokens,
> > regular
> > requests
> >
> > On SASL_SSL port, we support the new sequence
> > SSL handshake bytes, ApiVersionRequest (optional),
> > SaslHandshakeRequest,
> > SASL tokens, regular requests
> >
> > 2. We don't wrap SASL tokens in Kafka protocol. Similar to your argument
> > about SSL handshake, those SASL tokens are generated by SASL library and
> > Kafka doesn't really control its versioning. Kafka only controls the
> > selection of SASL mechanism, which will be versioned in
> > SaslHandshakeRequest.
> >
> > Does that work for you?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Apr 11, 2016 at 11:15 AM, Magnus Edenhill 
> > wrote:
> >
> > > Hey Jun, see inline
> > >
> > > 2016-04-11 19:19 GMT+02:00 Jun Rao :
> > >
> > > > Hi, Magnus,
> > > >
> > > > Let me understand your proposal in more details just from the
> client's
> > > > perspective. My understanding of your proposal is the following.
> > > >
> > > > On plaintext port, the client will send the following bytes in order.
> > > > ApiVersionRequest, SaslHandshakeRequest, SASL tokens (if SASL is
> > > > enabled), regular requests
> > > >
> > > > On SSL port, the client will send the following bytes in order.
> > > > SSL handshake bytes, ApiVersionRequest, SaslHandshakeRequest,
> SASL
> > > > tokens (if SASL is enabled), regular requests
> > > >
> > >
> > >
> > > Yup!
> > > "SASL tokens" is a series of proper Kafka protocol
> SaslHandshakeRequests
> > > until
> > > the handshake is done.
> > >
> > >
> > >
> > > >
> > > > Is that right? Since we can use either SSL or SASL for
> authentication,
> > > it's
> > > > weird that in one case, we require ApiVersionRequest to happen before
> > > > authentication and in another case we require the reverse.
> > > >
> > >
> > > Since the SSL/TLS is standardised and taken care of for us by the SSL
> > > libraries it
> > > doesnt make sense to reimplement that on top of Kafka, so it isn't
> really
> > > comparable.
> > > But for SASL there is no standardised handshake protocol so we must
> > either
> > > conceive one from scratch, or use the protocol that we already have
> > > (Kafka).
> > > For the initial SASL implementation in 0.9 the first option was chosen
> > and
> > > while
> > > it required a new protocol implementation in supporting clients and the
> > > broker
> > > it served its purpose. But not for long,  it already needs to evolve,
> > > and this gives us a golden[1] opportunity to make the implementation
> > > reusable, evolvable, less complex
> > > and in line with all our other protocol work, by using the  protocol
> > stack
> > > of Kafka which the
> > > broker and all clients already have in place.
> > >
> > > Not taking this chance and instead diverging the custom SASL handshake
> > > protocol
> > > even further from Kafka seems to me a weird choice.
> > >
> > > The current KIP-43 proposal does not have a clear compatibility story;
> it
> > > doesnt seem to be possible
> > > to upgrade clients before brokers, while this might be okay for the
> Java
> > > client, the KIP-35 discussion
> > > has hopefully proven that this assumption can't be made for the entire
> > > eco-system.
> > >
> > > Let me be clear that there isn't anything technically wrong with the
> > KIP-43
> > > proposal (well,
> > > except for the hack to check byte[0] for 0x60 perhaps), but I'm worried
> > the
> > > proposal will eventually lead to
> > > reimplementing Api Versioning, KIP-35, etc, in 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

Hi Jun,

I understand the point about the SASL tokens being similar to the SSL
handshake in a way. However, is there any SASL library that handles the
network communication for these tokens? I couldn't find any and without
that, there isn't much benefit in deviating from Kafka's protocol (we
basically save the space taken by the request header). It's worth
mentioning that we are already adding the message size before the opaque
bytes provided by the library, so one could say we are already extending
the protocol.

If we leave versioning aside, adding the standard Kafka request header to
those messages may also help from a debugging perspective as would then
include client id and correlation id along with the message.

Ismael

On Tue, Apr 12, 2016 at 2:13 AM, Jun Rao  wrote:

> Magnus,
>
> That sounds reasonable. To reduce the changes on the server side, I'd
> suggest the following minor tweaks on the proposal.
>
> 1. Continue supporting the separate SASL and SASL_SSL port.
>
> On SASL port, we support the new sequence
> ApiVersionRequest (optional), SaslHandshakeRequest, SASL tokens,
> regular
> requests
>
> On SASL_SSL port, we support the new sequence
> SSL handshake bytes, ApiVersionRequest (optional),
> SaslHandshakeRequest,
> SASL tokens, regular requests
>
> 2. We don't wrap SASL tokens in Kafka protocol. Similar to your argument
> about SSL handshake, those SASL tokens are generated by SASL library and
> Kafka doesn't really control its versioning. Kafka only controls the
> selection of SASL mechanism, which will be versioned in
> SaslHandshakeRequest.
>
> Does that work for you?
>
> Thanks,
>
> Jun
>
>
> On Mon, Apr 11, 2016 at 11:15 AM, Magnus Edenhill 
> wrote:
>
> > Hey Jun, see inline
> >
> > 2016-04-11 19:19 GMT+02:00 Jun Rao :
> >
> > > Hi, Magnus,
> > >
> > > Let me understand your proposal in more details just from the client's
> > > perspective. My understanding of your proposal is the following.
> > >
> > > On plaintext port, the client will send the following bytes in order.
> > > ApiVersionRequest, SaslHandshakeRequest, SASL tokens (if SASL is
> > > enabled), regular requests
> > >
> > > On SSL port, the client will send the following bytes in order.
> > > SSL handshake bytes, ApiVersionRequest, SaslHandshakeRequest, SASL
> > > tokens (if SASL is enabled), regular requests
> > >
> >
> >
> > Yup!
> > "SASL tokens" is a series of proper Kafka protocol SaslHandshakeRequests
> > until
> > the handshake is done.
> >
> >
> >
> > >
> > > Is that right? Since we can use either SSL or SASL for authentication,
> > it's
> > > weird that in one case, we require ApiVersionRequest to happen before
> > > authentication and in another case we require the reverse.
> > >
> >
> > Since the SSL/TLS is standardised and taken care of for us by the SSL
> > libraries it
> > doesnt make sense to reimplement that on top of Kafka, so it isn't really
> > comparable.
> > But for SASL there is no standardised handshake protocol so we must
> either
> > conceive one from scratch, or use the protocol that we already have
> > (Kafka).
> > For the initial SASL implementation in 0.9 the first option was chosen
> and
> > while
> > it required a new protocol implementation in supporting clients and the
> > broker
> > it served its purpose. But not for long,  it already needs to evolve,
> > and this gives us a golden[1] opportunity to make the implementation
> > reusable, evolvable, less complex
> > and in line with all our other protocol work, by using the  protocol
> stack
> > of Kafka which the
> > broker and all clients already have in place.
> >
> > Not taking this chance and instead diverging the custom SASL handshake
> > protocol
> > even further from Kafka seems to me a weird choice.
> >
> > The current KIP-43 proposal does not have a clear compatibility story; it
> > doesnt seem to be possible
> > to upgrade clients before brokers, while this might be okay for the Java
> > client, the KIP-35 discussion
> > has hopefully proven that this assumption can't be made for the entire
> > eco-system.
> >
> > Let me be clear that there isn't anything technically wrong with the
> KIP-43
> > proposal (well,
> > except for the hack to check byte[0] for 0x60 perhaps), but I'm worried
> the
> > proposal will eventually lead to
> > reimplementing Api Versioning, KIP-35, etc, in the custom SASL handshake,
> > and this is just redundant,
> > there is no technical reason for doing so and it'll just make protocol
> > semantics and implementations more complex.
> >
> >
> > Regards,
> > Magnus
> >
> > [1]: Timing is good for this change since only two clients, Java and C,
> > currently supports
> > the existing SASL handshake so far.
> >
> >
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Apr 11, 2016 at 12:20 AM, Magnus Edenhill 
> > > wrote:
> > >
> > > > 2016-04-11 3:01 GMT+02:00 Jun Rao :
> > > >
> > > > > 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Rajini Sivaram]

Jun's tweaked proposal sounds good to me. In terms of completing KIP-43,
this changes the format of the request-response for exchanging mechanisms,
but not the overall logic. Since the request format in KIP-43 is worth
changing anyway, I will update the KIP and the PR.

On Tue, Apr 12, 2016 at 2:13 AM, Jun Rao  wrote:

> Magnus,
>
> That sounds reasonable. To reduce the changes on the server side, I'd
> suggest the following minor tweaks on the proposal.
>
> 1. Continue supporting the separate SASL and SASL_SSL port.
>
> On SASL port, we support the new sequence
> ApiVersionRequest (optional), SaslHandshakeRequest, SASL tokens,
> regular
> requests
>
> On SASL_SSL port, we support the new sequence
> SSL handshake bytes, ApiVersionRequest (optional),
> SaslHandshakeRequest,
> SASL tokens, regular requests
>
> 2. We don't wrap SASL tokens in Kafka protocol. Similar to your argument
> about SSL handshake, those SASL tokens are generated by SASL library and
> Kafka doesn't really control its versioning. Kafka only controls the
> selection of SASL mechanism, which will be versioned in
> SaslHandshakeRequest.
>
> Does that work for you?
>
> Thanks,
>
> Jun
>
>
> On Mon, Apr 11, 2016 at 11:15 AM, Magnus Edenhill 
> wrote:
>
> > Hey Jun, see inline
> >
> > 2016-04-11 19:19 GMT+02:00 Jun Rao :
> >
> > > Hi, Magnus,
> > >
> > > Let me understand your proposal in more details just from the client's
> > > perspective. My understanding of your proposal is the following.
> > >
> > > On plaintext port, the client will send the following bytes in order.
> > > ApiVersionRequest, SaslHandshakeRequest, SASL tokens (if SASL is
> > > enabled), regular requests
> > >
> > > On SSL port, the client will send the following bytes in order.
> > > SSL handshake bytes, ApiVersionRequest, SaslHandshakeRequest, SASL
> > > tokens (if SASL is enabled), regular requests
> > >
> >
> >
> > Yup!
> > "SASL tokens" is a series of proper Kafka protocol SaslHandshakeRequests
> > until
> > the handshake is done.
> >
> >
> >
> > >
> > > Is that right? Since we can use either SSL or SASL for authentication,
> > it's
> > > weird that in one case, we require ApiVersionRequest to happen before
> > > authentication and in another case we require the reverse.
> > >
> >
> > Since the SSL/TLS is standardised and taken care of for us by the SSL
> > libraries it
> > doesnt make sense to reimplement that on top of Kafka, so it isn't really
> > comparable.
> > But for SASL there is no standardised handshake protocol so we must
> either
> > conceive one from scratch, or use the protocol that we already have
> > (Kafka).
> > For the initial SASL implementation in 0.9 the first option was chosen
> and
> > while
> > it required a new protocol implementation in supporting clients and the
> > broker
> > it served its purpose. But not for long,  it already needs to evolve,
> > and this gives us a golden[1] opportunity to make the implementation
> > reusable, evolvable, less complex
> > and in line with all our other protocol work, by using the  protocol
> stack
> > of Kafka which the
> > broker and all clients already have in place.
> >
> > Not taking this chance and instead diverging the custom SASL handshake
> > protocol
> > even further from Kafka seems to me a weird choice.
> >
> > The current KIP-43 proposal does not have a clear compatibility story; it
> > doesnt seem to be possible
> > to upgrade clients before brokers, while this might be okay for the Java
> > client, the KIP-35 discussion
> > has hopefully proven that this assumption can't be made for the entire
> > eco-system.
> >
> > Let me be clear that there isn't anything technically wrong with the
> KIP-43
> > proposal (well,
> > except for the hack to check byte[0] for 0x60 perhaps), but I'm worried
> the
> > proposal will eventually lead to
> > reimplementing Api Versioning, KIP-35, etc, in the custom SASL handshake,
> > and this is just redundant,
> > there is no technical reason for doing so and it'll just make protocol
> > semantics and implementations more complex.
> >
> >
> > Regards,
> > Magnus
> >
> > [1]: Timing is good for this change since only two clients, Java and C,
> > currently supports
> > the existing SASL handshake so far.
> >
> >
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Apr 11, 2016 at 12:20 AM, Magnus Edenhill 
> > > wrote:
> > >
> > > > 2016-04-11 3:01 GMT+02:00 Jun Rao :
> > > >
> > > > > Thinking about ApiVersionRequest a bit more. There are quite a few
> > > things
> > > > > special about it. In the ideal case, (1) its version should never
> > > change;
> > > > >
> > > >
> > > > The only thing we know of the future is that we dont know anything,
> we
> > > > can't
> > > > think of every possible future use case, that's why need to be able
> to
> > > > evolve interfaces
> > > > as requirements and use-cases change. This is the gist of KIP-35, and
> > 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jun Rao]

Magnus,

That sounds reasonable. To reduce the changes on the server side, I'd
suggest the following minor tweaks on the proposal.

1. Continue supporting the separate SASL and SASL_SSL port.

On SASL port, we support the new sequence
ApiVersionRequest (optional), SaslHandshakeRequest, SASL tokens, regular
requests

On SASL_SSL port, we support the new sequence
SSL handshake bytes, ApiVersionRequest (optional), SaslHandshakeRequest,
SASL tokens, regular requests

2. We don't wrap SASL tokens in Kafka protocol. Similar to your argument
about SSL handshake, those SASL tokens are generated by SASL library and
Kafka doesn't really control its versioning. Kafka only controls the
selection of SASL mechanism, which will be versioned in
SaslHandshakeRequest.

Does that work for you?

Thanks,

Jun


On Mon, Apr 11, 2016 at 11:15 AM, Magnus Edenhill 
wrote:

> Hey Jun, see inline
>
> 2016-04-11 19:19 GMT+02:00 Jun Rao :
>
> > Hi, Magnus,
> >
> > Let me understand your proposal in more details just from the client's
> > perspective. My understanding of your proposal is the following.
> >
> > On plaintext port, the client will send the following bytes in order.
> > ApiVersionRequest, SaslHandshakeRequest, SASL tokens (if SASL is
> > enabled), regular requests
> >
> > On SSL port, the client will send the following bytes in order.
> > SSL handshake bytes, ApiVersionRequest, SaslHandshakeRequest, SASL
> > tokens (if SASL is enabled), regular requests
> >
>
>
> Yup!
> "SASL tokens" is a series of proper Kafka protocol SaslHandshakeRequests
> until
> the handshake is done.
>
>
>
> >
> > Is that right? Since we can use either SSL or SASL for authentication,
> it's
> > weird that in one case, we require ApiVersionRequest to happen before
> > authentication and in another case we require the reverse.
> >
>
> Since the SSL/TLS is standardised and taken care of for us by the SSL
> libraries it
> doesnt make sense to reimplement that on top of Kafka, so it isn't really
> comparable.
> But for SASL there is no standardised handshake protocol so we must either
> conceive one from scratch, or use the protocol that we already have
> (Kafka).
> For the initial SASL implementation in 0.9 the first option was chosen and
> while
> it required a new protocol implementation in supporting clients and the
> broker
> it served its purpose. But not for long,  it already needs to evolve,
> and this gives us a golden[1] opportunity to make the implementation
> reusable, evolvable, less complex
> and in line with all our other protocol work, by using the  protocol stack
> of Kafka which the
> broker and all clients already have in place.
>
> Not taking this chance and instead diverging the custom SASL handshake
> protocol
> even further from Kafka seems to me a weird choice.
>
> The current KIP-43 proposal does not have a clear compatibility story; it
> doesnt seem to be possible
> to upgrade clients before brokers, while this might be okay for the Java
> client, the KIP-35 discussion
> has hopefully proven that this assumption can't be made for the entire
> eco-system.
>
> Let me be clear that there isn't anything technically wrong with the KIP-43
> proposal (well,
> except for the hack to check byte[0] for 0x60 perhaps), but I'm worried the
> proposal will eventually lead to
> reimplementing Api Versioning, KIP-35, etc, in the custom SASL handshake,
> and this is just redundant,
> there is no technical reason for doing so and it'll just make protocol
> semantics and implementations more complex.
>
>
> Regards,
> Magnus
>
> [1]: Timing is good for this change since only two clients, Java and C,
> currently supports
> the existing SASL handshake so far.
>
>
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Apr 11, 2016 at 12:20 AM, Magnus Edenhill 
> > wrote:
> >
> > > 2016-04-11 3:01 GMT+02:00 Jun Rao :
> > >
> > > > Thinking about ApiVersionRequest a bit more. There are quite a few
> > things
> > > > special about it. In the ideal case, (1) its version should never
> > change;
> > > >
> > >
> > > The only thing we know of the future is that we dont know anything, we
> > > can't
> > > think of every possible future use case, that's why need to be able to
> > > evolve interfaces
> > > as requirements and use-cases change. This is the gist of KIP-35, and
> > > hampering
> > > KIP-35 itself, by not letting it also evolve, does not seem right to me
> > at
> > > all.
> > >
> > >
> > >
> > >
> > > > (2) it needs to be done before authentication (either SSL/SASL); (3)
> it
> > > is
> > > > required to be issued at the beginning of each connection but never
> > needs
> > > > to be issued again on the same connection. So, instead of modeling it
> > as
> > > a
> > > > regular request, it seems a cleaner approach is to just bake that
> into
> > > the
> > > > initial connection handshake even before the authentication layer. So
> > the
> > > > sequencing in a 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Magnus Edenhill]

Hey Jun, see inline

2016-04-11 19:19 GMT+02:00 Jun Rao :

> Hi, Magnus,
>
> Let me understand your proposal in more details just from the client's
> perspective. My understanding of your proposal is the following.
>
> On plaintext port, the client will send the following bytes in order.
> ApiVersionRequest, SaslHandshakeRequest, SASL tokens (if SASL is
> enabled), regular requests
>
> On SSL port, the client will send the following bytes in order.
> SSL handshake bytes, ApiVersionRequest, SaslHandshakeRequest, SASL
> tokens (if SASL is enabled), regular requests
>


Yup!
"SASL tokens" is a series of proper Kafka protocol SaslHandshakeRequests
until
the handshake is done.



>
> Is that right? Since we can use either SSL or SASL for authentication, it's
> weird that in one case, we require ApiVersionRequest to happen before
> authentication and in another case we require the reverse.
>

Since the SSL/TLS is standardised and taken care of for us by the SSL
libraries it
doesnt make sense to reimplement that on top of Kafka, so it isn't really
comparable.
But for SASL there is no standardised handshake protocol so we must either
conceive one from scratch, or use the protocol that we already have (Kafka).
For the initial SASL implementation in 0.9 the first option was chosen and
while
it required a new protocol implementation in supporting clients and the
broker
it served its purpose. But not for long,  it already needs to evolve,
and this gives us a golden[1] opportunity to make the implementation
reusable, evolvable, less complex
and in line with all our other protocol work, by using the  protocol stack
of Kafka which the
broker and all clients already have in place.

Not taking this chance and instead diverging the custom SASL handshake
protocol
even further from Kafka seems to me a weird choice.

The current KIP-43 proposal does not have a clear compatibility story; it
doesnt seem to be possible
to upgrade clients before brokers, while this might be okay for the Java
client, the KIP-35 discussion
has hopefully proven that this assumption can't be made for the entire
eco-system.

Let me be clear that there isn't anything technically wrong with the KIP-43
proposal (well,
except for the hack to check byte[0] for 0x60 perhaps), but I'm worried the
proposal will eventually lead to
reimplementing Api Versioning, KIP-35, etc, in the custom SASL handshake,
and this is just redundant,
there is no technical reason for doing so and it'll just make protocol
semantics and implementations more complex.


Regards,
Magnus

[1]: Timing is good for this change since only two clients, Java and C,
currently supports
the existing SASL handshake so far.


>
> Thanks,
>
> Jun
>
> On Mon, Apr 11, 2016 at 12:20 AM, Magnus Edenhill 
> wrote:
>
> > 2016-04-11 3:01 GMT+02:00 Jun Rao :
> >
> > > Thinking about ApiVersionRequest a bit more. There are quite a few
> things
> > > special about it. In the ideal case, (1) its version should never
> change;
> > >
> >
> > The only thing we know of the future is that we dont know anything, we
> > can't
> > think of every possible future use case, that's why need to be able to
> > evolve interfaces
> > as requirements and use-cases change. This is the gist of KIP-35, and
> > hampering
> > KIP-35 itself, by not letting it also evolve, does not seem right to me
> at
> > all.
> >
> >
> >
> >
> > > (2) it needs to be done before authentication (either SSL/SASL); (3) it
> > is
> > > required to be issued at the beginning of each connection but never
> needs
> > > to be issued again on the same connection. So, instead of modeling it
> as
> > a
> > > regular request, it seems a cleaner approach is to just bake that into
> > the
> > > initial connection handshake even before the authentication layer. So
> the
> > > sequencing in a connection will be api discovery, authentication,
> > followed
> > > by regular requests. I am not sure we can still add this in a backward
> > > compatible way now (e.g., not sure what the initial bytes from an SSL
> > > connection will look like). Even if we can do this in a backward
> > compatible
> > > way, it's probably non-trivial amount of work though.
> > >
> >
> > I have the luxory of not knowing the broker internals, so I can only
> > discuss
> > this on a conceptual design level.
> >
> > In its simplest form each API request type has a NeedsAuth flag and the
> > broker protocol request layer simply checks if the current session is
> > Authenticated
> > before processing the request: If not the session is closed and an error
> is
> > logged.
> > The only two API requests that dont have the NeedsAuth flag would be
> > SaslHandshakeRequest
> > and ApiVersionRequest, the latter could also use filtering to only return
> > the same two
> > requests in ApiVersionResponse before the client is authenticated (as not
> > to "leak" information).
> > If authentication is not configured on the broker all sessions 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jun Rao]

Hi, Magnus,

Let me understand your proposal in more details just from the client's
perspective. My understanding of your proposal is the following.

On plaintext port, the client will send the following bytes in order.
ApiVersionRequest, SaslHandshakeRequest, SASL tokens (if SASL is
enabled), regular requests

On SSL port, the client will send the following bytes in order.
SSL handshake bytes, ApiVersionRequest, SaslHandshakeRequest, SASL
tokens (if SASL is enabled), regular requests

Is that right? Since we can use either SSL or SASL for authentication, it's
weird that in one case, we require ApiVersionRequest to happen before
authentication and in another case we require the reverse.

Thanks,

Jun

On Mon, Apr 11, 2016 at 12:20 AM, Magnus Edenhill 
wrote:

> 2016-04-11 3:01 GMT+02:00 Jun Rao :
>
> > Thinking about ApiVersionRequest a bit more. There are quite a few things
> > special about it. In the ideal case, (1) its version should never change;
> >
>
> The only thing we know of the future is that we dont know anything, we
> can't
> think of every possible future use case, that's why need to be able to
> evolve interfaces
> as requirements and use-cases change. This is the gist of KIP-35, and
> hampering
> KIP-35 itself, by not letting it also evolve, does not seem right to me at
> all.
>
>
>
>
> > (2) it needs to be done before authentication (either SSL/SASL); (3) it
> is
> > required to be issued at the beginning of each connection but never needs
> > to be issued again on the same connection. So, instead of modeling it as
> a
> > regular request, it seems a cleaner approach is to just bake that into
> the
> > initial connection handshake even before the authentication layer. So the
> > sequencing in a connection will be api discovery, authentication,
> followed
> > by regular requests. I am not sure we can still add this in a backward
> > compatible way now (e.g., not sure what the initial bytes from an SSL
> > connection will look like). Even if we can do this in a backward
> compatible
> > way, it's probably non-trivial amount of work though.
> >
>
> I have the luxory of not knowing the broker internals, so I can only
> discuss
> this on a conceptual design level.
>
> In its simplest form each API request type has a NeedsAuth flag and the
> broker protocol request layer simply checks if the current session is
> Authenticated
> before processing the request: If not the session is closed and an error is
> logged.
> The only two API requests that dont have the NeedsAuth flag would be
> SaslHandshakeRequest
> and ApiVersionRequest, the latter could also use filtering to only return
> the same two
> requests in ApiVersionResponse before the client is authenticated (as not
> to "leak" information).
> If authentication is not configured on the broker all sessions are deemed
> authenticated by default.
>
>
> Re backwards compatibility:
> My suggestion is to keep the current special SASL handshake protocol on the
> SASL_PLAIN/SASL_SSL
> endpoints, but use the new in-band Kafka SaslHandshakeRequest API on the
> PLAIN/SSL endpoints.
> This way the broker is backwards compatible with older clients that only
> supports the special SASL protocol,
> and newer cliets are also backwards compatible with older brokers that only
> supports the special SASL protocol.
> Newer clients connecting to new brokers will be configured to use non-SASL
> ports and use the
> in-band Kafka SaslHandshakeRequest to authenticate.
>
> Using the existing standard Kafka protocol and the new future-proof
> functionality of ApiVersionRequest
> allows the in-band authentication mechanisms and semantics to naturally
> evolve over time
> without breaking existing clients.
>
>
>
>
>
> >
> > We started KIP-35 with supporting a client to know if a version is
> > supported by the broker. It's now evolved into supporting a client to
> > implement multiple versions of the protocol and dynamically pick a
> version
> > supported by the broker. The former is likely solvable without
> > ApiVersionRequest. How important is the latter? What if the C client just
> > follows the java client model by implementing one version of protocol
> per C
> > client release (which seems easier to implement)?
> >
>
> We've discussed this at length and it is not an option for librdkafka, nor
> kafka-python, and
> probably other clients as well, due to usability/UX and maintenance
> reasons.
> (There's even discussion of making the Java client more version agnostic!)
>
>
>
>
>
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Apr 8, 2016 at 4:20 PM, Jun Rao  wrote:
> >
> > > Magnus,
> > >
> > > A while back, we had another proposal for the broker to just send the
> > > correlation id and an empty payload if it receives an unsupported
> version
> > > of the request. I didn't see that in the rejected section. It seems
> > simpler
> > > than the current proposal where the client has to issue an
> > > 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Magnus Edenhill]

2016-04-11 3:01 GMT+02:00 Jun Rao :

> Thinking about ApiVersionRequest a bit more. There are quite a few things
> special about it. In the ideal case, (1) its version should never change;
>

The only thing we know of the future is that we dont know anything, we can't
think of every possible future use case, that's why need to be able to
evolve interfaces
as requirements and use-cases change. This is the gist of KIP-35, and
hampering
KIP-35 itself, by not letting it also evolve, does not seem right to me at
all.




> (2) it needs to be done before authentication (either SSL/SASL); (3) it is
> required to be issued at the beginning of each connection but never needs
> to be issued again on the same connection. So, instead of modeling it as a
> regular request, it seems a cleaner approach is to just bake that into the
> initial connection handshake even before the authentication layer. So the
> sequencing in a connection will be api discovery, authentication, followed
> by regular requests. I am not sure we can still add this in a backward
> compatible way now (e.g., not sure what the initial bytes from an SSL
> connection will look like). Even if we can do this in a backward compatible
> way, it's probably non-trivial amount of work though.
>

I have the luxory of not knowing the broker internals, so I can only discuss
this on a conceptual design level.

In its simplest form each API request type has a NeedsAuth flag and the
broker protocol request layer simply checks if the current session is
Authenticated
before processing the request: If not the session is closed and an error is
logged.
The only two API requests that dont have the NeedsAuth flag would be
SaslHandshakeRequest
and ApiVersionRequest, the latter could also use filtering to only return
the same two
requests in ApiVersionResponse before the client is authenticated (as not
to "leak" information).
If authentication is not configured on the broker all sessions are deemed
authenticated by default.


Re backwards compatibility:
My suggestion is to keep the current special SASL handshake protocol on the
SASL_PLAIN/SASL_SSL
endpoints, but use the new in-band Kafka SaslHandshakeRequest API on the
PLAIN/SSL endpoints.
This way the broker is backwards compatible with older clients that only
supports the special SASL protocol,
and newer cliets are also backwards compatible with older brokers that only
supports the special SASL protocol.
Newer clients connecting to new brokers will be configured to use non-SASL
ports and use the
in-band Kafka SaslHandshakeRequest to authenticate.

Using the existing standard Kafka protocol and the new future-proof
functionality of ApiVersionRequest
allows the in-band authentication mechanisms and semantics to naturally
evolve over time
without breaking existing clients.





>
> We started KIP-35 with supporting a client to know if a version is
> supported by the broker. It's now evolved into supporting a client to
> implement multiple versions of the protocol and dynamically pick a version
> supported by the broker. The former is likely solvable without
> ApiVersionRequest. How important is the latter? What if the C client just
> follows the java client model by implementing one version of protocol per C
> client release (which seems easier to implement)?
>

We've discussed this at length and it is not an option for librdkafka, nor
kafka-python, and
probably other clients as well, due to usability/UX and maintenance reasons.
(There's even discussion of making the Java client more version agnostic!)





>
> Thanks,
>
> Jun
>
>
> On Fri, Apr 8, 2016 at 4:20 PM, Jun Rao  wrote:
>
> > Magnus,
> >
> > A while back, we had another proposal for the broker to just send the
> > correlation id and an empty payload if it receives an unsupported version
> > of the request. I didn't see that in the rejected section. It seems
> simpler
> > than the current proposal where the client has to issue an
> > ApiVersionRequest on every connection. Could you document the reason why
> we
> > rejected it?
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Apr 5, 2016 at 1:47 PM, Ashish Singh 
> wrote:
> >
> >> On Fri, Apr 1, 2016 at 1:32 AM, Ismael Juma  wrote:
> >>
> >> > Two more things:
> >> >
> >> > 3. We talk about backporting of new request versions to stable
> branches
> >> in
> >> > the KIP. In practice, we can't do that until the Java client is
> changed
> >> so
> >> > that it doesn't blindly use the latest protocol version. Otherwise, if
> >> new
> >> > request versions were added to 0.9.0.2, the client would break when
> >> talking
> >> > to a 0.9.0.1 broker (given Jason's proposal, it would fail a bit more
> >> > gracefully, but that's still not good enough for a stable branch). It
> >> may
> >> > be worth making this clear in the KIP (yes, it is a bit orthogonal and
> >> > doesn't prevent the KIP from being adopted, but good to avoid
> >> confusion).
> >> >
> 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Magnus Edenhill]

Hi Jun,

the "send empty response instead of disconnect on unsupported protocol"
was actually a sub-proposal and was never meant as an alternative to the
proper feature detection proposed by KIP-35.


I've added it back to the list of rejected alternatives on the wiki page,
thanks
for pointing this out. There's an example and motivation on there too.

/Magnus


2016-04-09 1:20 GMT+02:00 Jun Rao :

> Magnus,
>
> A while back, we had another proposal for the broker to just send the
> correlation id and an empty payload if it receives an unsupported version
> of the request. I didn't see that in the rejected section. It seems simpler
> than the current proposal where the client has to issue an
> ApiVersionRequest on every connection. Could you document the reason why we
> rejected it?
>
> Thanks,
>
> Jun
>
> On Tue, Apr 5, 2016 at 1:47 PM, Ashish Singh  wrote:
>
> > On Fri, Apr 1, 2016 at 1:32 AM, Ismael Juma  wrote:
> >
> > > Two more things:
> > >
> > > 3. We talk about backporting of new request versions to stable branches
> > in
> > > the KIP. In practice, we can't do that until the Java client is changed
> > so
> > > that it doesn't blindly use the latest protocol version. Otherwise, if
> > new
> > > request versions were added to 0.9.0.2, the client would break when
> > talking
> > > to a 0.9.0.1 broker (given Jason's proposal, it would fail a bit more
> > > gracefully, but that's still not good enough for a stable branch). It
> may
> > > be worth making this clear in the KIP (yes, it is a bit orthogonal and
> > > doesn't prevent the KIP from being adopted, but good to avoid
> confusion).
> > >
> > Good point. Adding this note and also adding a note that Kafka has not
> > backported an api version so far.
> >
> > >
> > > 4. The paragraph below is a bit confusing. It starts talking about
> 0.9.0
> > > and trunk and then switches to 0.9.1. Is that intentional?
> > >
> > Yes.
> >
> > >
> > > "Deprecation of a protocol version will be done by marking a protocol
> > > version as deprecated in protocol documentation. Documentation shall
> also
> > > be used to indicate a protocol version that must not be used, or for
> any
> > > such information.For instance, say 0.9.0 had protocol versions [0] for
> > api
> > > key 1. On trunk, version 1 of the api key was added. Users running off
> > > trunk started using version 1 of the api and found out a major bug. To
> > > rectify that version 2 of the api is added to trunk. For some reason,
> it
> > is
> > > now deemed important to have version 2 of the api in 0.9.1 as well. To
> do
> > > so, version 1 and version 2 both of the api will be backported to the
> > 0.9.1
> > > branch. 0.9.1 broker will return 0 as min supported version for the api
> > and
> > > 2 for the max supported version for the api. However, the version 1
> > should
> > > be clearly marked as deprecated on its documentation. It will be
> client's
> > > responsibility to make sure they are not using any such deprecated
> > version
> > > to the best knowledge of the client at the time of development (or
> > > alternatively by configuration)."
> > >
> > > Ismael
> > >
> > >
> > >
> > > On Fri, Apr 1, 2016 at 9:22 AM, Ismael Juma  wrote:
> > >
> > > > A couple of questions:
> > > >
> > > > 1. The KIP says "Specific version may be deprecated through protocol
> > > > documentation but must still be supported (although it is fair to
> > return
> > > an
> > > > error code if the specific API supports it).". It may be worth
> > expanding
> > > > this a little more. For example, what does it mean to support the
> API?
> > I
> > > > guess this means that the broker must not disconnect the client and
> the
> > > > broker must return a valid protocol response. Given that it says that
> > it
> > > is
> > > > "fair" (I would probably replace "fair" with "valid") to return an
> > error
> > > > code if the specific API supports it, it sounds like we are saying
> that
> > > we
> > > > don't have to maintain the semantic behaviour (i.e. we could _always_
> > > > return an error for a deprecated API?). Is this true?
> > > >
> > > > 2. ApiVersionQueryRequest seems a bit verbose, why not
> > ApiVersionRequest?
> > > >
> > > > Ismael
> > > >
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jun Rao]

Thinking about ApiVersionRequest a bit more. There are quite a few things
special about it. In the ideal case, (1) its version should never change;
(2) it needs to be done before authentication (either SSL/SASL); (3) it is
required to be issued at the beginning of each connection but never needs
to be issued again on the same connection. So, instead of modeling it as a
regular request, it seems a cleaner approach is to just bake that into the
initial connection handshake even before the authentication layer. So the
sequencing in a connection will be api discovery, authentication, followed
by regular requests. I am not sure we can still add this in a backward
compatible way now (e.g., not sure what the initial bytes from an SSL
connection will look like). Even if we can do this in a backward compatible
way, it's probably non-trivial amount of work though.

We started KIP-35 with supporting a client to know if a version is
supported by the broker. It's now evolved into supporting a client to
implement multiple versions of the protocol and dynamically pick a version
supported by the broker. The former is likely solvable without
ApiVersionRequest. How important is the latter? What if the C client just
follows the java client model by implementing one version of protocol per C
client release (which seems easier to implement)?

Thanks,

Jun


On Fri, Apr 8, 2016 at 4:20 PM, Jun Rao  wrote:

> Magnus,
>
> A while back, we had another proposal for the broker to just send the
> correlation id and an empty payload if it receives an unsupported version
> of the request. I didn't see that in the rejected section. It seems simpler
> than the current proposal where the client has to issue an
> ApiVersionRequest on every connection. Could you document the reason why we
> rejected it?
>
> Thanks,
>
> Jun
>
> On Tue, Apr 5, 2016 at 1:47 PM, Ashish Singh  wrote:
>
>> On Fri, Apr 1, 2016 at 1:32 AM, Ismael Juma  wrote:
>>
>> > Two more things:
>> >
>> > 3. We talk about backporting of new request versions to stable branches
>> in
>> > the KIP. In practice, we can't do that until the Java client is changed
>> so
>> > that it doesn't blindly use the latest protocol version. Otherwise, if
>> new
>> > request versions were added to 0.9.0.2, the client would break when
>> talking
>> > to a 0.9.0.1 broker (given Jason's proposal, it would fail a bit more
>> > gracefully, but that's still not good enough for a stable branch). It
>> may
>> > be worth making this clear in the KIP (yes, it is a bit orthogonal and
>> > doesn't prevent the KIP from being adopted, but good to avoid
>> confusion).
>> >
>> Good point. Adding this note and also adding a note that Kafka has not
>> backported an api version so far.
>>
>> >
>> > 4. The paragraph below is a bit confusing. It starts talking about 0.9.0
>> > and trunk and then switches to 0.9.1. Is that intentional?
>> >
>> Yes.
>>
>> >
>> > "Deprecation of a protocol version will be done by marking a protocol
>> > version as deprecated in protocol documentation. Documentation shall
>> also
>> > be used to indicate a protocol version that must not be used, or for any
>> > such information.For instance, say 0.9.0 had protocol versions [0] for
>> api
>> > key 1. On trunk, version 1 of the api key was added. Users running off
>> > trunk started using version 1 of the api and found out a major bug. To
>> > rectify that version 2 of the api is added to trunk. For some reason,
>> it is
>> > now deemed important to have version 2 of the api in 0.9.1 as well. To
>> do
>> > so, version 1 and version 2 both of the api will be backported to the
>> 0.9.1
>> > branch. 0.9.1 broker will return 0 as min supported version for the api
>> and
>> > 2 for the max supported version for the api. However, the version 1
>> should
>> > be clearly marked as deprecated on its documentation. It will be
>> client's
>> > responsibility to make sure they are not using any such deprecated
>> version
>> > to the best knowledge of the client at the time of development (or
>> > alternatively by configuration)."
>> >
>> > Ismael
>> >
>> >
>> >
>> > On Fri, Apr 1, 2016 at 9:22 AM, Ismael Juma  wrote:
>> >
>> > > A couple of questions:
>> > >
>> > > 1. The KIP says "Specific version may be deprecated through protocol
>> > > documentation but must still be supported (although it is fair to
>> return
>> > an
>> > > error code if the specific API supports it).". It may be worth
>> expanding
>> > > this a little more. For example, what does it mean to support the
>> API? I
>> > > guess this means that the broker must not disconnect the client and
>> the
>> > > broker must return a valid protocol response. Given that it says that
>> it
>> > is
>> > > "fair" (I would probably replace "fair" with "valid") to return an
>> error
>> > > code if the specific API supports it, it sounds like we are saying
>> that
>> > we
>> > > don't have to maintain the semantic 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jun Rao]

Magnus,

A while back, we had another proposal for the broker to just send the
correlation id and an empty payload if it receives an unsupported version
of the request. I didn't see that in the rejected section. It seems simpler
than the current proposal where the client has to issue an
ApiVersionRequest on every connection. Could you document the reason why we
rejected it?

Thanks,

Jun

On Tue, Apr 5, 2016 at 1:47 PM, Ashish Singh  wrote:

> On Fri, Apr 1, 2016 at 1:32 AM, Ismael Juma  wrote:
>
> > Two more things:
> >
> > 3. We talk about backporting of new request versions to stable branches
> in
> > the KIP. In practice, we can't do that until the Java client is changed
> so
> > that it doesn't blindly use the latest protocol version. Otherwise, if
> new
> > request versions were added to 0.9.0.2, the client would break when
> talking
> > to a 0.9.0.1 broker (given Jason's proposal, it would fail a bit more
> > gracefully, but that's still not good enough for a stable branch). It may
> > be worth making this clear in the KIP (yes, it is a bit orthogonal and
> > doesn't prevent the KIP from being adopted, but good to avoid confusion).
> >
> Good point. Adding this note and also adding a note that Kafka has not
> backported an api version so far.
>
> >
> > 4. The paragraph below is a bit confusing. It starts talking about 0.9.0
> > and trunk and then switches to 0.9.1. Is that intentional?
> >
> Yes.
>
> >
> > "Deprecation of a protocol version will be done by marking a protocol
> > version as deprecated in protocol documentation. Documentation shall also
> > be used to indicate a protocol version that must not be used, or for any
> > such information.For instance, say 0.9.0 had protocol versions [0] for
> api
> > key 1. On trunk, version 1 of the api key was added. Users running off
> > trunk started using version 1 of the api and found out a major bug. To
> > rectify that version 2 of the api is added to trunk. For some reason, it
> is
> > now deemed important to have version 2 of the api in 0.9.1 as well. To do
> > so, version 1 and version 2 both of the api will be backported to the
> 0.9.1
> > branch. 0.9.1 broker will return 0 as min supported version for the api
> and
> > 2 for the max supported version for the api. However, the version 1
> should
> > be clearly marked as deprecated on its documentation. It will be client's
> > responsibility to make sure they are not using any such deprecated
> version
> > to the best knowledge of the client at the time of development (or
> > alternatively by configuration)."
> >
> > Ismael
> >
> >
> >
> > On Fri, Apr 1, 2016 at 9:22 AM, Ismael Juma  wrote:
> >
> > > A couple of questions:
> > >
> > > 1. The KIP says "Specific version may be deprecated through protocol
> > > documentation but must still be supported (although it is fair to
> return
> > an
> > > error code if the specific API supports it).". It may be worth
> expanding
> > > this a little more. For example, what does it mean to support the API?
> I
> > > guess this means that the broker must not disconnect the client and the
> > > broker must return a valid protocol response. Given that it says that
> it
> > is
> > > "fair" (I would probably replace "fair" with "valid") to return an
> error
> > > code if the specific API supports it, it sounds like we are saying that
> > we
> > > don't have to maintain the semantic behaviour (i.e. we could _always_
> > > return an error for a deprecated API?). Is this true?
> > >
> > > 2. ApiVersionQueryRequest seems a bit verbose, why not
> ApiVersionRequest?
> > >
> > > Ismael
> > >
> >
>
>
>
> --
>
> Regards,
> Ashish
>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

On Fri, Apr 1, 2016 at 1:32 AM, Ismael Juma  wrote:

> Two more things:
>
> 3. We talk about backporting of new request versions to stable branches in
> the KIP. In practice, we can't do that until the Java client is changed so
> that it doesn't blindly use the latest protocol version. Otherwise, if new
> request versions were added to 0.9.0.2, the client would break when talking
> to a 0.9.0.1 broker (given Jason's proposal, it would fail a bit more
> gracefully, but that's still not good enough for a stable branch). It may
> be worth making this clear in the KIP (yes, it is a bit orthogonal and
> doesn't prevent the KIP from being adopted, but good to avoid confusion).
>
Good point. Adding this note and also adding a note that Kafka has not
backported an api version so far.

>
> 4. The paragraph below is a bit confusing. It starts talking about 0.9.0
> and trunk and then switches to 0.9.1. Is that intentional?
>
Yes.

>
> "Deprecation of a protocol version will be done by marking a protocol
> version as deprecated in protocol documentation. Documentation shall also
> be used to indicate a protocol version that must not be used, or for any
> such information.For instance, say 0.9.0 had protocol versions [0] for api
> key 1. On trunk, version 1 of the api key was added. Users running off
> trunk started using version 1 of the api and found out a major bug. To
> rectify that version 2 of the api is added to trunk. For some reason, it is
> now deemed important to have version 2 of the api in 0.9.1 as well. To do
> so, version 1 and version 2 both of the api will be backported to the 0.9.1
> branch. 0.9.1 broker will return 0 as min supported version for the api and
> 2 for the max supported version for the api. However, the version 1 should
> be clearly marked as deprecated on its documentation. It will be client's
> responsibility to make sure they are not using any such deprecated version
> to the best knowledge of the client at the time of development (or
> alternatively by configuration)."
>
> Ismael
>
>
>
> On Fri, Apr 1, 2016 at 9:22 AM, Ismael Juma  wrote:
>
> > A couple of questions:
> >
> > 1. The KIP says "Specific version may be deprecated through protocol
> > documentation but must still be supported (although it is fair to return
> an
> > error code if the specific API supports it).". It may be worth expanding
> > this a little more. For example, what does it mean to support the API? I
> > guess this means that the broker must not disconnect the client and the
> > broker must return a valid protocol response. Given that it says that it
> is
> > "fair" (I would probably replace "fair" with "valid") to return an error
> > code if the specific API supports it, it sounds like we are saying that
> we
> > don't have to maintain the semantic behaviour (i.e. we could _always_
> > return an error for a deprecated API?). Is this true?
> >
> > 2. ApiVersionQueryRequest seems a bit verbose, why not ApiVersionRequest?
> >
> > Ismael
> >
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Ismael, thanks for the review.

On Fri, Apr 1, 2016 at 1:22 AM, Ismael Juma  wrote:

> A couple of questions:
>
> 1. The KIP says "Specific version may be deprecated through protocol
> documentation but must still be supported (although it is fair to return an
> error code if the specific API supports it).". It may be worth expanding
> this a little more. For example, what does it mean to support the API? I
> guess this means that the broker must not disconnect the client and the
> broker must return a valid protocol response.

Sure, will add to KIP.

> Given that it says that it is
> "fair" (I would probably replace "fair" with "valid") to return an error
> code if the specific API supports it, it sounds like we are saying that we
> don't have to maintain the semantic behaviour (i.e. we could _always_
> return an error for a deprecated API?). Is this true?
>
Yes. Typically we should support deprecated APIs, however in cases where an
intermediate buggy version was backported, if the api has error field, it
should be valid to use it to signal an error. Example of such backporting
is provided on KIP, below is excerpt. Note that this KIP is not asking for
any new error type.
> For instance, say 0.9.0 had protocol versions [0] for api key 1. On
trunk, version 1 of the api key was added. Users running off trunk started
using version 1 of the api and found out a major bug. To rectify that
version 2 of the api is added to trunk. For some reason, it is now deemed
important to have version 2 of the api in 0.9.1 as well. To do so, version
1 and version 2 both of the api will be backported to the 0.9.1 branch.
0.9.1 broker will return 0 as min supported version for the api and 2 for
the max supported version for the api. However, the version 1 should be
clearly marked as deprecated on its documentation. It will be client's
responsibility to make sure they are not using any such deprecated version
to the best knowledge of the client at the time of development (or
alternatively by configuration).

>
> 2. ApiVersionQueryRequest seems a bit verbose, why not ApiVersionRequest?
>
Adopted.

>
> Ismael
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Magnus Edenhill]

Hey,

people have had concerns about the complexity of the client of mapping API
versions to features,
so I implemented this in librdkafka and it is rather straight forward.

See here:
https://github.com/edenhill/librdkafka/blob/KIP-35/src/rdkafka_feature.c#L52

Consider it a proof of concept at this point.

Thanks,
Magnus




2016-04-05 19:40 GMT+02:00 Magnus Edenhill :

> Ashish, thanks, didnt know that.
>
> For ApiVersionRequest requesting no Apis  to be returned doesnt make sense
> so the distinction isn't necessary,
> but I'm fine with adding Null to be more in line with future protocol
> requests, as long as it doesn't delay
> this KIP any longer! :)
>
> 2016-04-05 19:38 GMT+02:00 Ashish Singh :
>
>> Magnus, it is proposed to be changed in version 1,
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-MetadataSchema
>> .
>>
>> On Tue, Apr 5, 2016 at 10:23 AM, Magnus Edenhill 
>> wrote:
>>
>> > Empty arrays are already used in MetadataRequest to retrieve all topics
>> in
>> > the cluster,
>> > ApiVersion request will have the same standard semantics.
>> >
>> >
>> > 2016-04-05 19:01 GMT+02:00 Ewen Cheslack-Postava :
>> >
>> > > Also, just a thought but is empty list the sentinel we really want to
>> > > indicate we want all API versions? We've got nullable string and
>> nullable
>> > > bytes in the protocol. Should we have nullable array support as well
>> and
>> > > use that to indicate we want everything? I can't think of a use case
>> for
>> > > sending an empty list, but null seems like a better sentinel than an
>> > empty
>> > > list.
>> > >
>> > > -Ewen
>> > >
>> > > On Tue, Apr 5, 2016 at 9:23 AM, Ewen Cheslack-Postava <
>> e...@confluent.io
>> > >
>> > > wrote:
>> > >
>> > > >
>> > > >
>> > > > On Mon, Apr 4, 2016 at 11:24 AM, Gwen Shapira 
>> > wrote:
>> > > >
>> > > >> >
>> > > >> >
>> > > >> >In case of connection closures, the KIP recommends that
>> clients
>> > > >> should
>> > > >> >use some other method of determining the apiRequest version to
>> > use,
>> > > >> > like,
>> > > >> >probing. For instance, client will send V0 version of
>> apiVersion
>> > > >> request
>> > > >> >and will try higher versions incrementally. In case b, client
>> > will
>> > > >> >eventually get apiVersion response and know what api versions
>> it
>> > > >> should
>> > > >> >use. For case a and c, client will eventually give up and
>> > propagate
>> > > >> an
>> > > >> >error to application.
>> > > >> >
>> > > >> >
>> > > >> I strongly disagree that we should recommend this probing method.
>> > > >> Probing is essentially what clients do now (since we lack any way
>> to
>> > > >> communicate versions), and is what we are trying to solve with this
>> > KIP.
>> > > >> Considering that different brokers could have different versions,
>> and
>> > > that
>> > > >> brokers can change version at any point, this sounds slow,
>> difficult
>> > to
>> > > >> implement and fragile.
>> > > >>
>> > > >> Also note that even with this method, without VersionRequest v0, we
>> > will
>> > > >> break clients in the one way Kafka currently promises to never
>> break:
>> > > Old
>> > > >> clients won't be able to work with new brokers.
>> > > >>
>> > > >> If this is part of KIP-35, I am against it.
>> > > >>
>> > > >> Since all Request/Responses in our protocol have versions,
>> publishing
>> > > >> versions for each request/response should be something we can
>> easily
>> > > >> support into the future. It sounds far easier than asking every
>> single
>> > > >> client to implement the method you specified above.
>> > > >>
>> > > >
>> > > > Gwen,
>> > > >
>> > > > Agreed, and I think it would be fine to make permanent support
>> (barring
>> > > > massive changes to the protocol) part of the KIP text. There's
>> really
>> > no
>> > > > reason not to and it basically just turns this into the basis for a
>> > > pretty
>> > > > simple handshake protocol.
>> > > >
>> > > > (I'm tempted to not even bring this up given that we're converging,
>> but
>> > > > one reason I could see this being changed in the future is that
>> > protocol
>> > > > support is only conveyed in one direction. This could also be turned
>> > > into a
>> > > > slightly more general handshake approach where the client also
>> > advertises
>> > > > what it supports. However, given the way request/response versioning
>> > > works,
>> > > > I can't think of a reason we'd need this atm.)
>> > > >
>> > > > -Ewen
>> > > >
>> > > >
>> > > >>
>> > > >> Gwen
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Thanks,
>> > > > Ewen
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Thanks,
>> > > Ewen
>> > >
>> >
>>
>>
>>
>> --
>>
>> Regards,
>> Ashish
>>
>
>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Magnus Edenhill]

Ashish, thanks, didnt know that.

For ApiVersionRequest requesting no Apis  to be returned doesnt make sense
so the distinction isn't necessary,
but I'm fine with adding Null to be more in line with future protocol
requests, as long as it doesn't delay
this KIP any longer! :)

2016-04-05 19:38 GMT+02:00 Ashish Singh :

> Magnus, it is proposed to be changed in version 1,
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-MetadataSchema
> .
>
> On Tue, Apr 5, 2016 at 10:23 AM, Magnus Edenhill 
> wrote:
>
> > Empty arrays are already used in MetadataRequest to retrieve all topics
> in
> > the cluster,
> > ApiVersion request will have the same standard semantics.
> >
> >
> > 2016-04-05 19:01 GMT+02:00 Ewen Cheslack-Postava :
> >
> > > Also, just a thought but is empty list the sentinel we really want to
> > > indicate we want all API versions? We've got nullable string and
> nullable
> > > bytes in the protocol. Should we have nullable array support as well
> and
> > > use that to indicate we want everything? I can't think of a use case
> for
> > > sending an empty list, but null seems like a better sentinel than an
> > empty
> > > list.
> > >
> > > -Ewen
> > >
> > > On Tue, Apr 5, 2016 at 9:23 AM, Ewen Cheslack-Postava <
> e...@confluent.io
> > >
> > > wrote:
> > >
> > > >
> > > >
> > > > On Mon, Apr 4, 2016 at 11:24 AM, Gwen Shapira 
> > wrote:
> > > >
> > > >> >
> > > >> >
> > > >> >In case of connection closures, the KIP recommends that clients
> > > >> should
> > > >> >use some other method of determining the apiRequest version to
> > use,
> > > >> > like,
> > > >> >probing. For instance, client will send V0 version of
> apiVersion
> > > >> request
> > > >> >and will try higher versions incrementally. In case b, client
> > will
> > > >> >eventually get apiVersion response and know what api versions
> it
> > > >> should
> > > >> >use. For case a and c, client will eventually give up and
> > propagate
> > > >> an
> > > >> >error to application.
> > > >> >
> > > >> >
> > > >> I strongly disagree that we should recommend this probing method.
> > > >> Probing is essentially what clients do now (since we lack any way to
> > > >> communicate versions), and is what we are trying to solve with this
> > KIP.
> > > >> Considering that different brokers could have different versions,
> and
> > > that
> > > >> brokers can change version at any point, this sounds slow, difficult
> > to
> > > >> implement and fragile.
> > > >>
> > > >> Also note that even with this method, without VersionRequest v0, we
> > will
> > > >> break clients in the one way Kafka currently promises to never
> break:
> > > Old
> > > >> clients won't be able to work with new brokers.
> > > >>
> > > >> If this is part of KIP-35, I am against it.
> > > >>
> > > >> Since all Request/Responses in our protocol have versions,
> publishing
> > > >> versions for each request/response should be something we can easily
> > > >> support into the future. It sounds far easier than asking every
> single
> > > >> client to implement the method you specified above.
> > > >>
> > > >
> > > > Gwen,
> > > >
> > > > Agreed, and I think it would be fine to make permanent support
> (barring
> > > > massive changes to the protocol) part of the KIP text. There's really
> > no
> > > > reason not to and it basically just turns this into the basis for a
> > > pretty
> > > > simple handshake protocol.
> > > >
> > > > (I'm tempted to not even bring this up given that we're converging,
> but
> > > > one reason I could see this being changed in the future is that
> > protocol
> > > > support is only conveyed in one direction. This could also be turned
> > > into a
> > > > slightly more general handshake approach where the client also
> > advertises
> > > > what it supports. However, given the way request/response versioning
> > > works,
> > > > I can't think of a reason we'd need this atm.)
> > > >
> > > > -Ewen
> > > >
> > > >
> > > >>
> > > >> Gwen
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Ewen
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Ewen
> > >
> >
>
>
>
> --
>
> Regards,
> Ashish
>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Magnus, it is proposed to be changed in version 1,
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-MetadataSchema
.

On Tue, Apr 5, 2016 at 10:23 AM, Magnus Edenhill  wrote:

> Empty arrays are already used in MetadataRequest to retrieve all topics in
> the cluster,
> ApiVersion request will have the same standard semantics.
>
>
> 2016-04-05 19:01 GMT+02:00 Ewen Cheslack-Postava :
>
> > Also, just a thought but is empty list the sentinel we really want to
> > indicate we want all API versions? We've got nullable string and nullable
> > bytes in the protocol. Should we have nullable array support as well and
> > use that to indicate we want everything? I can't think of a use case for
> > sending an empty list, but null seems like a better sentinel than an
> empty
> > list.
> >
> > -Ewen
> >
> > On Tue, Apr 5, 2016 at 9:23 AM, Ewen Cheslack-Postava  >
> > wrote:
> >
> > >
> > >
> > > On Mon, Apr 4, 2016 at 11:24 AM, Gwen Shapira 
> wrote:
> > >
> > >> >
> > >> >
> > >> >In case of connection closures, the KIP recommends that clients
> > >> should
> > >> >use some other method of determining the apiRequest version to
> use,
> > >> > like,
> > >> >probing. For instance, client will send V0 version of apiVersion
> > >> request
> > >> >and will try higher versions incrementally. In case b, client
> will
> > >> >eventually get apiVersion response and know what api versions it
> > >> should
> > >> >use. For case a and c, client will eventually give up and
> propagate
> > >> an
> > >> >error to application.
> > >> >
> > >> >
> > >> I strongly disagree that we should recommend this probing method.
> > >> Probing is essentially what clients do now (since we lack any way to
> > >> communicate versions), and is what we are trying to solve with this
> KIP.
> > >> Considering that different brokers could have different versions, and
> > that
> > >> brokers can change version at any point, this sounds slow, difficult
> to
> > >> implement and fragile.
> > >>
> > >> Also note that even with this method, without VersionRequest v0, we
> will
> > >> break clients in the one way Kafka currently promises to never break:
> > Old
> > >> clients won't be able to work with new brokers.
> > >>
> > >> If this is part of KIP-35, I am against it.
> > >>
> > >> Since all Request/Responses in our protocol have versions, publishing
> > >> versions for each request/response should be something we can easily
> > >> support into the future. It sounds far easier than asking every single
> > >> client to implement the method you specified above.
> > >>
> > >
> > > Gwen,
> > >
> > > Agreed, and I think it would be fine to make permanent support (barring
> > > massive changes to the protocol) part of the KIP text. There's really
> no
> > > reason not to and it basically just turns this into the basis for a
> > pretty
> > > simple handshake protocol.
> > >
> > > (I'm tempted to not even bring this up given that we're converging, but
> > > one reason I could see this being changed in the future is that
> protocol
> > > support is only conveyed in one direction. This could also be turned
> > into a
> > > slightly more general handshake approach where the client also
> advertises
> > > what it supports. However, given the way request/response versioning
> > works,
> > > I can't think of a reason we'd need this atm.)
> > >
> > > -Ewen
> > >
> > >
> > >>
> > >> Gwen
> > >>
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Ewen
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Magnus Edenhill]

Empty arrays are already used in MetadataRequest to retrieve all topics in
the cluster,
ApiVersion request will have the same standard semantics.


2016-04-05 19:01 GMT+02:00 Ewen Cheslack-Postava :

> Also, just a thought but is empty list the sentinel we really want to
> indicate we want all API versions? We've got nullable string and nullable
> bytes in the protocol. Should we have nullable array support as well and
> use that to indicate we want everything? I can't think of a use case for
> sending an empty list, but null seems like a better sentinel than an empty
> list.
>
> -Ewen
>
> On Tue, Apr 5, 2016 at 9:23 AM, Ewen Cheslack-Postava 
> wrote:
>
> >
> >
> > On Mon, Apr 4, 2016 at 11:24 AM, Gwen Shapira  wrote:
> >
> >> >
> >> >
> >> >In case of connection closures, the KIP recommends that clients
> >> should
> >> >use some other method of determining the apiRequest version to use,
> >> > like,
> >> >probing. For instance, client will send V0 version of apiVersion
> >> request
> >> >and will try higher versions incrementally. In case b, client will
> >> >eventually get apiVersion response and know what api versions it
> >> should
> >> >use. For case a and c, client will eventually give up and propagate
> >> an
> >> >error to application.
> >> >
> >> >
> >> I strongly disagree that we should recommend this probing method.
> >> Probing is essentially what clients do now (since we lack any way to
> >> communicate versions), and is what we are trying to solve with this KIP.
> >> Considering that different brokers could have different versions, and
> that
> >> brokers can change version at any point, this sounds slow, difficult to
> >> implement and fragile.
> >>
> >> Also note that even with this method, without VersionRequest v0, we will
> >> break clients in the one way Kafka currently promises to never break:
> Old
> >> clients won't be able to work with new brokers.
> >>
> >> If this is part of KIP-35, I am against it.
> >>
> >> Since all Request/Responses in our protocol have versions, publishing
> >> versions for each request/response should be something we can easily
> >> support into the future. It sounds far easier than asking every single
> >> client to implement the method you specified above.
> >>
> >
> > Gwen,
> >
> > Agreed, and I think it would be fine to make permanent support (barring
> > massive changes to the protocol) part of the KIP text. There's really no
> > reason not to and it basically just turns this into the basis for a
> pretty
> > simple handshake protocol.
> >
> > (I'm tempted to not even bring this up given that we're converging, but
> > one reason I could see this being changed in the future is that protocol
> > support is only conveyed in one direction. This could also be turned
> into a
> > slightly more general handshake approach where the client also advertises
> > what it supports. However, given the way request/response versioning
> works,
> > I can't think of a reason we'd need this atm.)
> >
> > -Ewen
> >
> >
> >>
> >> Gwen
> >>
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>
>
>
> --
> Thanks,
> Ewen
>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Sounds fair. I am OK with putting down, permanent support of ApiVersion api
versions, as a limitation in KIP.

On Tue, Apr 5, 2016 at 9:23 AM, Ewen Cheslack-Postava 
wrote:

> On Mon, Apr 4, 2016 at 11:24 AM, Gwen Shapira  wrote:
>
> > >
> > >
> > >In case of connection closures, the KIP recommends that clients
> should
> > >use some other method of determining the apiRequest version to use,
> > > like,
> > >probing. For instance, client will send V0 version of apiVersion
> > request
> > >and will try higher versions incrementally. In case b, client will
> > >eventually get apiVersion response and know what api versions it
> > should
> > >use. For case a and c, client will eventually give up and propagate
> an
> > >error to application.
> > >
> > >
> > I strongly disagree that we should recommend this probing method.
> > Probing is essentially what clients do now (since we lack any way to
> > communicate versions), and is what we are trying to solve with this KIP.
> > Considering that different brokers could have different versions, and
> that
> > brokers can change version at any point, this sounds slow, difficult to
> > implement and fragile.
> >
> > Also note that even with this method, without VersionRequest v0, we will
> > break clients in the one way Kafka currently promises to never break: Old
> > clients won't be able to work with new brokers.
> >
> > If this is part of KIP-35, I am against it.
> >
> > Since all Request/Responses in our protocol have versions, publishing
> > versions for each request/response should be something we can easily
> > support into the future. It sounds far easier than asking every single
> > client to implement the method you specified above.
> >
>
> Gwen,
>
> Agreed, and I think it would be fine to make permanent support (barring
> massive changes to the protocol) part of the KIP text. There's really no
> reason not to and it basically just turns this into the basis for a pretty
> simple handshake protocol.
>
> (I'm tempted to not even bring this up given that we're converging, but one
> reason I could see this being changed in the future is that protocol
> support is only conveyed in one direction. This could also be turned into a
> slightly more general handshake approach where the client also advertises
> what it supports. However, given the way request/response versioning works,
> I can't think of a reason we'd need this atm.)
>
> -Ewen
>
>
> >
> > Gwen
> >
>
>
>
> --
> Thanks,
> Ewen
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Null array sounds good to me as well.

On Tue, Apr 5, 2016 at 10:06 AM, Ismael Juma  wrote:

> Yeah, we should use nullable arrays (which have been introduced in
> KIP-4-Metadata) instead of empty list to indicate all versions.
>
> Ismael
> On 5 Apr 2016 18:01, "Ewen Cheslack-Postava"  wrote:
>
> > Also, just a thought but is empty list the sentinel we really want to
> > indicate we want all API versions? We've got nullable string and nullable
> > bytes in the protocol. Should we have nullable array support as well and
> > use that to indicate we want everything? I can't think of a use case for
> > sending an empty list, but null seems like a better sentinel than an
> empty
> > list.
> >
> > -Ewen
> >
> > On Tue, Apr 5, 2016 at 9:23 AM, Ewen Cheslack-Postava  >
> > wrote:
> >
> > >
> > >
> > > On Mon, Apr 4, 2016 at 11:24 AM, Gwen Shapira 
> wrote:
> > >
> > >> >
> > >> >
> > >> >In case of connection closures, the KIP recommends that clients
> > >> should
> > >> >use some other method of determining the apiRequest version to
> use,
> > >> > like,
> > >> >probing. For instance, client will send V0 version of apiVersion
> > >> request
> > >> >and will try higher versions incrementally. In case b, client
> will
> > >> >eventually get apiVersion response and know what api versions it
> > >> should
> > >> >use. For case a and c, client will eventually give up and
> propagate
> > >> an
> > >> >error to application.
> > >> >
> > >> >
> > >> I strongly disagree that we should recommend this probing method.
> > >> Probing is essentially what clients do now (since we lack any way to
> > >> communicate versions), and is what we are trying to solve with this
> KIP.
> > >> Considering that different brokers could have different versions, and
> > that
> > >> brokers can change version at any point, this sounds slow, difficult
> to
> > >> implement and fragile.
> > >>
> > >> Also note that even with this method, without VersionRequest v0, we
> will
> > >> break clients in the one way Kafka currently promises to never break:
> > Old
> > >> clients won't be able to work with new brokers.
> > >>
> > >> If this is part of KIP-35, I am against it.
> > >>
> > >> Since all Request/Responses in our protocol have versions, publishing
> > >> versions for each request/response should be something we can easily
> > >> support into the future. It sounds far easier than asking every single
> > >> client to implement the method you specified above.
> > >>
> > >
> > > Gwen,
> > >
> > > Agreed, and I think it would be fine to make permanent support (barring
> > > massive changes to the protocol) part of the KIP text. There's really
> no
> > > reason not to and it basically just turns this into the basis for a
> > pretty
> > > simple handshake protocol.
> > >
> > > (I'm tempted to not even bring this up given that we're converging, but
> > > one reason I could see this being changed in the future is that
> protocol
> > > support is only conveyed in one direction. This could also be turned
> > into a
> > > slightly more general handshake approach where the client also
> advertises
> > > what it supports. However, given the way request/response versioning
> > works,
> > > I can't think of a reason we'd need this atm.)
> > >
> > > -Ewen
> > >
> > >
> > >>
> > >> Gwen
> > >>
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Ewen
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

Yeah, we should use nullable arrays (which have been introduced in
KIP-4-Metadata) instead of empty list to indicate all versions.

Ismael
On 5 Apr 2016 18:01, "Ewen Cheslack-Postava"  wrote:

> Also, just a thought but is empty list the sentinel we really want to
> indicate we want all API versions? We've got nullable string and nullable
> bytes in the protocol. Should we have nullable array support as well and
> use that to indicate we want everything? I can't think of a use case for
> sending an empty list, but null seems like a better sentinel than an empty
> list.
>
> -Ewen
>
> On Tue, Apr 5, 2016 at 9:23 AM, Ewen Cheslack-Postava 
> wrote:
>
> >
> >
> > On Mon, Apr 4, 2016 at 11:24 AM, Gwen Shapira  wrote:
> >
> >> >
> >> >
> >> >In case of connection closures, the KIP recommends that clients
> >> should
> >> >use some other method of determining the apiRequest version to use,
> >> > like,
> >> >probing. For instance, client will send V0 version of apiVersion
> >> request
> >> >and will try higher versions incrementally. In case b, client will
> >> >eventually get apiVersion response and know what api versions it
> >> should
> >> >use. For case a and c, client will eventually give up and propagate
> >> an
> >> >error to application.
> >> >
> >> >
> >> I strongly disagree that we should recommend this probing method.
> >> Probing is essentially what clients do now (since we lack any way to
> >> communicate versions), and is what we are trying to solve with this KIP.
> >> Considering that different brokers could have different versions, and
> that
> >> brokers can change version at any point, this sounds slow, difficult to
> >> implement and fragile.
> >>
> >> Also note that even with this method, without VersionRequest v0, we will
> >> break clients in the one way Kafka currently promises to never break:
> Old
> >> clients won't be able to work with new brokers.
> >>
> >> If this is part of KIP-35, I am against it.
> >>
> >> Since all Request/Responses in our protocol have versions, publishing
> >> versions for each request/response should be something we can easily
> >> support into the future. It sounds far easier than asking every single
> >> client to implement the method you specified above.
> >>
> >
> > Gwen,
> >
> > Agreed, and I think it would be fine to make permanent support (barring
> > massive changes to the protocol) part of the KIP text. There's really no
> > reason not to and it basically just turns this into the basis for a
> pretty
> > simple handshake protocol.
> >
> > (I'm tempted to not even bring this up given that we're converging, but
> > one reason I could see this being changed in the future is that protocol
> > support is only conveyed in one direction. This could also be turned
> into a
> > slightly more general handshake approach where the client also advertises
> > what it supports. However, given the way request/response versioning
> works,
> > I can't think of a reason we'd need this atm.)
> >
> > -Ewen
> >
> >
> >>
> >> Gwen
> >>
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>
>
>
> --
> Thanks,
> Ewen
>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ewen Cheslack-Postava]

Also, just a thought but is empty list the sentinel we really want to
indicate we want all API versions? We've got nullable string and nullable
bytes in the protocol. Should we have nullable array support as well and
use that to indicate we want everything? I can't think of a use case for
sending an empty list, but null seems like a better sentinel than an empty
list.

-Ewen

On Tue, Apr 5, 2016 at 9:23 AM, Ewen Cheslack-Postava 
wrote:

>
>
> On Mon, Apr 4, 2016 at 11:24 AM, Gwen Shapira  wrote:
>
>> >
>> >
>> >In case of connection closures, the KIP recommends that clients
>> should
>> >use some other method of determining the apiRequest version to use,
>> > like,
>> >probing. For instance, client will send V0 version of apiVersion
>> request
>> >and will try higher versions incrementally. In case b, client will
>> >eventually get apiVersion response and know what api versions it
>> should
>> >use. For case a and c, client will eventually give up and propagate
>> an
>> >error to application.
>> >
>> >
>> I strongly disagree that we should recommend this probing method.
>> Probing is essentially what clients do now (since we lack any way to
>> communicate versions), and is what we are trying to solve with this KIP.
>> Considering that different brokers could have different versions, and that
>> brokers can change version at any point, this sounds slow, difficult to
>> implement and fragile.
>>
>> Also note that even with this method, without VersionRequest v0, we will
>> break clients in the one way Kafka currently promises to never break: Old
>> clients won't be able to work with new brokers.
>>
>> If this is part of KIP-35, I am against it.
>>
>> Since all Request/Responses in our protocol have versions, publishing
>> versions for each request/response should be something we can easily
>> support into the future. It sounds far easier than asking every single
>> client to implement the method you specified above.
>>
>
> Gwen,
>
> Agreed, and I think it would be fine to make permanent support (barring
> massive changes to the protocol) part of the KIP text. There's really no
> reason not to and it basically just turns this into the basis for a pretty
> simple handshake protocol.
>
> (I'm tempted to not even bring this up given that we're converging, but
> one reason I could see this being changed in the future is that protocol
> support is only conveyed in one direction. This could also be turned into a
> slightly more general handshake approach where the client also advertises
> what it supports. However, given the way request/response versioning works,
> I can't think of a reason we'd need this atm.)
>
> -Ewen
>
>
>>
>> Gwen
>>
>
>
>
> --
> Thanks,
> Ewen
>



-- 
Thanks,
Ewen

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ewen Cheslack-Postava]

On Mon, Apr 4, 2016 at 11:24 AM, Gwen Shapira  wrote:

> >
> >
> >In case of connection closures, the KIP recommends that clients should
> >use some other method of determining the apiRequest version to use,
> > like,
> >probing. For instance, client will send V0 version of apiVersion
> request
> >and will try higher versions incrementally. In case b, client will
> >eventually get apiVersion response and know what api versions it
> should
> >use. For case a and c, client will eventually give up and propagate an
> >error to application.
> >
> >
> I strongly disagree that we should recommend this probing method.
> Probing is essentially what clients do now (since we lack any way to
> communicate versions), and is what we are trying to solve with this KIP.
> Considering that different brokers could have different versions, and that
> brokers can change version at any point, this sounds slow, difficult to
> implement and fragile.
>
> Also note that even with this method, without VersionRequest v0, we will
> break clients in the one way Kafka currently promises to never break: Old
> clients won't be able to work with new brokers.
>
> If this is part of KIP-35, I am against it.
>
> Since all Request/Responses in our protocol have versions, publishing
> versions for each request/response should be something we can easily
> support into the future. It sounds far easier than asking every single
> client to implement the method you specified above.
>

Gwen,

Agreed, and I think it would be fine to make permanent support (barring
massive changes to the protocol) part of the KIP text. There's really no
reason not to and it basically just turns this into the basis for a pretty
simple handshake protocol.

(I'm tempted to not even bring this up given that we're converging, but one
reason I could see this being changed in the future is that protocol
support is only conveyed in one direction. This could also be turned into a
slightly more general handshake approach where the client also advertises
what it supports. However, given the way request/response versioning works,
I can't think of a reason we'd need this atm.)

-Ewen


>
> Gwen
>



-- 
Thanks,
Ewen

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

>
>
>In case of connection closures, the KIP recommends that clients should
>use some other method of determining the apiRequest version to use,
> like,
>probing. For instance, client will send V0 version of apiVersion request
>and will try higher versions incrementally. In case b, client will
>eventually get apiVersion response and know what api versions it should
>use. For case a and c, client will eventually give up and propagate an
>error to application.
>
>
I strongly disagree that we should recommend this probing method.
Probing is essentially what clients do now (since we lack any way to
communicate versions), and is what we are trying to solve with this KIP.
Considering that different brokers could have different versions, and that
brokers can change version at any point, this sounds slow, difficult to
implement and fragile.

Also note that even with this method, without VersionRequest v0, we will
break clients in the one way Kafka currently promises to never break: Old
clients won't be able to work with new brokers.

If this is part of KIP-35, I am against it.

Since all Request/Responses in our protocol have versions, publishing
versions for each request/response should be something we can easily
support into the future. It sounds far easier than asking every single
client to implement the method you specified above.

Gwen

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ewen Cheslack-Postava]

Sorry for jumping over some feedback, but not sure the best way to provide
this feedback. Somewhat indirectly targeting feedback from Jay:

> "I actually do think good is the enemy of perfect (not sure if that makes
sense but I think you get what I mean)."

Yes I get it, but I don't agree (so surprising, I know!). One of the
benefits of 0.8 protocol changes is to make it possible to make incremental
protocol changes
Having a compatibility story for fundamental changes like
VersionRequest/Response is important. Yes, we want to modify these *core*
interfaces as infrequently as possible. But more important than trying to
*not* change them is *the ability* to change them. This is important to
both Jay & Gwen's concerns (and probably others, but calling out those
specifically...).

A lot of what we're talking about here is putting *some versioning/upgrade
support* in place. Yes, we have to live with it and support it for a long
time, but we have opportunity to evolve it as necessary as long as we plan
for that evolution. I think we're underestimating our ability to be
backwards compatible and also support future changes

I think the KIP-35 solution is reasonable because it solves the immediate
problem, doesn't have real practical objections, and doesn't block future
improvements. I personally think there are more meaningful ways to express
broker/client support (features!), but this initial step is *purely
informational* and doesn't block us from supporting more expressive info
moving forward. And frankly, I can't see how it could possibly be a
significant burden moving forward: if we express this as "advertisements"
-- we can easily guarantee we'll continue to advertise features/protocol
support/etc, but beyond that there aren't many guarantees we need to
satisfy. This is not a real burden on the broker, but provides extremely
valuable information to clients.

I'm one of the few people who suggested a wildly different alternative at
the (beginning? middle?) of this conversation. I think I'm the only one
that really buys into the idea of formalizing `features` as opposed to
protocol versions. I also have no qualms about evolving the current
proposal to support these features via an updated, new version of
ApiVersionQueryRequest because I think we do this all in a compatible way.
Continuing to support the "old" format doesn't look like it is a real
problem to me because the features we propose to express are very simple
and relatable, any broker or client should have no trouble with the
mapping.







On Fri, Apr 1, 2016 at 2:41 PM, Ashish Singh  wrote:

> Gwen, I do not think it will break clients. Just to be clear we have
> following options.
>
>1.
>
>Versioned apiVersion request and response.
>KIP-35 aware clients will send apiRequest, version Vn, to get api
>versions from a broker. If a broker supports Vn of apiVersion request,
> it
>will respond. If not, client will face a connection closure, which can
> be
>due to many reasons.
>a. Broker has older version of Kafka and it does not support apiVersion
>request at all.
>b. Broker does not support version Vn of apiVersion request.
>c. Connection closure was caused by issues other than unknown apiVersion
>request.
>
>In case of connection closures, the KIP recommends that clients should
>use some other method of determining the apiRequest version to use,
> like,
>probing. For instance, client will send V0 version of apiVersion request
>and will try higher versions incrementally. In case b, client will
>eventually get apiVersion response and know what api versions it should
>use. For case a and c, client will eventually give up and propagate an
>error to application.
>
>Note that though the KIP suggests to keep apiVersion request versioned,
>there is no realistic reason, at this time, that warrants a version
> bump.
>However, it is keeping the option open.
>2.
>
>apiVersion request and response with no versioning.
>KIP-35 aware clients will send apiRequest to get api versions from a
>broker. If a broker supports apiVersion request, it will respond. If
> not,
>client will face a connection closure, which can be due to following
>reasons.
>a. Broker has older version of Kafka and it does not support apiVersion
>request at all.
>b. Connection closure was caused by issues other than unknown apiVersion
>request.
>
>This approach does not require client to do any linear probing in case
>of connection closure, however there will be no way in future to change
>apiVersion req/resp format.
>
> Hopefully this gives more information to decide between the above mentioned
> options. What do you think now?
> ​
>
> On Fri, Apr 1, 2016 at 2:06 PM, Gwen Shapira  wrote:
>
> > It isn't a guarantee, it is a limitation.
> >
> > It doesn't even matter what we do (have versions, no versions, deprecate
> or
> > 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Gwen, I do not think it will break clients. Just to be clear we have
following options.

   1.

   Versioned apiVersion request and response.
   KIP-35 aware clients will send apiRequest, version Vn, to get api
   versions from a broker. If a broker supports Vn of apiVersion request, it
   will respond. If not, client will face a connection closure, which can be
   due to many reasons.
   a. Broker has older version of Kafka and it does not support apiVersion
   request at all.
   b. Broker does not support version Vn of apiVersion request.
   c. Connection closure was caused by issues other than unknown apiVersion
   request.

   In case of connection closures, the KIP recommends that clients should
   use some other method of determining the apiRequest version to use, like,
   probing. For instance, client will send V0 version of apiVersion request
   and will try higher versions incrementally. In case b, client will
   eventually get apiVersion response and know what api versions it should
   use. For case a and c, client will eventually give up and propagate an
   error to application.

   Note that though the KIP suggests to keep apiVersion request versioned,
   there is no realistic reason, at this time, that warrants a version bump.
   However, it is keeping the option open.
   2.

   apiVersion request and response with no versioning.
   KIP-35 aware clients will send apiRequest to get api versions from a
   broker. If a broker supports apiVersion request, it will respond. If not,
   client will face a connection closure, which can be due to following
   reasons.
   a. Broker has older version of Kafka and it does not support apiVersion
   request at all.
   b. Connection closure was caused by issues other than unknown apiVersion
   request.

   This approach does not require client to do any linear probing in case
   of connection closure, however there will be no way in future to change
   apiVersion req/resp format.

Hopefully this gives more information to decide between the above mentioned
options. What do you think now?
​

On Fri, Apr 1, 2016 at 2:06 PM, Gwen Shapira  wrote:

> It isn't a guarantee, it is a limitation.
>
> It doesn't even matter what we do (have versions, no versions, deprecate or
> not) - clients will never be able to use version other than v0 (how will
> they know it is there?) and if we ever remove v0, we will immediately break
> all clients.
>
> If we are ok making changes that immediately break all clients, we don't
> need this KIP. I thought the entire point was to fix this.
>
> So, by having this KIP we agree to never ever remove VersionRequest v0 (as
> it is in the KIP).  I agree it could be hard, but no one promised that
> implementing a server that is backward and forward compatible is easy.
> Everyone who votes on this KIP must be aware that whatever we do here is
> here to stay.
>
> Which is exactly why this discussion is so important.
>
> Gwen
>
> On Fri, Apr 1, 2016 at 1:52 PM, Ashish Singh  wrote:
>
> > Never ever changing the version of VersionReq/resp will be ideal, but it
> is
> > hard to guarantee that we will never need a change. We should definitely
> > try to avoid it and honestly I do not see a reason, as of now, why it
> will
> > change. I would say, we should keep it versioned and document any risk of
> > changing it. Does that sound OK?
> >
> > On Fri, Apr 1, 2016 at 1:12 PM, Gwen Shapira  wrote:
> >
> > > I am happy with:
> > > 1. Adding only validation support in the client for now, deferring
> > backward
> > > compatibility for later. With the odd side-effect of having the C and
> > > Python clients actually better than Java for a bit.
> > > 2. Never ever changing the version of VersionRequest and
> VersionResponse.
> > >
> > > I'd also like to see:
> > > 1. At least one client implement backward compatibility (so we can use
> it
> > > in testing)
> > > 2. Backward compatibility tests using that client (including an upgrade
> > > test)
> > > 3. Checklist for reviewers on when protocol bumps are required
> > >
> > > I am strongly against releasing an API that no one is using as
> > intended...
> > > so either we do it in our client or use a 3rd party client.
> > >
> > > Gwen
> > >
> > > On Fri, Apr 1, 2016 at 1:05 PM, Jason Gustafson 
> > > wrote:
> > >
> > > > I'm sympathetic to Jay's argument, but I feel the version API can be
> > > > justified on its own without requiring a big change in the way we
> > manage
> > > > compatibility. For the official client compatibility story (i.e. what
> > the
> > > > Java client implements), we can use this API to verify compatibility
> > with
> > > > the broker, which is a significant gap at the moment (we can do this
> > now
> > > > before the release if we all agree without much delay). It also
> happens
> > > to
> > > > help clients trying to maintain backwards compatibility, which is
> > great.
> > > >
> > > > Maybe the 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

It isn't a guarantee, it is a limitation.

It doesn't even matter what we do (have versions, no versions, deprecate or
not) - clients will never be able to use version other than v0 (how will
they know it is there?) and if we ever remove v0, we will immediately break
all clients.

If we are ok making changes that immediately break all clients, we don't
need this KIP. I thought the entire point was to fix this.

So, by having this KIP we agree to never ever remove VersionRequest v0 (as
it is in the KIP).  I agree it could be hard, but no one promised that
implementing a server that is backward and forward compatible is easy.
Everyone who votes on this KIP must be aware that whatever we do here is
here to stay.

Which is exactly why this discussion is so important.

Gwen

On Fri, Apr 1, 2016 at 1:52 PM, Ashish Singh  wrote:

> Never ever changing the version of VersionReq/resp will be ideal, but it is
> hard to guarantee that we will never need a change. We should definitely
> try to avoid it and honestly I do not see a reason, as of now, why it will
> change. I would say, we should keep it versioned and document any risk of
> changing it. Does that sound OK?
>
> On Fri, Apr 1, 2016 at 1:12 PM, Gwen Shapira  wrote:
>
> > I am happy with:
> > 1. Adding only validation support in the client for now, deferring
> backward
> > compatibility for later. With the odd side-effect of having the C and
> > Python clients actually better than Java for a bit.
> > 2. Never ever changing the version of VersionRequest and VersionResponse.
> >
> > I'd also like to see:
> > 1. At least one client implement backward compatibility (so we can use it
> > in testing)
> > 2. Backward compatibility tests using that client (including an upgrade
> > test)
> > 3. Checklist for reviewers on when protocol bumps are required
> >
> > I am strongly against releasing an API that no one is using as
> intended...
> > so either we do it in our client or use a 3rd party client.
> >
> > Gwen
> >
> > On Fri, Apr 1, 2016 at 1:05 PM, Jason Gustafson 
> > wrote:
> >
> > > I'm sympathetic to Jay's argument, but I feel the version API can be
> > > justified on its own without requiring a big change in the way we
> manage
> > > compatibility. For the official client compatibility story (i.e. what
> the
> > > Java client implements), we can use this API to verify compatibility
> with
> > > the broker, which is a significant gap at the moment (we can do this
> now
> > > before the release if we all agree without much delay). It also happens
> > to
> > > help clients trying to maintain backwards compatibility, which is
> great.
> > >
> > > Maybe the question is whether this API will restrict later efforts to
> > > change the way we manage compatibility? I don't see much risk in terms
> of
> > > maintenance of the API, but there is some risk that it could become
> > > superfluous at some point. For example, if we changed the response
> header
> > > to include an error code for an unknown request version, then maybe the
> > > Java client would no longer need the initial call to the version API
> > > (though it might arguably still be a good idea to do so to detect
> > > incompatibility sooner). Maybe there is some risk also that it makes
> the
> > > compatibility story more confusing? I actually think it fits pretty
> well
> > > with the versioned API approach generally, though it's admittedly a
> > little
> > > odd to have the version API versioned itself. Practically speaking, I
> > think
> > > this means that we shouldn't increment the version. Otherwise the
> client
> > > will need another round trip (after always starting with v0), which
> might
> > > be doable even if it seems a little messy.
> > >
> > > Thoughts?
> > >
> > > -Jason
> > >
> > > On Fri, Apr 1, 2016 at 11:36 AM, Gwen Shapira 
> wrote:
> > >
> > > > My understanding is that we are not adding the client portion to the
> > KIP
> > > is
> > > > because we believe it will require a lot of discussion (read: will be
> > > > really hard to get right).
> > > >
> > > > Maybe it is a hint that this protocol is too difficult for clients to
> > > > implement?
> > > > I can't see why is it easy in C and Python and super difficult in
> Java.
> > > >
> > > > Even if it is too hard to implement just in Java for some reason (god
> > > knows
> > > > the language has issues), isn't it a good reason to come up with
> > > something
> > > > we can implement in a reasonable way?
> > > >
> > > > Gwen
> > > >
> > > > On Fri, Apr 1, 2016 at 10:58 AM, Ashish Singh 
> > > wrote:
> > > >
> > > > > That is a fair concern and I think eventually we might want to have
> > > java
> > > > > clients backwards compatible. However, blocking KIP-35 on that
> might
> > > not
> > > > be
> > > > > the best idea. The reason I say so is due to following reasons.
> > > > >
> > > > > 1. Backwards compatibility in java clients is a 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Never ever changing the version of VersionReq/resp will be ideal, but it is
hard to guarantee that we will never need a change. We should definitely
try to avoid it and honestly I do not see a reason, as of now, why it will
change. I would say, we should keep it versioned and document any risk of
changing it. Does that sound OK?

On Fri, Apr 1, 2016 at 1:12 PM, Gwen Shapira  wrote:

> I am happy with:
> 1. Adding only validation support in the client for now, deferring backward
> compatibility for later. With the odd side-effect of having the C and
> Python clients actually better than Java for a bit.
> 2. Never ever changing the version of VersionRequest and VersionResponse.
>
> I'd also like to see:
> 1. At least one client implement backward compatibility (so we can use it
> in testing)
> 2. Backward compatibility tests using that client (including an upgrade
> test)
> 3. Checklist for reviewers on when protocol bumps are required
>
> I am strongly against releasing an API that no one is using as intended...
> so either we do it in our client or use a 3rd party client.
>
> Gwen
>
> On Fri, Apr 1, 2016 at 1:05 PM, Jason Gustafson 
> wrote:
>
> > I'm sympathetic to Jay's argument, but I feel the version API can be
> > justified on its own without requiring a big change in the way we manage
> > compatibility. For the official client compatibility story (i.e. what the
> > Java client implements), we can use this API to verify compatibility with
> > the broker, which is a significant gap at the moment (we can do this now
> > before the release if we all agree without much delay). It also happens
> to
> > help clients trying to maintain backwards compatibility, which is great.
> >
> > Maybe the question is whether this API will restrict later efforts to
> > change the way we manage compatibility? I don't see much risk in terms of
> > maintenance of the API, but there is some risk that it could become
> > superfluous at some point. For example, if we changed the response header
> > to include an error code for an unknown request version, then maybe the
> > Java client would no longer need the initial call to the version API
> > (though it might arguably still be a good idea to do so to detect
> > incompatibility sooner). Maybe there is some risk also that it makes the
> > compatibility story more confusing? I actually think it fits pretty well
> > with the versioned API approach generally, though it's admittedly a
> little
> > odd to have the version API versioned itself. Practically speaking, I
> think
> > this means that we shouldn't increment the version. Otherwise the client
> > will need another round trip (after always starting with v0), which might
> > be doable even if it seems a little messy.
> >
> > Thoughts?
> >
> > -Jason
> >
> > On Fri, Apr 1, 2016 at 11:36 AM, Gwen Shapira  wrote:
> >
> > > My understanding is that we are not adding the client portion to the
> KIP
> > is
> > > because we believe it will require a lot of discussion (read: will be
> > > really hard to get right).
> > >
> > > Maybe it is a hint that this protocol is too difficult for clients to
> > > implement?
> > > I can't see why is it easy in C and Python and super difficult in Java.
> > >
> > > Even if it is too hard to implement just in Java for some reason (god
> > knows
> > > the language has issues), isn't it a good reason to come up with
> > something
> > > we can implement in a reasonable way?
> > >
> > > Gwen
> > >
> > > On Fri, Apr 1, 2016 at 10:58 AM, Ashish Singh 
> > wrote:
> > >
> > > > That is a fair concern and I think eventually we might want to have
> > java
> > > > clients backwards compatible. However, blocking KIP-35 on that might
> > not
> > > be
> > > > the best idea. The reason I say so is due to following reasons.
> > > >
> > > > 1. Backwards compatibility in java clients is a larger discussion, as
> > we
> > > > have already seen. Having a separate KIP focussing exactly on that
> will
> > > > help in reducing moving pieces in one KIP.
> > > > 2. It probably will fall out of 0.10 due to tight timeline, I am
> > assuming
> > > > we do not want to delay 0.10 a lot.
> > > > 3. Even if we make java clients backward compatible in 0.10, I do not
> > > think
> > > > it will be able to work with older releases as older broker versions
> > > still
> > > > won't provide info on supported api versions. If we add api versions
> > > > req/resp in 0.10, and add backwards compatibility in java clients in
> > > later
> > > > versions, they will probably work with 0.10 and we will be able to
> test
> > > > that.
> > > >
> > > > On Fri, Apr 1, 2016 at 10:28 AM, Ewen Cheslack-Postava <
> > > e...@confluent.io>
> > > > wrote:
> > > >
> > > > > On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira 
> > > wrote:
> > > > >
> > > > > > I have two concerns with the proposal as it is:
> > > > > >
> > > > > > 1. Having an API that publishes 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jason Gustafson]

I'm sympathetic to Jay's argument, but I feel the version API can be
justified on its own without requiring a big change in the way we manage
compatibility. For the official client compatibility story (i.e. what the
Java client implements), we can use this API to verify compatibility with
the broker, which is a significant gap at the moment (we can do this now
before the release if we all agree without much delay). It also happens to
help clients trying to maintain backwards compatibility, which is great.

Maybe the question is whether this API will restrict later efforts to
change the way we manage compatibility? I don't see much risk in terms of
maintenance of the API, but there is some risk that it could become
superfluous at some point. For example, if we changed the response header
to include an error code for an unknown request version, then maybe the
Java client would no longer need the initial call to the version API
(though it might arguably still be a good idea to do so to detect
incompatibility sooner). Maybe there is some risk also that it makes the
compatibility story more confusing? I actually think it fits pretty well
with the versioned API approach generally, though it's admittedly a little
odd to have the version API versioned itself. Practically speaking, I think
this means that we shouldn't increment the version. Otherwise the client
will need another round trip (after always starting with v0), which might
be doable even if it seems a little messy.

Thoughts?

-Jason

On Fri, Apr 1, 2016 at 11:36 AM, Gwen Shapira  wrote:

> My understanding is that we are not adding the client portion to the KIP is
> because we believe it will require a lot of discussion (read: will be
> really hard to get right).
>
> Maybe it is a hint that this protocol is too difficult for clients to
> implement?
> I can't see why is it easy in C and Python and super difficult in Java.
>
> Even if it is too hard to implement just in Java for some reason (god knows
> the language has issues), isn't it a good reason to come up with something
> we can implement in a reasonable way?
>
> Gwen
>
> On Fri, Apr 1, 2016 at 10:58 AM, Ashish Singh  wrote:
>
> > That is a fair concern and I think eventually we might want to have java
> > clients backwards compatible. However, blocking KIP-35 on that might not
> be
> > the best idea. The reason I say so is due to following reasons.
> >
> > 1. Backwards compatibility in java clients is a larger discussion, as we
> > have already seen. Having a separate KIP focussing exactly on that will
> > help in reducing moving pieces in one KIP.
> > 2. It probably will fall out of 0.10 due to tight timeline, I am assuming
> > we do not want to delay 0.10 a lot.
> > 3. Even if we make java clients backward compatible in 0.10, I do not
> think
> > it will be able to work with older releases as older broker versions
> still
> > won't provide info on supported api versions. If we add api versions
> > req/resp in 0.10, and add backwards compatibility in java clients in
> later
> > versions, they will probably work with 0.10 and we will be able to test
> > that.
> >
> > On Fri, Apr 1, 2016 at 10:28 AM, Ewen Cheslack-Postava <
> e...@confluent.io>
> > wrote:
> >
> > > On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira 
> wrote:
> > >
> > > > I have two concerns with the proposal as it is:
> > > >
> > > > 1. Having an API that publishes protocol is useless for clients if
> > > > developers don't bump the API when they should.
> > > > I would like to see good documentation on when protocols are bumped
> > and a
> > > > proposal on how this will be automatically validated to the extent
> > > possible
> > > > (and to what extent are we still at risk of accidental breakage).
> > > > Even if we don't implement all of this at first pass, I want to know
> > > which
> > > > direction we are going in order to solve the client compatibility
> > issue.
> > > >
> > > > 2. In addition to third-party clients, there are stream processing
> > > > frameworks who use the Java client we publish as their client and
> would
> > > > also like to enjoy the same compatibility benefits C and Python
> clients
> > > > will enjoy. It will be very silly if Apache Kafka clients are the
> worst
> > > > clients out there from compatibility POV. The KIP can be implemented
> in
> > > > parts, but I really want to see an effort to build the Java client
> > > > compatibility into the KIP and if possible into the release too.
> > > >
> > >
> > > I wouldn't conflate those two things. Changing the compatibility
> approach
> > > of the Java clients could easily be another KIP (and probably a large
> > one,
> > > too). There are already high quality, well maintained clients trying to
> > use
> > > a different approach and this addresses their needs. Blocking the
> entire
> > > ecosystem on the Java client seems problematic.
> > >
> > > That said, agreed that it would be bad for the Java 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

My understanding is that we are not adding the client portion to the KIP is
because we believe it will require a lot of discussion (read: will be
really hard to get right).

Maybe it is a hint that this protocol is too difficult for clients to
implement?
I can't see why is it easy in C and Python and super difficult in Java.

Even if it is too hard to implement just in Java for some reason (god knows
the language has issues), isn't it a good reason to come up with something
we can implement in a reasonable way?

Gwen

On Fri, Apr 1, 2016 at 10:58 AM, Ashish Singh  wrote:

> That is a fair concern and I think eventually we might want to have java
> clients backwards compatible. However, blocking KIP-35 on that might not be
> the best idea. The reason I say so is due to following reasons.
>
> 1. Backwards compatibility in java clients is a larger discussion, as we
> have already seen. Having a separate KIP focussing exactly on that will
> help in reducing moving pieces in one KIP.
> 2. It probably will fall out of 0.10 due to tight timeline, I am assuming
> we do not want to delay 0.10 a lot.
> 3. Even if we make java clients backward compatible in 0.10, I do not think
> it will be able to work with older releases as older broker versions still
> won't provide info on supported api versions. If we add api versions
> req/resp in 0.10, and add backwards compatibility in java clients in later
> versions, they will probably work with 0.10 and we will be able to test
> that.
>
> On Fri, Apr 1, 2016 at 10:28 AM, Ewen Cheslack-Postava 
> wrote:
>
> > On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira  wrote:
> >
> > > I have two concerns with the proposal as it is:
> > >
> > > 1. Having an API that publishes protocol is useless for clients if
> > > developers don't bump the API when they should.
> > > I would like to see good documentation on when protocols are bumped
> and a
> > > proposal on how this will be automatically validated to the extent
> > possible
> > > (and to what extent are we still at risk of accidental breakage).
> > > Even if we don't implement all of this at first pass, I want to know
> > which
> > > direction we are going in order to solve the client compatibility
> issue.
> > >
> > > 2. In addition to third-party clients, there are stream processing
> > > frameworks who use the Java client we publish as their client and would
> > > also like to enjoy the same compatibility benefits C and Python clients
> > > will enjoy. It will be very silly if Apache Kafka clients are the worst
> > > clients out there from compatibility POV. The KIP can be implemented in
> > > parts, but I really want to see an effort to build the Java client
> > > compatibility into the KIP and if possible into the release too.
> > >
> >
> > I wouldn't conflate those two things. Changing the compatibility approach
> > of the Java clients could easily be another KIP (and probably a large
> one,
> > too). There are already high quality, well maintained clients trying to
> use
> > a different approach and this addresses their needs. Blocking the entire
> > ecosystem on the Java client seems problematic.
> >
> > That said, agreed that it would be bad for the Java client to have the
> > worst compatibility story.
> >
> > -Ewen
> >
> >
> > >
> > > Gwen
> > >
> > > On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson 
> > > wrote:
> > >
> > > > Bumping this thread. I talked with Ashish and Magnus about this KIP
> > > offline
> > > > and I'm gradually coming over. The new API actually stands by itself
> > > > outside of the discussion about whether the client should support
> > > backwards
> > > > compatibility or not. For the Java client, we could continue to
> support
> > > the
> > > > current compatibility approach in which the client supports only
> > brokers
> > > > with the same version or greater. In that case, we would use this API
> > > only
> > > > to assert that the current API versions are all supported, and raise
> an
> > > > exception if they are not. This gives us the capability going forward
> > to
> > > > detect when the client is talking to an older broker, which we don't
> > have
> > > > right now. This check should be straightforward, so we could do it
> now,
> > > > which would resolve some of the uneasiness about having an unused
> > feature
> > > > which we depended on other clients to test for us. Does that make
> sense
> > > or
> > > > not?
> > > >
> > > > -Jason
> > > >
> > > > On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh 
> > > wrote:
> > > >
> > > > > We have proposed and discussed majorly three approaches so far,
> there
> > > > were
> > > > > many minor versions with small variations. Comparing them really
> > > > requires a
> > > > > side by side proposal and their pros/cons, and I agree with others
> > that
> > > > > this has been lacking in the KIP. We just updated the KIP with
> > > following
> > > > > 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jay Kreps]

Hey Ewen, with protocol design I actually do think good is the enemy of
perfect (not sure if that makes sense but I think you get what I mean).
Your comment seemed to be along the lines of "this isn't very good, but
let's do something". Can you elaborate on what you think we should be
doing?

The only two things I'd add to the comments so far:
1. We will probably live with this scheme for five years if we make this
change. Let's for the love of god not do something half assed. It would be
far far better not to do anything yet and come at this when we have the
time to think it through, then to do something that doesn't make sense.
2. As Gwen says unless/until someone actually sees this through--gets the
docs updated, implements the java client support, etc. we will be in a
worse situation than we currently are. Is anyone actually signing up for
that? I do think these things are coupled in actually getting us to a
better place though they are separate chunks of work.

I get that people want to see forward motion here but I really think this
is an area where thoughtful beats fast. If we do something half-assed we
first do the work of converting the world to that, then the work of undoing
it and doing the thing we should have originally done so it really doesn't
speed things up.

-Jay

On Fri, Apr 1, 2016 at 10:58 AM, Ashish Singh  wrote:

> That is a fair concern and I think eventually we might want to have java
> clients backwards compatible. However, blocking KIP-35 on that might not be
> the best idea. The reason I say so is due to following reasons.
>
> 1. Backwards compatibility in java clients is a larger discussion, as we
> have already seen. Having a separate KIP focussing exactly on that will
> help in reducing moving pieces in one KIP.
> 2. It probably will fall out of 0.10 due to tight timeline, I am assuming
> we do not want to delay 0.10 a lot.
> 3. Even if we make java clients backward compatible in 0.10, I do not think
> it will be able to work with older releases as older broker versions still
> won't provide info on supported api versions. If we add api versions
> req/resp in 0.10, and add backwards compatibility in java clients in later
> versions, they will probably work with 0.10 and we will be able to test
> that.
>
> On Fri, Apr 1, 2016 at 10:28 AM, Ewen Cheslack-Postava 
> wrote:
>
> > On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira  wrote:
> >
> > > I have two concerns with the proposal as it is:
> > >
> > > 1. Having an API that publishes protocol is useless for clients if
> > > developers don't bump the API when they should.
> > > I would like to see good documentation on when protocols are bumped
> and a
> > > proposal on how this will be automatically validated to the extent
> > possible
> > > (and to what extent are we still at risk of accidental breakage).
> > > Even if we don't implement all of this at first pass, I want to know
> > which
> > > direction we are going in order to solve the client compatibility
> issue.
> > >
> > > 2. In addition to third-party clients, there are stream processing
> > > frameworks who use the Java client we publish as their client and would
> > > also like to enjoy the same compatibility benefits C and Python clients
> > > will enjoy. It will be very silly if Apache Kafka clients are the worst
> > > clients out there from compatibility POV. The KIP can be implemented in
> > > parts, but I really want to see an effort to build the Java client
> > > compatibility into the KIP and if possible into the release too.
> > >
> >
> > I wouldn't conflate those two things. Changing the compatibility approach
> > of the Java clients could easily be another KIP (and probably a large
> one,
> > too). There are already high quality, well maintained clients trying to
> use
> > a different approach and this addresses their needs. Blocking the entire
> > ecosystem on the Java client seems problematic.
> >
> > That said, agreed that it would be bad for the Java client to have the
> > worst compatibility story.
> >
> > -Ewen
> >
> >
> > >
> > > Gwen
> > >
> > > On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson 
> > > wrote:
> > >
> > > > Bumping this thread. I talked with Ashish and Magnus about this KIP
> > > offline
> > > > and I'm gradually coming over. The new API actually stands by itself
> > > > outside of the discussion about whether the client should support
> > > backwards
> > > > compatibility or not. For the Java client, we could continue to
> support
> > > the
> > > > current compatibility approach in which the client supports only
> > brokers
> > > > with the same version or greater. In that case, we would use this API
> > > only
> > > > to assert that the current API versions are all supported, and raise
> an
> > > > exception if they are not. This gives us the capability going forward
> > to
> > > > detect when the client is talking to an older broker, which we don't
> > have

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

That is a fair concern and I think eventually we might want to have java
clients backwards compatible. However, blocking KIP-35 on that might not be
the best idea. The reason I say so is due to following reasons.

1. Backwards compatibility in java clients is a larger discussion, as we
have already seen. Having a separate KIP focussing exactly on that will
help in reducing moving pieces in one KIP.
2. It probably will fall out of 0.10 due to tight timeline, I am assuming
we do not want to delay 0.10 a lot.
3. Even if we make java clients backward compatible in 0.10, I do not think
it will be able to work with older releases as older broker versions still
won't provide info on supported api versions. If we add api versions
req/resp in 0.10, and add backwards compatibility in java clients in later
versions, they will probably work with 0.10 and we will be able to test
that.

On Fri, Apr 1, 2016 at 10:28 AM, Ewen Cheslack-Postava 
wrote:

> On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira  wrote:
>
> > I have two concerns with the proposal as it is:
> >
> > 1. Having an API that publishes protocol is useless for clients if
> > developers don't bump the API when they should.
> > I would like to see good documentation on when protocols are bumped and a
> > proposal on how this will be automatically validated to the extent
> possible
> > (and to what extent are we still at risk of accidental breakage).
> > Even if we don't implement all of this at first pass, I want to know
> which
> > direction we are going in order to solve the client compatibility issue.
> >
> > 2. In addition to third-party clients, there are stream processing
> > frameworks who use the Java client we publish as their client and would
> > also like to enjoy the same compatibility benefits C and Python clients
> > will enjoy. It will be very silly if Apache Kafka clients are the worst
> > clients out there from compatibility POV. The KIP can be implemented in
> > parts, but I really want to see an effort to build the Java client
> > compatibility into the KIP and if possible into the release too.
> >
>
> I wouldn't conflate those two things. Changing the compatibility approach
> of the Java clients could easily be another KIP (and probably a large one,
> too). There are already high quality, well maintained clients trying to use
> a different approach and this addresses their needs. Blocking the entire
> ecosystem on the Java client seems problematic.
>
> That said, agreed that it would be bad for the Java client to have the
> worst compatibility story.
>
> -Ewen
>
>
> >
> > Gwen
> >
> > On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson 
> > wrote:
> >
> > > Bumping this thread. I talked with Ashish and Magnus about this KIP
> > offline
> > > and I'm gradually coming over. The new API actually stands by itself
> > > outside of the discussion about whether the client should support
> > backwards
> > > compatibility or not. For the Java client, we could continue to support
> > the
> > > current compatibility approach in which the client supports only
> brokers
> > > with the same version or greater. In that case, we would use this API
> > only
> > > to assert that the current API versions are all supported, and raise an
> > > exception if they are not. This gives us the capability going forward
> to
> > > detect when the client is talking to an older broker, which we don't
> have
> > > right now. This check should be straightforward, so we could do it now,
> > > which would resolve some of the uneasiness about having an unused
> feature
> > > which we depended on other clients to test for us. Does that make sense
> > or
> > > not?
> > >
> > > -Jason
> > >
> > > On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh 
> > wrote:
> > >
> > > > We have proposed and discussed majorly three approaches so far, there
> > > were
> > > > many minor versions with small variations. Comparing them really
> > > requires a
> > > > side by side proposal and their pros/cons, and I agree with others
> that
> > > > this has been lacking in the KIP. We just updated the KIP with
> > following
> > > > details.
> > > >
> > > > 1. Provide proposed changes in all the three proposals we have
> > discussed
> > > so
> > > > far. Except the current proposal, these proposals are in rejected
> > > > alternatives.
> > > > 2. Provide reasoning on why the rejected proposals were rejected.
> > > > 3. Add scenarios for all of these proposals from a client developer
> and
> > > > core Kafka developer point of view.
> > > >
> > > > As we are really close to 0.10 deadline, a quick round of voting will
> > > > really help. If you really do not like the idea, please feel free to
> > say
> > > > so. If the vote fails for the current proposal, it can at lease
> provide
> > > > recommendations that we should consider for next version of proposal
> > and
> > > > put it up for vote again for next release. However, as stated 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ewen Cheslack-Postava]

On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira  wrote:

> I have two concerns with the proposal as it is:
>
> 1. Having an API that publishes protocol is useless for clients if
> developers don't bump the API when they should.
> I would like to see good documentation on when protocols are bumped and a
> proposal on how this will be automatically validated to the extent possible
> (and to what extent are we still at risk of accidental breakage).
> Even if we don't implement all of this at first pass, I want to know which
> direction we are going in order to solve the client compatibility issue.
>
> 2. In addition to third-party clients, there are stream processing
> frameworks who use the Java client we publish as their client and would
> also like to enjoy the same compatibility benefits C and Python clients
> will enjoy. It will be very silly if Apache Kafka clients are the worst
> clients out there from compatibility POV. The KIP can be implemented in
> parts, but I really want to see an effort to build the Java client
> compatibility into the KIP and if possible into the release too.
>

I wouldn't conflate those two things. Changing the compatibility approach
of the Java clients could easily be another KIP (and probably a large one,
too). There are already high quality, well maintained clients trying to use
a different approach and this addresses their needs. Blocking the entire
ecosystem on the Java client seems problematic.

That said, agreed that it would be bad for the Java client to have the
worst compatibility story.

-Ewen


>
> Gwen
>
> On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson 
> wrote:
>
> > Bumping this thread. I talked with Ashish and Magnus about this KIP
> offline
> > and I'm gradually coming over. The new API actually stands by itself
> > outside of the discussion about whether the client should support
> backwards
> > compatibility or not. For the Java client, we could continue to support
> the
> > current compatibility approach in which the client supports only brokers
> > with the same version or greater. In that case, we would use this API
> only
> > to assert that the current API versions are all supported, and raise an
> > exception if they are not. This gives us the capability going forward to
> > detect when the client is talking to an older broker, which we don't have
> > right now. This check should be straightforward, so we could do it now,
> > which would resolve some of the uneasiness about having an unused feature
> > which we depended on other clients to test for us. Does that make sense
> or
> > not?
> >
> > -Jason
> >
> > On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh 
> wrote:
> >
> > > We have proposed and discussed majorly three approaches so far, there
> > were
> > > many minor versions with small variations. Comparing them really
> > requires a
> > > side by side proposal and their pros/cons, and I agree with others that
> > > this has been lacking in the KIP. We just updated the KIP with
> following
> > > details.
> > >
> > > 1. Provide proposed changes in all the three proposals we have
> discussed
> > so
> > > far. Except the current proposal, these proposals are in rejected
> > > alternatives.
> > > 2. Provide reasoning on why the rejected proposals were rejected.
> > > 3. Add scenarios for all of these proposals from a client developer and
> > > core Kafka developer point of view.
> > >
> > > As we are really close to 0.10 deadline, a quick round of voting will
> > > really help. If you really do not like the idea, please feel free to
> say
> > > so. If the vote fails for the current proposal, it can at lease provide
> > > recommendations that we should consider for next version of proposal
> and
> > > put it up for vote again for next release. However, as stated earlier
> by
> > > multiple people having this ASAP will be awesome.
> > >
> > > On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers 
> > > wrote:
> > >
> > > > On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira 
> > wrote:
> > > >
> > > > > "I think I would make this approach work by looking at the released
> > > > server
> > > > > version documentation for each version that I am trying to support
> > and
> > > > test
> > > > > against*, manually identify the expected "protocol vectors" each
> > > > supports,
> > > > > store that as a map of vectors to "broker versions", check each
> > vector
> > > at
> > > > > runtime until I find a match, and write code compatibility checks
> > from
> > > > > there."
> > > > >
> > > > > How is this better than a global version ID?
> > > >
> > > >
> > > > As a client developer, it seems roughly the same. I think it probably
> > > > avoids the server development workflow issues, and possibly the need
> to
> > > > agree on semantics of the global version ID? But others surely are
> more
> > > > qualified than me to comment on that part.
> > > >
> > > > -Dana
> > > >
> > >
> > 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

Two more things:

3. We talk about backporting of new request versions to stable branches in
the KIP. In practice, we can't do that until the Java client is changed so
that it doesn't blindly use the latest protocol version. Otherwise, if new
request versions were added to 0.9.0.2, the client would break when talking
to a 0.9.0.1 broker (given Jason's proposal, it would fail a bit more
gracefully, but that's still not good enough for a stable branch). It may
be worth making this clear in the KIP (yes, it is a bit orthogonal and
doesn't prevent the KIP from being adopted, but good to avoid confusion).

4. The paragraph below is a bit confusing. It starts talking about 0.9.0
and trunk and then switches to 0.9.1. Is that intentional?

"Deprecation of a protocol version will be done by marking a protocol
version as deprecated in protocol documentation. Documentation shall also
be used to indicate a protocol version that must not be used, or for any
such information.For instance, say 0.9.0 had protocol versions [0] for api
key 1. On trunk, version 1 of the api key was added. Users running off
trunk started using version 1 of the api and found out a major bug. To
rectify that version 2 of the api is added to trunk. For some reason, it is
now deemed important to have version 2 of the api in 0.9.1 as well. To do
so, version 1 and version 2 both of the api will be backported to the 0.9.1
branch. 0.9.1 broker will return 0 as min supported version for the api and
2 for the max supported version for the api. However, the version 1 should
be clearly marked as deprecated on its documentation. It will be client's
responsibility to make sure they are not using any such deprecated version
to the best knowledge of the client at the time of development (or
alternatively by configuration)."

Ismael



On Fri, Apr 1, 2016 at 9:22 AM, Ismael Juma  wrote:

> A couple of questions:
>
> 1. The KIP says "Specific version may be deprecated through protocol
> documentation but must still be supported (although it is fair to return an
> error code if the specific API supports it).". It may be worth expanding
> this a little more. For example, what does it mean to support the API? I
> guess this means that the broker must not disconnect the client and the
> broker must return a valid protocol response. Given that it says that it is
> "fair" (I would probably replace "fair" with "valid") to return an error
> code if the specific API supports it, it sounds like we are saying that we
> don't have to maintain the semantic behaviour (i.e. we could _always_
> return an error for a deprecated API?). Is this true?
>
> 2. ApiVersionQueryRequest seems a bit verbose, why not ApiVersionRequest?
>
> Ismael
>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

A couple of questions:

1. The KIP says "Specific version may be deprecated through protocol
documentation but must still be supported (although it is fair to return an
error code if the specific API supports it).". It may be worth expanding
this a little more. For example, what does it mean to support the API? I
guess this means that the broker must not disconnect the client and the
broker must return a valid protocol response. Given that it says that it is
"fair" (I would probably replace "fair" with "valid") to return an error
code if the specific API supports it, it sounds like we are saying that we
don't have to maintain the semantic behaviour (i.e. we could _always_
return an error for a deprecated API?). Is this true?

2. ApiVersionQueryRequest seems a bit verbose, why not ApiVersionRequest?

Ismael

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ewen Cheslack-Postava]

I agree in a "perfect is the enemy of good" sense. I don't think what's
expressed in the API is necessarily the *ideal* thing for client
developers, which is who I think this API should be optimizing for. That
said, this is a practical solution to the problem and while finer-grained
than I think is actually needed, it's better to be too fine grained and
require a bit of extra effort for client developers than to be too
coarse-grained (e.g., global version numbers) and make it
difficult/impossible to get the desired behavior.

Short version: if Magnus and Dana think this is a reasonable solution, I'll
sign off on it since I see nothing glaringly wrong or bad about it and it
solves real problems for them :)

I think we arrived at a policy for Java client compatibility which works
fine when everything is developed in lockstep in the same codebase, but
isn't actually practical with other clients. In practice, no client
developers follow the same compatibility matrix -- they want clients
compiled with versions more recently than brokers to continue to work with
brokers. Although I understand how you can make the approach used with Java
clients work, I also understand why folks would want this other approach to
work. Given that a lot of clients want a different compatibility matrix
than the Java clients support and this KIP will enable that (even if it
requires some translations in the client libraries to figure out what
features are supported based on API versions), I think we should adopt this
support.

-Ewen

On Thu, Mar 31, 2016 at 3:05 PM, Ashish Singh  wrote:

> I agree with Jason. The blocking factor has been how to use the proposed
> changes in java client without making it super complicated. With Jason's
> suggestion we can get past this blocker, while keeping the core of the
> proposal intact.
>
>
> On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson 
> wrote:
>
> > Bumping this thread. I talked with Ashish and Magnus about this KIP
> offline
> > and I'm gradually coming over. The new API actually stands by itself
> > outside of the discussion about whether the client should support
> backwards
> > compatibility or not. For the Java client, we could continue to support
> the
> > current compatibility approach in which the client supports only brokers
> > with the same version or greater. In that case, we would use this API
> only
> > to assert that the current API versions are all supported, and raise an
> > exception if they are not. This gives us the capability going forward to
> > detect when the client is talking to an older broker, which we don't have
> > right now. This check should be straightforward, so we could do it now,
> > which would resolve some of the uneasiness about having an unused feature
> > which we depended on other clients to test for us. Does that make sense
> or
> > not?
> >
> > -Jason
> >
> > On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh 
> wrote:
> >
> > > We have proposed and discussed majorly three approaches so far, there
> > were
> > > many minor versions with small variations. Comparing them really
> > requires a
> > > side by side proposal and their pros/cons, and I agree with others that
> > > this has been lacking in the KIP. We just updated the KIP with
> following
> > > details.
> > >
> > > 1. Provide proposed changes in all the three proposals we have
> discussed
> > so
> > > far. Except the current proposal, these proposals are in rejected
> > > alternatives.
> > > 2. Provide reasoning on why the rejected proposals were rejected.
> > > 3. Add scenarios for all of these proposals from a client developer and
> > > core Kafka developer point of view.
> > >
> > > As we are really close to 0.10 deadline, a quick round of voting will
> > > really help. If you really do not like the idea, please feel free to
> say
> > > so. If the vote fails for the current proposal, it can at lease provide
> > > recommendations that we should consider for next version of proposal
> and
> > > put it up for vote again for next release. However, as stated earlier
> by
> > > multiple people having this ASAP will be awesome.
> > >
> > > On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers 
> > > wrote:
> > >
> > > > On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira 
> > wrote:
> > > >
> > > > > "I think I would make this approach work by looking at the released
> > > > server
> > > > > version documentation for each version that I am trying to support
> > and
> > > > test
> > > > > against*, manually identify the expected "protocol vectors" each
> > > > supports,
> > > > > store that as a map of vectors to "broker versions", check each
> > vector
> > > at
> > > > > runtime until I find a match, and write code compatibility checks
> > from
> > > > > there."
> > > > >
> > > > > How is this better than a global version ID?
> > > >
> > > >
> > > > As a client developer, it seems roughly the same. I think it 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

I agree with Jason. The blocking factor has been how to use the proposed
changes in java client without making it super complicated. With Jason's
suggestion we can get past this blocker, while keeping the core of the
proposal intact.


On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson  wrote:

> Bumping this thread. I talked with Ashish and Magnus about this KIP offline
> and I'm gradually coming over. The new API actually stands by itself
> outside of the discussion about whether the client should support backwards
> compatibility or not. For the Java client, we could continue to support the
> current compatibility approach in which the client supports only brokers
> with the same version or greater. In that case, we would use this API only
> to assert that the current API versions are all supported, and raise an
> exception if they are not. This gives us the capability going forward to
> detect when the client is talking to an older broker, which we don't have
> right now. This check should be straightforward, so we could do it now,
> which would resolve some of the uneasiness about having an unused feature
> which we depended on other clients to test for us. Does that make sense or
> not?
>
> -Jason
>
> On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh  wrote:
>
> > We have proposed and discussed majorly three approaches so far, there
> were
> > many minor versions with small variations. Comparing them really
> requires a
> > side by side proposal and their pros/cons, and I agree with others that
> > this has been lacking in the KIP. We just updated the KIP with following
> > details.
> >
> > 1. Provide proposed changes in all the three proposals we have discussed
> so
> > far. Except the current proposal, these proposals are in rejected
> > alternatives.
> > 2. Provide reasoning on why the rejected proposals were rejected.
> > 3. Add scenarios for all of these proposals from a client developer and
> > core Kafka developer point of view.
> >
> > As we are really close to 0.10 deadline, a quick round of voting will
> > really help. If you really do not like the idea, please feel free to say
> > so. If the vote fails for the current proposal, it can at lease provide
> > recommendations that we should consider for next version of proposal and
> > put it up for vote again for next release. However, as stated earlier by
> > multiple people having this ASAP will be awesome.
> >
> > On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers 
> > wrote:
> >
> > > On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira 
> wrote:
> > >
> > > > "I think I would make this approach work by looking at the released
> > > server
> > > > version documentation for each version that I am trying to support
> and
> > > test
> > > > against*, manually identify the expected "protocol vectors" each
> > > supports,
> > > > store that as a map of vectors to "broker versions", check each
> vector
> > at
> > > > runtime until I find a match, and write code compatibility checks
> from
> > > > there."
> > > >
> > > > How is this better than a global version ID?
> > >
> > >
> > > As a client developer, it seems roughly the same. I think it probably
> > > avoids the server development workflow issues, and possibly the need to
> > > agree on semantics of the global version ID? But others surely are more
> > > qualified than me to comment on that part.
> > >
> > > -Dana
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jason Gustafson]

Bumping this thread. I talked with Ashish and Magnus about this KIP offline
and I'm gradually coming over. The new API actually stands by itself
outside of the discussion about whether the client should support backwards
compatibility or not. For the Java client, we could continue to support the
current compatibility approach in which the client supports only brokers
with the same version or greater. In that case, we would use this API only
to assert that the current API versions are all supported, and raise an
exception if they are not. This gives us the capability going forward to
detect when the client is talking to an older broker, which we don't have
right now. This check should be straightforward, so we could do it now,
which would resolve some of the uneasiness about having an unused feature
which we depended on other clients to test for us. Does that make sense or
not?

-Jason

On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh  wrote:

> We have proposed and discussed majorly three approaches so far, there were
> many minor versions with small variations. Comparing them really requires a
> side by side proposal and their pros/cons, and I agree with others that
> this has been lacking in the KIP. We just updated the KIP with following
> details.
>
> 1. Provide proposed changes in all the three proposals we have discussed so
> far. Except the current proposal, these proposals are in rejected
> alternatives.
> 2. Provide reasoning on why the rejected proposals were rejected.
> 3. Add scenarios for all of these proposals from a client developer and
> core Kafka developer point of view.
>
> As we are really close to 0.10 deadline, a quick round of voting will
> really help. If you really do not like the idea, please feel free to say
> so. If the vote fails for the current proposal, it can at lease provide
> recommendations that we should consider for next version of proposal and
> put it up for vote again for next release. However, as stated earlier by
> multiple people having this ASAP will be awesome.
>
> On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers 
> wrote:
>
> > On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira  wrote:
> >
> > > "I think I would make this approach work by looking at the released
> > server
> > > version documentation for each version that I am trying to support and
> > test
> > > against*, manually identify the expected "protocol vectors" each
> > supports,
> > > store that as a map of vectors to "broker versions", check each vector
> at
> > > runtime until I find a match, and write code compatibility checks from
> > > there."
> > >
> > > How is this better than a global version ID?
> >
> >
> > As a client developer, it seems roughly the same. I think it probably
> > avoids the server development workflow issues, and possibly the need to
> > agree on semantics of the global version ID? But others surely are more
> > qualified than me to comment on that part.
> >
> > -Dana
> >
>
>
>
> --
>
> Regards,
> Ashish
>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Dana Powers]

I like it. I think the revised KIP-35 approach is simple, easy to
understand, and workable. I think it will solve the main problem that I
have had, and I assume other 3rd party client devs have had as well, and
should open the door to more backwards-compatible drivers and less
fracturing of client development.

As much as I think a single broker version could be easier to work with,
the conference call discussion convinced me that my thinking of broker
version is not totally aligned with the server development workflow and the
design ideas of the folks that have thought about this much more deeply
than I have.

I think I would make this approach work by looking at the released server
version documentation for each version that I am trying to support and test
against*, manually identify the expected "protocol vectors" each supports,
store that as a map of vectors to "broker versions", check each vector at
runtime until I find a match, and write code compatibility checks from
there.  I admit that I am not sure whether this would work if I was trying
to support trunk deploys where the "fallback" vector may not be as clearly
defined. But I do think that if a client is targeting trunk it probably
isn't also targeting backwards compatibility, and so it likely would have
little reason to use this API anyways (unless for nicer error messages if
particular apis are unsupported on the target broker).

I think Magnus has much deeper insight into this than I do, and I wish I
had time to test a patch this weekend. Hopefully I will be able to get to
that next week. But I'm confident that if it works for librdkafka, it will
work for most if not all other 3rd party clients.

-Dana

* FWIW, every push / pull request to kafka-python gets automatically tested
against a number of public kafka releases via travis-ci. Currently we test
against 0.8.0, 0.8.1.1, 0.8.2.2, and 0.9.0.1 .

On Thu, Mar 17, 2016 at 12:28 PM, Magnus Edenhill 
wrote:

> I dont really see how first checking if the broker supports a certain set
> of API versions
> is so much more complex than calling the same set of API versions during
> operation?
> The only difference is that without the check we might only make it halfway
> through,
> i.e., being able to join a group, consume messages, but then fail to commit
> them
>
> Another concrete example is support for KIP-31 and KIP-32:
> How will a client with support for 0.8.0-0.10.0 know when to use the new
> Message format?
> If a client uses the old Message format (v0) the broker will need to
> convert message formats
> in the slow path which will increase the broker CPU usage and decrease its
> performance.
> Sure, clients can expose a configuration property to let the user enable
> this but most
> users will miss this (there are a lot of config properties already).
>
>
>
>
> 2016-03-17 19:43 GMT+01:00 Gwen Shapira :
>
> > My problem with per-api is that not all combinations are supported. So
> the
> > client will need to figure out which "safe" combination to revert to -
> > which is a lot like global numbers, but less linear...
> >
> >
> > On Thu, Mar 17, 2016 at 11:29 AM, Joel Koshy 
> wrote:
> >
> > > @Gwen - yes it did come back to per-API versions - I think that
> happened
> > > toward the end of the KIP hangout. I already said this, but personally
> > I'm
> > > also in favor of per-API versions. A global/aggregate API version also
> > > works but that is just a proxy to a vector of per-API versions and at
> the
> > > end of the day a client implementation will need to somehow derive
> *some*
> > > per-API version vector to talk to the broker. The other challenges
> (that
> > > I'm sure can be solved) with the global API version are automatically
> > > computing it in the first place or if set manually, automatically
> > verifying
> > > that it was correctly bumped; then there are the nuances of API
> evolution
> > > across multiple branches and thereby the need for some convention to
> > manage
> > > trees of aggregate API versions.
> > >
> > > Jason and Jay had brought up (earlier in this thread) some practical
> > > difficulties in using per-API versions by client implementations that I
> > > think is worth quoting verbatim to make sure people are aware of the
> > > implications:
> > >
> > > Maybe you could try to capture feature
> > > > compatibility by checking the versions for a subset of request types?
> > For
> > > > example, to ensure that you can use the new consumer API, you check
> > that
> > > > the group coordinator request is present, the offset commit request
> > > version
> > > > is greater than 2, the offset fetch request is greater than 1, and
> the
> > > join
> > > > group request is present. And to ensure compatibility with KIP-32,
> > maybe
> > > > you only need to check the appropriate versions of the fetch and
> > produce
> > > > requests. That sounds kind of complicated to keep track of and you
> > > probably
> > > > 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

We have proposed and discussed majorly three approaches so far, there were
many minor versions with small variations. Comparing them really requires a
side by side proposal and their pros/cons, and I agree with others that
this has been lacking in the KIP. We just updated the KIP with following
details.

1. Provide proposed changes in all the three proposals we have discussed so
far. Except the current proposal, these proposals are in rejected
alternatives.
2. Provide reasoning on why the rejected proposals were rejected.
3. Add scenarios for all of these proposals from a client developer and
core Kafka developer point of view.

As we are really close to 0.10 deadline, a quick round of voting will
really help. If you really do not like the idea, please feel free to say
so. If the vote fails for the current proposal, it can at lease provide
recommendations that we should consider for next version of proposal and
put it up for vote again for next release. However, as stated earlier by
multiple people having this ASAP will be awesome.

On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers  wrote:

> On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira  wrote:
>
> > "I think I would make this approach work by looking at the released
> server
> > version documentation for each version that I am trying to support and
> test
> > against*, manually identify the expected "protocol vectors" each
> supports,
> > store that as a map of vectors to "broker versions", check each vector at
> > runtime until I find a match, and write code compatibility checks from
> > there."
> >
> > How is this better than a global version ID?
>
>
> As a client developer, it seems roughly the same. I think it probably
> avoids the server development workflow issues, and possibly the need to
> agree on semantics of the global version ID? But others surely are more
> qualified than me to comment on that part.
>
> -Dana
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

"I think I would make this approach work by looking at the released server
version documentation for each version that I am trying to support and test
against*, manually identify the expected "protocol vectors" each supports,
store that as a map of vectors to "broker versions", check each vector at
runtime until I find a match, and write code compatibility checks from
there."

How is this better than a global version ID?

On Thu, Mar 17, 2016 at 1:31 PM, Dana Powers  wrote:

> I like it. I think the revised KIP-35 approach is simple, easy to
> understand, and workable. I think it will solve the main problem that I
> have had, and I assume other 3rd party client devs have had as well, and
> should open the door to more backwards-compatible drivers and less
> fracturing of client development.
>
> As much as I think a single broker version could be easier to work with,
> the conference call discussion convinced me that my thinking of broker
> version is not totally aligned with the server development workflow and the
> design ideas of the folks that have thought about this much more deeply
> than I have.
>
> I think I would make this approach work by looking at the released server
> version documentation for each version that I am trying to support and test
> against*, manually identify the expected "protocol vectors" each supports,
> store that as a map of vectors to "broker versions", check each vector at
> runtime until I find a match, and write code compatibility checks from
> there.  I admit that I am not sure whether this would work if I was trying
> to support trunk deploys where the "fallback" vector may not be as clearly
> defined. But I do think that if a client is targeting trunk it probably
> isn't also targeting backwards compatibility, and so it likely would have
> little reason to use this API anyways (unless for nicer error messages if
> particular apis are unsupported on the target broker).
>
> I think Magnus has much deeper insight into this than I do, and I wish I
> had time to test a patch this weekend. Hopefully I will be able to get to
> that next week. But I'm confident that if it works for librdkafka, it will
> work for most if not all other 3rd party clients.
>
> -Dana
>
> * FWIW, every push / pull request to kafka-python gets automatically tested
> against a number of public kafka releases via travis-ci. Currently we test
> against 0.8.0, 0.8.1.1, 0.8.2.2, and 0.9.0.1 .
>
> On Thu, Mar 17, 2016 at 12:28 PM, Magnus Edenhill 
> wrote:
>
> > I dont really see how first checking if the broker supports a certain set
> > of API versions
> > is so much more complex than calling the same set of API versions during
> > operation?
> > The only difference is that without the check we might only make it
> halfway
> > through,
> > i.e., being able to join a group, consume messages, but then fail to
> commit
> > them
> >
> > Another concrete example is support for KIP-31 and KIP-32:
> > How will a client with support for 0.8.0-0.10.0 know when to use the new
> > Message format?
> > If a client uses the old Message format (v0) the broker will need to
> > convert message formats
> > in the slow path which will increase the broker CPU usage and decrease
> its
> > performance.
> > Sure, clients can expose a configuration property to let the user enable
> > this but most
> > users will miss this (there are a lot of config properties already).
> >
> >
> >
> >
> > 2016-03-17 19:43 GMT+01:00 Gwen Shapira :
> >
> > > My problem with per-api is that not all combinations are supported. So
> > the
> > > client will need to figure out which "safe" combination to revert to -
> > > which is a lot like global numbers, but less linear...
> > >
> > >
> > > On Thu, Mar 17, 2016 at 11:29 AM, Joel Koshy 
> > wrote:
> > >
> > > > @Gwen - yes it did come back to per-API versions - I think that
> > happened
> > > > toward the end of the KIP hangout. I already said this, but
> personally
> > > I'm
> > > > also in favor of per-API versions. A global/aggregate API version
> also
> > > > works but that is just a proxy to a vector of per-API versions and at
> > the
> > > > end of the day a client implementation will need to somehow derive
> > *some*
> > > > per-API version vector to talk to the broker. The other challenges
> > (that
> > > > I'm sure can be solved) with the global API version are automatically
> > > > computing it in the first place or if set manually, automatically
> > > verifying
> > > > that it was correctly bumped; then there are the nuances of API
> > evolution
> > > > across multiple branches and thereby the need for some convention to
> > > manage
> > > > trees of aggregate API versions.
> > > >
> > > > Jason and Jay had brought up (earlier in this thread) some practical
> > > > difficulties in using per-API versions by client implementations
> that I
> > > > think is worth quoting verbatim to make sure people are aware 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Dana Powers]

On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira  wrote:

> "I think I would make this approach work by looking at the released server
> version documentation for each version that I am trying to support and test
> against*, manually identify the expected "protocol vectors" each supports,
> store that as a map of vectors to "broker versions", check each vector at
> runtime until I find a match, and write code compatibility checks from
> there."
>
> How is this better than a global version ID?


As a client developer, it seems roughly the same. I think it probably
avoids the server development workflow issues, and possibly the need to
agree on semantics of the global version ID? But others surely are more
qualified than me to comment on that part.

-Dana

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Jay, I think you are correct. Once a client gets the metadata from a
broker, it does not really need to get brokers and topics info from other
brokers again. However, with the current suggestion the client will still
have to send metadata request to get broker's supported protocols. Client
can send metadata request with topics as null to avoid getting redundant
topics info, but it will still get brokers info. Due to this redundancy of
information and the fact that not all clients would want to have supported
protocols information in the metadata response, I agree that it is better
to have a separate request/response.

I quickly checked with Magnus and he too seems fine with this change. Do
you think with this change, the proposal will seem reasonable? If so, I can
update the KIP.

On Wed, Mar 16, 2016 at 6:46 AM, Jay Kreps  wrote:

> Yeah, Magnus, I'm not arguing for either approach (1) or approach (2),
> I'm saying the only reason it would make sense to combine the version
> check and the metadata request would be if the version check was
> cluster wide like the other metadata. That was where that idea
> originated and if you make the check per-connection it no longer makes
> sense, right?
>
> -Jay
>
> On Wed, Mar 16, 2016 at 12:36 AM, Magnus Edenhill 
> wrote:
> > 1. But there is no way of conveying version mismatch in the protocol,
> that's
> > one of the reasons for this KIP in the first place :)
> > Even if there was (e.g., the empty response hack) it makes the client
> > implementation
> > more complex since the cached cluster-level version support returned by
> the
> > broker
> > can't really be trusted so then there is an initial "We can probably use
> > this version"
> > state followed by a "phew, request succeeded" or "darn, seems the request
> > wasn't supported" state.
> >
> > Additionally these late errors are going to be a problem in some cases,
> > let's say
> > a certain client really needs OffsetCommit v9, so it starts a consumer,
> > processes
> > a bunch of messages and then attempts to commit the latest processed
> offset
> > only to find
> > broker didnt support v9 thus failing the commit. We then have a situation
> > where
> > messages were processed but they can't be committed, thus resulting in
> > multiple
> > processing, or likewise.
> >
> > Re caching:
> > A couple of years back we had the discussion on cached cluster ISR info
> in
> > the
> > Metadata response, since the information was cached it couldn't be
> trusted
> > and was thus
> > deemed useless. Let's not make that mistake again.
> >
> > Re in Metadata or not:
> > I'm personally fine with having a specific request to query the current
> > broker's API support,
> > the client will fire off both MetadataRequests + ApiQueryRequest back to
> > back so there is
> > no latency penalty. This is a cleaner solution and I'm a weak +1 for it
> > unless other client devs oppose.
> >
> >
> > /Magnus
> >
> >
> > 2016-03-16 0:37 GMT+01:00 Jay Kreps :
> >
> >> Yeah I think there are two possible approaches:
> >> 1. You get the versions in the metadata request and cache them and
> >> invalidate that cache if you get a version mismatch error (basically
> >> as we do with leadership information).
> >> 2. You check each connection
> >>
> >> I think combining metadata request and version check only makes sense
> >> in (1), right? If it is (2) I don't see how you save anything and the
> >> requests don't really make sense because you're mixing cluster wide
> >> state about partitions with info about the answering broker.
> >>
> >> -Jay
> >>
> >> On Tue, Mar 15, 2016 at 4:25 PM, Magnus Edenhill 
> >> wrote:
> >> > Hey Jay,
> >> >
> >> > as discussed earlier it is not safe to cache/relay a broker's version
> or
> >> > its supported API versions,
> >> > by the time the client connects the broker might have upgraded to
> another
> >> > version which effectively
> >> > makes this information useless in a cached form.
> >> >
> >> > The complexity of querying for protocol verion is very implementation
> >> > dependent and
> >> > hard to generalize on, I dont foresee any bigger problems adding
> support
> >> > for an extra protocol version
> >> > querying state in librdkafka, but other client devs should chime in.
> >> > There are already post-connect,pre-operation states for dealing with
> SSL
> >> > and SASL.
> >> >
> >> > The reason for putting the API versioning stuff in the Metadata
> request
> >> is
> >> > that it is already used
> >> > for bootstrapping a client and/or connection and thus saves us a
> >> round-trip
> >> > (and possibly a state).
> >> >
> >> >
> >> > For how this will be used; I can't speak for other client devs but
> aim to
> >> > make a mapping between
> >> > the features my client exposes to a set of specific APIs and their
> >> minimum
> >> > version..
> >> > E.g.: Balanced consumer groups requires JoinGroup >= V0, LeaveGroup >=
> >> V0,
> >> > 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

My problem with per-api is that not all combinations are supported. So the
client will need to figure out which "safe" combination to revert to -
which is a lot like global numbers, but less linear...


On Thu, Mar 17, 2016 at 11:29 AM, Joel Koshy  wrote:

> @Gwen - yes it did come back to per-API versions - I think that happened
> toward the end of the KIP hangout. I already said this, but personally I'm
> also in favor of per-API versions. A global/aggregate API version also
> works but that is just a proxy to a vector of per-API versions and at the
> end of the day a client implementation will need to somehow derive *some*
> per-API version vector to talk to the broker. The other challenges (that
> I'm sure can be solved) with the global API version are automatically
> computing it in the first place or if set manually, automatically verifying
> that it was correctly bumped; then there are the nuances of API evolution
> across multiple branches and thereby the need for some convention to manage
> trees of aggregate API versions.
>
> Jason and Jay had brought up (earlier in this thread) some practical
> difficulties in using per-API versions by client implementations that I
> think is worth quoting verbatim to make sure people are aware of the
> implications:
>
> Maybe you could try to capture feature
> > compatibility by checking the versions for a subset of request types? For
> > example, to ensure that you can use the new consumer API, you check that
> > the group coordinator request is present, the offset commit request
> version
> > is greater than 2, the offset fetch request is greater than 1, and the
> join
> > group request is present. And to ensure compatibility with KIP-32, maybe
> > you only need to check the appropriate versions of the fetch and produce
> > requests. That sounds kind of complicated to keep track of and you
> probably
> > end up trying to handle combinations which aren't even possible in
> > practice.
> >
>
> Unless I'm missing something I don't recollect any other significant
> drawbacks to per-API versions or are there any?
>
> For the above concern I think indicating support for certain features
> ultimately has to be accomplished one way or the other by documenting what
> per-API request version ranges need to be used. This is obviously more
> tedious for more complex client-server interactions such as the new
> consumer's rebalance process; but not so much for other features such as
> KIP-32; or reading throttle time in produce/fetch responses that were
> delayed due to quota violations.
>
> On Thu, Mar 17, 2016 at 1:50 AM, Magnus Edenhill 
> wrote:
>
> > I'm sorry for the confusion but I was referring to a simplified version
> of
> > the KIP
> > that I'd failed to document on the wiki yet, my bad!
> >
> > The now updated KIP leaves the existing MetadataRequest untouched and
> > instead
> > adds a specific API to query the broker for supported APIs:
> > ApiVersionQueryRequest.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> >
> > With all the new protocol additions of the upcoming 0.10.0 release it is
> > imperative
> > to get this support in now, so let's be constructive, pragmatic and most
> > importantly quick
> > in commenting on this latest proposal so we can start a voting thread
> ASAP.
> >
> > Ashish will have the patch ready within a day and librdkafka will be used
> > to verify the functionality,
> > we should have a technical green light before the weekend. Additional
> > client devs
> > are very welcome to join in.
> >
> > I'll send a message to kafka-clients to pull in any 3rd party client devs
> > that have missed
> > the recent uprise in discussion.
> >
> >
> > /Magnus
> >
> > 2016-03-17 5:21 GMT+01:00 Ewen Cheslack-Postava :
> >
> > > I'm a fan of separating feature support/versions into a separate
> request
> > > from metadata -- they're different requests and I don't see how the
> > current
> > > model of metadata can properly map to the request version info we need.
> > The
> > > complexity of support differing across brokers seems like something
> > clients
> > > need to handle regardless of any solution we come up with here (current
> > > auto detection approaches are vulnerable to similar issues).
> > >
> > > More importantly -- has this KIP ever made it over to the kafka-clients
> > > list, or have we reached out to other client library developers? My
> brief
> > > scan of the list suggests it hasn't, but maybe I missed something.
> We've
> > > got Magnus & Dana participating, which is great and accounts for some
> > > really important clients, but have we reached out as broadly as we can
> to
> > > other client developers to get feedback? We're largely operating in an
> > echo
> > > chamber, and some of the current challenges might trace back to that
> mode
> > > of operation -- the Java client, which has the benefit of development,
> 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Magnus Edenhill]

I dont really see how first checking if the broker supports a certain set
of API versions
is so much more complex than calling the same set of API versions during
operation?
The only difference is that without the check we might only make it halfway
through,
i.e., being able to join a group, consume messages, but then fail to commit
them

Another concrete example is support for KIP-31 and KIP-32:
How will a client with support for 0.8.0-0.10.0 know when to use the new
Message format?
If a client uses the old Message format (v0) the broker will need to
convert message formats
in the slow path which will increase the broker CPU usage and decrease its
performance.
Sure, clients can expose a configuration property to let the user enable
this but most
users will miss this (there are a lot of config properties already).




2016-03-17 19:43 GMT+01:00 Gwen Shapira :

> My problem with per-api is that not all combinations are supported. So the
> client will need to figure out which "safe" combination to revert to -
> which is a lot like global numbers, but less linear...
>
>
> On Thu, Mar 17, 2016 at 11:29 AM, Joel Koshy  wrote:
>
> > @Gwen - yes it did come back to per-API versions - I think that happened
> > toward the end of the KIP hangout. I already said this, but personally
> I'm
> > also in favor of per-API versions. A global/aggregate API version also
> > works but that is just a proxy to a vector of per-API versions and at the
> > end of the day a client implementation will need to somehow derive *some*
> > per-API version vector to talk to the broker. The other challenges (that
> > I'm sure can be solved) with the global API version are automatically
> > computing it in the first place or if set manually, automatically
> verifying
> > that it was correctly bumped; then there are the nuances of API evolution
> > across multiple branches and thereby the need for some convention to
> manage
> > trees of aggregate API versions.
> >
> > Jason and Jay had brought up (earlier in this thread) some practical
> > difficulties in using per-API versions by client implementations that I
> > think is worth quoting verbatim to make sure people are aware of the
> > implications:
> >
> > Maybe you could try to capture feature
> > > compatibility by checking the versions for a subset of request types?
> For
> > > example, to ensure that you can use the new consumer API, you check
> that
> > > the group coordinator request is present, the offset commit request
> > version
> > > is greater than 2, the offset fetch request is greater than 1, and the
> > join
> > > group request is present. And to ensure compatibility with KIP-32,
> maybe
> > > you only need to check the appropriate versions of the fetch and
> produce
> > > requests. That sounds kind of complicated to keep track of and you
> > probably
> > > end up trying to handle combinations which aren't even possible in
> > > practice.
> > >
> >
> > Unless I'm missing something I don't recollect any other significant
> > drawbacks to per-API versions or are there any?
> >
> > For the above concern I think indicating support for certain features
> > ultimately has to be accomplished one way or the other by documenting
> what
> > per-API request version ranges need to be used. This is obviously more
> > tedious for more complex client-server interactions such as the new
> > consumer's rebalance process; but not so much for other features such as
> > KIP-32; or reading throttle time in produce/fetch responses that were
> > delayed due to quota violations.
> >
> > On Thu, Mar 17, 2016 at 1:50 AM, Magnus Edenhill 
> > wrote:
> >
> > > I'm sorry for the confusion but I was referring to a simplified version
> > of
> > > the KIP
> > > that I'd failed to document on the wiki yet, my bad!
> > >
> > > The now updated KIP leaves the existing MetadataRequest untouched and
> > > instead
> > > adds a specific API to query the broker for supported APIs:
> > > ApiVersionQueryRequest.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> > >
> > > With all the new protocol additions of the upcoming 0.10.0 release it
> is
> > > imperative
> > > to get this support in now, so let's be constructive, pragmatic and
> most
> > > importantly quick
> > > in commenting on this latest proposal so we can start a voting thread
> > ASAP.
> > >
> > > Ashish will have the patch ready within a day and librdkafka will be
> used
> > > to verify the functionality,
> > > we should have a technical green light before the weekend. Additional
> > > client devs
> > > are very welcome to join in.
> > >
> > > I'll send a message to kafka-clients to pull in any 3rd party client
> devs
> > > that have missed
> > > the recent uprise in discussion.
> > >
> > >
> > > /Magnus
> > >
> > > 2016-03-17 5:21 GMT+01:00 Ewen Cheslack-Postava :
> > >
> > > > I'm a fan of separating feature 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

I was a bit surprised to discover that the latest KIP includes an array of
versions per request. Especially since we spent an hour discussing global
APIs and their tradeoffs.

I'm pointing this out in case anyone else missed that - to make sure we are
all on the same page in this discussion.

Gwen

On Tue, Mar 15, 2016 at 10:41 PM, Ashish Singh  wrote:

> Sent incomplete message by mistake.
>
> I do not have a preference for (1) or (2). (1) basically makes us view
> supported versions for all brokers as a part of cluster wide state, however
> (2) lets client deal with individual brokers. From usage perspective,
> clients would need to choose a certain feature or not based on minimal
> supported version across all brokers they are talking to. In (1) this state
> will be maintained and provided by server, however in (2) the onus will be
> on client to maintain the state. What do you suggest will be the right
> thing to do here?
>
>
> On Tue, Mar 15, 2016 at 10:27 PM, Ashish Singh 
> wrote:
>
> > I do not have a preference for 1 or 2. 1 basically makes us view
> supported
> > versions for all brokers
> >
> > On Tuesday, March 15, 2016, Jay Kreps  wrote:
> >
> > > Yeah I think there are two possible approaches:
> > > 1. You get the versions in the metadata request and cache them and
> > > invalidate that cache if you get a version mismatch error (basically
> > > as we do with leadership information).
> > > 2. You check each connection
> > >
> > > I think combining metadata request and version check only makes sense
> > > in (1), right? If it is (2) I don't see how you save anything and the
> > > requests don't really make sense because you're mixing cluster wide
> > > state about partitions with info about the answering broker.
> >
> >
> > > -Jay
> > >
> > > On Tue, Mar 15, 2016 at 4:25 PM, Magnus Edenhill  > > > wrote:
> > > > Hey Jay,
> > > >
> > > > as discussed earlier it is not safe to cache/relay a broker's version
> > or
> > > > its supported API versions,
> > > > by the time the client connects the broker might have upgraded to
> > another
> > > > version which effectively
> > > > makes this information useless in a cached form.
> > > >
> > > > The complexity of querying for protocol verion is very implementation
> > > > dependent and
> > > > hard to generalize on, I dont foresee any bigger problems adding
> > support
> > > > for an extra protocol version
> > > > querying state in librdkafka, but other client devs should chime in.
> > > > There are already post-connect,pre-operation states for dealing with
> > SSL
> > > > and SASL.
> > > >
> > > > The reason for putting the API versioning stuff in the Metadata
> request
> > > is
> > > > that it is already used
> > > > for bootstrapping a client and/or connection and thus saves us a
> > > round-trip
> > > > (and possibly a state).
> > > >
> > > >
> > > > For how this will be used; I can't speak for other client devs but
> aim
> > to
> > > > make a mapping between
> > > > the features my client exposes to a set of specific APIs and their
> > > minimum
> > > > version..
> > > > E.g.: Balanced consumer groups requires JoinGroup >= V0, LeaveGroup
> >=
> > > V0,
> > > > SyncGroup >= V0, and so on.
> > > > If those requirements can be fullfilled then the feature is enabled,
> > > > otherwise an error is returned to the user.
> > > >
> > > > /Magnus
> > > >
> > > >
> > > > 2016-03-15 23:35 GMT+01:00 Jay Kreps  > >:
> > > >
> > > >> Hey Ashish,
> > > >>
> > > >> Can you expand in the proposal on how this would be used by clients?
> > > >> This proposal only has one slot for api versions, though in fact
> there
> > > >> is potentially a different version on each broker. I think the
> > > >> proposal is that every single time the client establishes a
> connection
> > > >> it would then need to issue a metadata request on that connection to
> > > >> check supported versions. Is that correct?
> > > >>
> > > >> The point of merging version information with metadata request was
> > > >> that the client wouldn't have to manage this additional state for
> each
> > > >> connection, but rather the broker would gather the information and
> > > >> give a summary of all brokers in the cluster. (Managing the state
> > > >> doesn't seem complex but actually since the full state machine for a
> > > >> request is something like begin connecting=>connection
> complete=>begin
> > > >> sending request=>do work sending=>await response=>do work reading
> > > >> response adding to the state machine around this is not as simple as
> > > >> it seems...you can see the code in the java client around this).
> > > >>
> > > >> It sounds like in this proposal you are proposing merging with the
> > > >> metadata request but not summarizing across the cluster? Can you
> > > >> explain the thinking vs a separate request?
> > > >>
> > > >> It would really be good if 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jay Kreps]

Yeah, Magnus, I'm not arguing for either approach (1) or approach (2),
I'm saying the only reason it would make sense to combine the version
check and the metadata request would be if the version check was
cluster wide like the other metadata. That was where that idea
originated and if you make the check per-connection it no longer makes
sense, right?

-Jay

On Wed, Mar 16, 2016 at 12:36 AM, Magnus Edenhill  wrote:
> 1. But there is no way of conveying version mismatch in the protocol, that's
> one of the reasons for this KIP in the first place :)
> Even if there was (e.g., the empty response hack) it makes the client
> implementation
> more complex since the cached cluster-level version support returned by the
> broker
> can't really be trusted so then there is an initial "We can probably use
> this version"
> state followed by a "phew, request succeeded" or "darn, seems the request
> wasn't supported" state.
>
> Additionally these late errors are going to be a problem in some cases,
> let's say
> a certain client really needs OffsetCommit v9, so it starts a consumer,
> processes
> a bunch of messages and then attempts to commit the latest processed offset
> only to find
> broker didnt support v9 thus failing the commit. We then have a situation
> where
> messages were processed but they can't be committed, thus resulting in
> multiple
> processing, or likewise.
>
> Re caching:
> A couple of years back we had the discussion on cached cluster ISR info in
> the
> Metadata response, since the information was cached it couldn't be trusted
> and was thus
> deemed useless. Let's not make that mistake again.
>
> Re in Metadata or not:
> I'm personally fine with having a specific request to query the current
> broker's API support,
> the client will fire off both MetadataRequests + ApiQueryRequest back to
> back so there is
> no latency penalty. This is a cleaner solution and I'm a weak +1 for it
> unless other client devs oppose.
>
>
> /Magnus
>
>
> 2016-03-16 0:37 GMT+01:00 Jay Kreps :
>
>> Yeah I think there are two possible approaches:
>> 1. You get the versions in the metadata request and cache them and
>> invalidate that cache if you get a version mismatch error (basically
>> as we do with leadership information).
>> 2. You check each connection
>>
>> I think combining metadata request and version check only makes sense
>> in (1), right? If it is (2) I don't see how you save anything and the
>> requests don't really make sense because you're mixing cluster wide
>> state about partitions with info about the answering broker.
>>
>> -Jay
>>
>> On Tue, Mar 15, 2016 at 4:25 PM, Magnus Edenhill 
>> wrote:
>> > Hey Jay,
>> >
>> > as discussed earlier it is not safe to cache/relay a broker's version or
>> > its supported API versions,
>> > by the time the client connects the broker might have upgraded to another
>> > version which effectively
>> > makes this information useless in a cached form.
>> >
>> > The complexity of querying for protocol verion is very implementation
>> > dependent and
>> > hard to generalize on, I dont foresee any bigger problems adding support
>> > for an extra protocol version
>> > querying state in librdkafka, but other client devs should chime in.
>> > There are already post-connect,pre-operation states for dealing with SSL
>> > and SASL.
>> >
>> > The reason for putting the API versioning stuff in the Metadata request
>> is
>> > that it is already used
>> > for bootstrapping a client and/or connection and thus saves us a
>> round-trip
>> > (and possibly a state).
>> >
>> >
>> > For how this will be used; I can't speak for other client devs but aim to
>> > make a mapping between
>> > the features my client exposes to a set of specific APIs and their
>> minimum
>> > version..
>> > E.g.: Balanced consumer groups requires JoinGroup >= V0, LeaveGroup >=
>> V0,
>> > SyncGroup >= V0, and so on.
>> > If those requirements can be fullfilled then the feature is enabled,
>> > otherwise an error is returned to the user.
>> >
>> > /Magnus
>> >
>> >
>> > 2016-03-15 23:35 GMT+01:00 Jay Kreps :
>> >
>> >> Hey Ashish,
>> >>
>> >> Can you expand in the proposal on how this would be used by clients?
>> >> This proposal only has one slot for api versions, though in fact there
>> >> is potentially a different version on each broker. I think the
>> >> proposal is that every single time the client establishes a connection
>> >> it would then need to issue a metadata request on that connection to
>> >> check supported versions. Is that correct?
>> >>
>> >> The point of merging version information with metadata request was
>> >> that the client wouldn't have to manage this additional state for each
>> >> connection, but rather the broker would gather the information and
>> >> give a summary of all brokers in the cluster. (Managing the state
>> >> doesn't seem complex but actually since the full state machine for a
>> >> request is 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ewen Cheslack-Postava]

I'm a fan of separating feature support/versions into a separate request
from metadata -- they're different requests and I don't see how the current
model of metadata can properly map to the request version info we need. The
complexity of support differing across brokers seems like something clients
need to handle regardless of any solution we come up with here (current
auto detection approaches are vulnerable to similar issues).

More importantly -- has this KIP ever made it over to the kafka-clients
list, or have we reached out to other client library developers? My brief
scan of the list suggests it hasn't, but maybe I missed something. We've
got Magnus & Dana participating, which is great and accounts for some
really important clients, but have we reached out as broadly as we can to
other client developers to get feedback? We're largely operating in an echo
chamber, and some of the current challenges might trace back to that mode
of operation -- the Java client, which has the benefit of development,
testing, and release alongside the broker, doesn't follow the same
compatibility and release approach as most other clients.

I'm happy to help with that outreach and bring feedback back into this
discussion, but I think this will end up being longer process than we can
resolve in the next few days...

-Ewen


On Wed, Mar 16, 2016 at 5:22 PM, Gwen Shapira  wrote:

> I was a bit surprised to discover that the latest KIP includes an array of
> versions per request. Especially since we spent an hour discussing global
> APIs and their tradeoffs.
>
> I'm pointing this out in case anyone else missed that - to make sure we are
> all on the same page in this discussion.
>
> Gwen
>
> On Tue, Mar 15, 2016 at 10:41 PM, Ashish Singh 
> wrote:
>
> > Sent incomplete message by mistake.
> >
> > I do not have a preference for (1) or (2). (1) basically makes us view
> > supported versions for all brokers as a part of cluster wide state,
> however
> > (2) lets client deal with individual brokers. From usage perspective,
> > clients would need to choose a certain feature or not based on minimal
> > supported version across all brokers they are talking to. In (1) this
> state
> > will be maintained and provided by server, however in (2) the onus will
> be
> > on client to maintain the state. What do you suggest will be the right
> > thing to do here?
> >
> >
> > On Tue, Mar 15, 2016 at 10:27 PM, Ashish Singh 
> > wrote:
> >
> > > I do not have a preference for 1 or 2. 1 basically makes us view
> > supported
> > > versions for all brokers
> > >
> > > On Tuesday, March 15, 2016, Jay Kreps  wrote:
> > >
> > > > Yeah I think there are two possible approaches:
> > > > 1. You get the versions in the metadata request and cache them and
> > > > invalidate that cache if you get a version mismatch error (basically
> > > > as we do with leadership information).
> > > > 2. You check each connection
> > > >
> > > > I think combining metadata request and version check only makes sense
> > > > in (1), right? If it is (2) I don't see how you save anything and the
> > > > requests don't really make sense because you're mixing cluster wide
> > > > state about partitions with info about the answering broker.
> > >
> > >
> > > > -Jay
> > > >
> > > > On Tue, Mar 15, 2016 at 4:25 PM, Magnus Edenhill  > > > > wrote:
> > > > > Hey Jay,
> > > > >
> > > > > as discussed earlier it is not safe to cache/relay a broker's
> version
> > > or
> > > > > its supported API versions,
> > > > > by the time the client connects the broker might have upgraded to
> > > another
> > > > > version which effectively
> > > > > makes this information useless in a cached form.
> > > > >
> > > > > The complexity of querying for protocol verion is very
> implementation
> > > > > dependent and
> > > > > hard to generalize on, I dont foresee any bigger problems adding
> > > support
> > > > > for an extra protocol version
> > > > > querying state in librdkafka, but other client devs should chime
> in.
> > > > > There are already post-connect,pre-operation states for dealing
> with
> > > SSL
> > > > > and SASL.
> > > > >
> > > > > The reason for putting the API versioning stuff in the Metadata
> > request
> > > > is
> > > > > that it is already used
> > > > > for bootstrapping a client and/or connection and thus saves us a
> > > > round-trip
> > > > > (and possibly a state).
> > > > >
> > > > >
> > > > > For how this will be used; I can't speak for other client devs but
> > aim
> > > to
> > > > > make a mapping between
> > > > > the features my client exposes to a set of specific APIs and their
> > > > minimum
> > > > > version..
> > > > > E.g.: Balanced consumer groups requires JoinGroup >= V0, LeaveGroup
> > >=
> > > > V0,
> > > > > SyncGroup >= V0, and so on.
> > > > > If those requirements can be fullfilled then the feature is
> enabled,
> > > > > otherwise an 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

>
>
> Anyway, I'm not particularly strong in favor of either approach, but I
> think what's concerning Gwen (and myself) is that we're going down to the
> wire with a significant decision and the KIP itself seems different every
> time we look at it. And still a little incomplete. It would be nice, for
> example, if the rejected alternatives explained why each alternative was
> rejected. It would also be nice to discuss the way the current client
> versioning works and why it is inadequate for third-party clients.
>

+1


>
> Thanks,
> Jason
>
> On Thu, Mar 17, 2016 at 12:28 PM, Magnus Edenhill 
> wrote:
>
> > I dont really see how first checking if the broker supports a certain set
> > of API versions
> > is so much more complex than calling the same set of API versions during
> > operation?
> > The only difference is that without the check we might only make it
> halfway
> > through,
> > i.e., being able to join a group, consume messages, but then fail to
> commit
> > them
> >
> > Another concrete example is support for KIP-31 and KIP-32:
> > How will a client with support for 0.8.0-0.10.0 know when to use the new
> > Message format?
> > If a client uses the old Message format (v0) the broker will need to
> > convert message formats
> > in the slow path which will increase the broker CPU usage and decrease
> its
> > performance.
> > Sure, clients can expose a configuration property to let the user enable
> > this but most
> > users will miss this (there are a lot of config properties already).
> >
> >
> >
> >
> > 2016-03-17 19:43 GMT+01:00 Gwen Shapira :
> >
> > > My problem with per-api is that not all combinations are supported. So
> > the
> > > client will need to figure out which "safe" combination to revert to -
> > > which is a lot like global numbers, but less linear...
> > >
> > >
> > > On Thu, Mar 17, 2016 at 11:29 AM, Joel Koshy 
> > wrote:
> > >
> > > > @Gwen - yes it did come back to per-API versions - I think that
> > happened
> > > > toward the end of the KIP hangout. I already said this, but
> personally
> > > I'm
> > > > also in favor of per-API versions. A global/aggregate API version
> also
> > > > works but that is just a proxy to a vector of per-API versions and at
> > the
> > > > end of the day a client implementation will need to somehow derive
> > *some*
> > > > per-API version vector to talk to the broker. The other challenges
> > (that
> > > > I'm sure can be solved) with the global API version are automatically
> > > > computing it in the first place or if set manually, automatically
> > > verifying
> > > > that it was correctly bumped; then there are the nuances of API
> > evolution
> > > > across multiple branches and thereby the need for some convention to
> > > manage
> > > > trees of aggregate API versions.
> > > >
> > > > Jason and Jay had brought up (earlier in this thread) some practical
> > > > difficulties in using per-API versions by client implementations
> that I
> > > > think is worth quoting verbatim to make sure people are aware of the
> > > > implications:
> > > >
> > > > Maybe you could try to capture feature
> > > > > compatibility by checking the versions for a subset of request
> types?
> > > For
> > > > > example, to ensure that you can use the new consumer API, you check
> > > that
> > > > > the group coordinator request is present, the offset commit request
> > > > version
> > > > > is greater than 2, the offset fetch request is greater than 1, and
> > the
> > > > join
> > > > > group request is present. And to ensure compatibility with KIP-32,
> > > maybe
> > > > > you only need to check the appropriate versions of the fetch and
> > > produce
> > > > > requests. That sounds kind of complicated to keep track of and you
> > > > probably
> > > > > end up trying to handle combinations which aren't even possible in
> > > > > practice.
> > > > >
> > > >
> > > > Unless I'm missing something I don't recollect any other significant
> > > > drawbacks to per-API versions or are there any?
> > > >
> > > > For the above concern I think indicating support for certain features
> > > > ultimately has to be accomplished one way or the other by documenting
> > > what
> > > > per-API request version ranges need to be used. This is obviously
> more
> > > > tedious for more complex client-server interactions such as the new
> > > > consumer's rebalance process; but not so much for other features such
> > as
> > > > KIP-32; or reading throttle time in produce/fetch responses that were
> > > > delayed due to quota violations.
> > > >
> > > > On Thu, Mar 17, 2016 at 1:50 AM, Magnus Edenhill  >
> > > > wrote:
> > > >
> > > > > I'm sorry for the confusion but I was referring to a simplified
> > version
> > > > of
> > > > > the KIP
> > > > > that I'd failed to document on the wiki yet, my bad!
> > > > >
> > > > > The now updated KIP leaves the existing MetadataRequest untouched
> and
> > > > > 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Joel Koshy]

@Gwen - yes it did come back to per-API versions - I think that happened
toward the end of the KIP hangout. I already said this, but personally I'm
also in favor of per-API versions. A global/aggregate API version also
works but that is just a proxy to a vector of per-API versions and at the
end of the day a client implementation will need to somehow derive *some*
per-API version vector to talk to the broker. The other challenges (that
I'm sure can be solved) with the global API version are automatically
computing it in the first place or if set manually, automatically verifying
that it was correctly bumped; then there are the nuances of API evolution
across multiple branches and thereby the need for some convention to manage
trees of aggregate API versions.

Jason and Jay had brought up (earlier in this thread) some practical
difficulties in using per-API versions by client implementations that I
think is worth quoting verbatim to make sure people are aware of the
implications:

Maybe you could try to capture feature
> compatibility by checking the versions for a subset of request types? For
> example, to ensure that you can use the new consumer API, you check that
> the group coordinator request is present, the offset commit request version
> is greater than 2, the offset fetch request is greater than 1, and the join
> group request is present. And to ensure compatibility with KIP-32, maybe
> you only need to check the appropriate versions of the fetch and produce
> requests. That sounds kind of complicated to keep track of and you probably
> end up trying to handle combinations which aren't even possible in
> practice.
>

Unless I'm missing something I don't recollect any other significant
drawbacks to per-API versions or are there any?

For the above concern I think indicating support for certain features
ultimately has to be accomplished one way or the other by documenting what
per-API request version ranges need to be used. This is obviously more
tedious for more complex client-server interactions such as the new
consumer's rebalance process; but not so much for other features such as
KIP-32; or reading throttle time in produce/fetch responses that were
delayed due to quota violations.

On Thu, Mar 17, 2016 at 1:50 AM, Magnus Edenhill  wrote:

> I'm sorry for the confusion but I was referring to a simplified version of
> the KIP
> that I'd failed to document on the wiki yet, my bad!
>
> The now updated KIP leaves the existing MetadataRequest untouched and
> instead
> adds a specific API to query the broker for supported APIs:
> ApiVersionQueryRequest.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
>
> With all the new protocol additions of the upcoming 0.10.0 release it is
> imperative
> to get this support in now, so let's be constructive, pragmatic and most
> importantly quick
> in commenting on this latest proposal so we can start a voting thread ASAP.
>
> Ashish will have the patch ready within a day and librdkafka will be used
> to verify the functionality,
> we should have a technical green light before the weekend. Additional
> client devs
> are very welcome to join in.
>
> I'll send a message to kafka-clients to pull in any 3rd party client devs
> that have missed
> the recent uprise in discussion.
>
>
> /Magnus
>
> 2016-03-17 5:21 GMT+01:00 Ewen Cheslack-Postava :
>
> > I'm a fan of separating feature support/versions into a separate request
> > from metadata -- they're different requests and I don't see how the
> current
> > model of metadata can properly map to the request version info we need.
> The
> > complexity of support differing across brokers seems like something
> clients
> > need to handle regardless of any solution we come up with here (current
> > auto detection approaches are vulnerable to similar issues).
> >
> > More importantly -- has this KIP ever made it over to the kafka-clients
> > list, or have we reached out to other client library developers? My brief
> > scan of the list suggests it hasn't, but maybe I missed something. We've
> > got Magnus & Dana participating, which is great and accounts for some
> > really important clients, but have we reached out as broadly as we can to
> > other client developers to get feedback? We're largely operating in an
> echo
> > chamber, and some of the current challenges might trace back to that mode
> > of operation -- the Java client, which has the benefit of development,
> > testing, and release alongside the broker, doesn't follow the same
> > compatibility and release approach as most other clients.
> >
> > I'm happy to help with that outreach and bring feedback back into this
> > discussion, but I think this will end up being longer process than we can
> > resolve in the next few days...
> >
> > -Ewen
> >
> >
> > On Wed, Mar 16, 2016 at 5:22 PM, Gwen Shapira  wrote:
> >
> > > I was a bit surprised to discover that the latest 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Dana Powers]

On Thu, Mar 17, 2016 at 2:07 PM, Jason Gustafson  wrote:

> It would also be nice to discuss the way the current client
> versioning works and why it is inadequate for third-party clients.
>

My understanding of the java client versioning is that it is not
backwards-compatible. Instructions are to use a java client that is <= your
broker version.

The main inadequacy of this model is that some clients want to deliver
backwards-compatible client software. I assume that doesn't need an
explanation, but these are my reasons anyways: it makes users happier,
reduces maintenance burden, and avoids unnecessary client fragmentation.

The other inadequacy, assuming you did want to follow the same practice as
the java client, is that third-party clients are not versioned in lockstep
with the broker, so we cannot use a similar policy of "use a client version
that is <= your broker version". Instead, third-party clients must manage
and communicate the relationships between client versions and broker
versions independently. That is difficult and leads to lots of user
confusion.

-Dana

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jason Gustafson]

My point was only that the way people would probably end up using the
request for all versions as a proxy for determining the broker version. For
example, if the JoinGroup v0 is present, then it must be 0.9. Or if you do
it in terms of features, then you need to check 5 or 6 different request
types to verify that the coordination protocol was supported. So a single
version seemed simpler and closer to what (I guessed) clients would
actually be able to code against. That said, it doesn't prevent clients
from having to track a big version matrix (i.e. you still need to know
which api versions are supported for each broker version), so maybe it
doesn't help as much as I hoped. And as pointed out in the discussion, it
doesn't allow backporting, which may or may not be an issue depending on
whether we want to support that. If we don't, then I'd say the single
version still gets edge in terms of simplicity.

Anyway, I'm not particularly strong in favor of either approach, but I
think what's concerning Gwen (and myself) is that we're going down to the
wire with a significant decision and the KIP itself seems different every
time we look at it. And still a little incomplete. It would be nice, for
example, if the rejected alternatives explained why each alternative was
rejected. It would also be nice to discuss the way the current client
versioning works and why it is inadequate for third-party clients.

Thanks,
Jason

On Thu, Mar 17, 2016 at 12:28 PM, Magnus Edenhill 
wrote:

> I dont really see how first checking if the broker supports a certain set
> of API versions
> is so much more complex than calling the same set of API versions during
> operation?
> The only difference is that without the check we might only make it halfway
> through,
> i.e., being able to join a group, consume messages, but then fail to commit
> them
>
> Another concrete example is support for KIP-31 and KIP-32:
> How will a client with support for 0.8.0-0.10.0 know when to use the new
> Message format?
> If a client uses the old Message format (v0) the broker will need to
> convert message formats
> in the slow path which will increase the broker CPU usage and decrease its
> performance.
> Sure, clients can expose a configuration property to let the user enable
> this but most
> users will miss this (there are a lot of config properties already).
>
>
>
>
> 2016-03-17 19:43 GMT+01:00 Gwen Shapira :
>
> > My problem with per-api is that not all combinations are supported. So
> the
> > client will need to figure out which "safe" combination to revert to -
> > which is a lot like global numbers, but less linear...
> >
> >
> > On Thu, Mar 17, 2016 at 11:29 AM, Joel Koshy 
> wrote:
> >
> > > @Gwen - yes it did come back to per-API versions - I think that
> happened
> > > toward the end of the KIP hangout. I already said this, but personally
> > I'm
> > > also in favor of per-API versions. A global/aggregate API version also
> > > works but that is just a proxy to a vector of per-API versions and at
> the
> > > end of the day a client implementation will need to somehow derive
> *some*
> > > per-API version vector to talk to the broker. The other challenges
> (that
> > > I'm sure can be solved) with the global API version are automatically
> > > computing it in the first place or if set manually, automatically
> > verifying
> > > that it was correctly bumped; then there are the nuances of API
> evolution
> > > across multiple branches and thereby the need for some convention to
> > manage
> > > trees of aggregate API versions.
> > >
> > > Jason and Jay had brought up (earlier in this thread) some practical
> > > difficulties in using per-API versions by client implementations that I
> > > think is worth quoting verbatim to make sure people are aware of the
> > > implications:
> > >
> > > Maybe you could try to capture feature
> > > > compatibility by checking the versions for a subset of request types?
> > For
> > > > example, to ensure that you can use the new consumer API, you check
> > that
> > > > the group coordinator request is present, the offset commit request
> > > version
> > > > is greater than 2, the offset fetch request is greater than 1, and
> the
> > > join
> > > > group request is present. And to ensure compatibility with KIP-32,
> > maybe
> > > > you only need to check the appropriate versions of the fetch and
> > produce
> > > > requests. That sounds kind of complicated to keep track of and you
> > > probably
> > > > end up trying to handle combinations which aren't even possible in
> > > > practice.
> > > >
> > >
> > > Unless I'm missing something I don't recollect any other significant
> > > drawbacks to per-API versions or are there any?
> > >
> > > For the above concern I think indicating support for certain features
> > > ultimately has to be accomplished one way or the other by documenting
> > what
> > > per-API request version ranges need to be used. This is 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Magnus Edenhill]

1. But there is no way of conveying version mismatch in the protocol, that's
one of the reasons for this KIP in the first place :)
Even if there was (e.g., the empty response hack) it makes the client
implementation
more complex since the cached cluster-level version support returned by the
broker
can't really be trusted so then there is an initial "We can probably use
this version"
state followed by a "phew, request succeeded" or "darn, seems the request
wasn't supported" state.

Additionally these late errors are going to be a problem in some cases,
let's say
a certain client really needs OffsetCommit v9, so it starts a consumer,
processes
a bunch of messages and then attempts to commit the latest processed offset
only to find
broker didnt support v9 thus failing the commit. We then have a situation
where
messages were processed but they can't be committed, thus resulting in
multiple
processing, or likewise.

Re caching:
A couple of years back we had the discussion on cached cluster ISR info in
the
Metadata response, since the information was cached it couldn't be trusted
and was thus
deemed useless. Let's not make that mistake again.

Re in Metadata or not:
I'm personally fine with having a specific request to query the current
broker's API support,
the client will fire off both MetadataRequests + ApiQueryRequest back to
back so there is
no latency penalty. This is a cleaner solution and I'm a weak +1 for it
unless other client devs oppose.


/Magnus


2016-03-16 0:37 GMT+01:00 Jay Kreps :

> Yeah I think there are two possible approaches:
> 1. You get the versions in the metadata request and cache them and
> invalidate that cache if you get a version mismatch error (basically
> as we do with leadership information).
> 2. You check each connection
>
> I think combining metadata request and version check only makes sense
> in (1), right? If it is (2) I don't see how you save anything and the
> requests don't really make sense because you're mixing cluster wide
> state about partitions with info about the answering broker.
>
> -Jay
>
> On Tue, Mar 15, 2016 at 4:25 PM, Magnus Edenhill 
> wrote:
> > Hey Jay,
> >
> > as discussed earlier it is not safe to cache/relay a broker's version or
> > its supported API versions,
> > by the time the client connects the broker might have upgraded to another
> > version which effectively
> > makes this information useless in a cached form.
> >
> > The complexity of querying for protocol verion is very implementation
> > dependent and
> > hard to generalize on, I dont foresee any bigger problems adding support
> > for an extra protocol version
> > querying state in librdkafka, but other client devs should chime in.
> > There are already post-connect,pre-operation states for dealing with SSL
> > and SASL.
> >
> > The reason for putting the API versioning stuff in the Metadata request
> is
> > that it is already used
> > for bootstrapping a client and/or connection and thus saves us a
> round-trip
> > (and possibly a state).
> >
> >
> > For how this will be used; I can't speak for other client devs but aim to
> > make a mapping between
> > the features my client exposes to a set of specific APIs and their
> minimum
> > version..
> > E.g.: Balanced consumer groups requires JoinGroup >= V0, LeaveGroup >=
> V0,
> > SyncGroup >= V0, and so on.
> > If those requirements can be fullfilled then the feature is enabled,
> > otherwise an error is returned to the user.
> >
> > /Magnus
> >
> >
> > 2016-03-15 23:35 GMT+01:00 Jay Kreps :
> >
> >> Hey Ashish,
> >>
> >> Can you expand in the proposal on how this would be used by clients?
> >> This proposal only has one slot for api versions, though in fact there
> >> is potentially a different version on each broker. I think the
> >> proposal is that every single time the client establishes a connection
> >> it would then need to issue a metadata request on that connection to
> >> check supported versions. Is that correct?
> >>
> >> The point of merging version information with metadata request was
> >> that the client wouldn't have to manage this additional state for each
> >> connection, but rather the broker would gather the information and
> >> give a summary of all brokers in the cluster. (Managing the state
> >> doesn't seem complex but actually since the full state machine for a
> >> request is something like begin connecting=>connection complete=>begin
> >> sending request=>do work sending=>await response=>do work reading
> >> response adding to the state machine around this is not as simple as
> >> it seems...you can see the code in the java client around this).
> >>
> >> It sounds like in this proposal you are proposing merging with the
> >> metadata request but not summarizing across the cluster? Can you
> >> explain the thinking vs a separate request?
> >>
> >> It would really be good if the KIP can summarize the whole interaction
> >> and how clients will work.
> >>
> 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Sent incomplete message by mistake.

I do not have a preference for (1) or (2). (1) basically makes us view
supported versions for all brokers as a part of cluster wide state, however
(2) lets client deal with individual brokers. From usage perspective,
clients would need to choose a certain feature or not based on minimal
supported version across all brokers they are talking to. In (1) this state
will be maintained and provided by server, however in (2) the onus will be
on client to maintain the state. What do you suggest will be the right
thing to do here?


On Tue, Mar 15, 2016 at 10:27 PM, Ashish Singh  wrote:

> I do not have a preference for 1 or 2. 1 basically makes us view supported
> versions for all brokers
>
> On Tuesday, March 15, 2016, Jay Kreps  wrote:
>
> > Yeah I think there are two possible approaches:
> > 1. You get the versions in the metadata request and cache them and
> > invalidate that cache if you get a version mismatch error (basically
> > as we do with leadership information).
> > 2. You check each connection
> >
> > I think combining metadata request and version check only makes sense
> > in (1), right? If it is (2) I don't see how you save anything and the
> > requests don't really make sense because you're mixing cluster wide
> > state about partitions with info about the answering broker.
>
>
> > -Jay
> >
> > On Tue, Mar 15, 2016 at 4:25 PM, Magnus Edenhill  > > wrote:
> > > Hey Jay,
> > >
> > > as discussed earlier it is not safe to cache/relay a broker's version
> or
> > > its supported API versions,
> > > by the time the client connects the broker might have upgraded to
> another
> > > version which effectively
> > > makes this information useless in a cached form.
> > >
> > > The complexity of querying for protocol verion is very implementation
> > > dependent and
> > > hard to generalize on, I dont foresee any bigger problems adding
> support
> > > for an extra protocol version
> > > querying state in librdkafka, but other client devs should chime in.
> > > There are already post-connect,pre-operation states for dealing with
> SSL
> > > and SASL.
> > >
> > > The reason for putting the API versioning stuff in the Metadata request
> > is
> > > that it is already used
> > > for bootstrapping a client and/or connection and thus saves us a
> > round-trip
> > > (and possibly a state).
> > >
> > >
> > > For how this will be used; I can't speak for other client devs but aim
> to
> > > make a mapping between
> > > the features my client exposes to a set of specific APIs and their
> > minimum
> > > version..
> > > E.g.: Balanced consumer groups requires JoinGroup >= V0, LeaveGroup >=
> > V0,
> > > SyncGroup >= V0, and so on.
> > > If those requirements can be fullfilled then the feature is enabled,
> > > otherwise an error is returned to the user.
> > >
> > > /Magnus
> > >
> > >
> > > 2016-03-15 23:35 GMT+01:00 Jay Kreps  >:
> > >
> > >> Hey Ashish,
> > >>
> > >> Can you expand in the proposal on how this would be used by clients?
> > >> This proposal only has one slot for api versions, though in fact there
> > >> is potentially a different version on each broker. I think the
> > >> proposal is that every single time the client establishes a connection
> > >> it would then need to issue a metadata request on that connection to
> > >> check supported versions. Is that correct?
> > >>
> > >> The point of merging version information with metadata request was
> > >> that the client wouldn't have to manage this additional state for each
> > >> connection, but rather the broker would gather the information and
> > >> give a summary of all brokers in the cluster. (Managing the state
> > >> doesn't seem complex but actually since the full state machine for a
> > >> request is something like begin connecting=>connection complete=>begin
> > >> sending request=>do work sending=>await response=>do work reading
> > >> response adding to the state machine around this is not as simple as
> > >> it seems...you can see the code in the java client around this).
> > >>
> > >> It sounds like in this proposal you are proposing merging with the
> > >> metadata request but not summarizing across the cluster? Can you
> > >> explain the thinking vs a separate request?
> > >>
> > >> It would really be good if the KIP can summarize the whole interaction
> > >> and how clients will work.
> > >>
> > >> -Jay
> > >>
> > >> On Tue, Mar 15, 2016 at 3:24 PM, Ashish Singh  > > wrote:
> > >> > Magnus and I had a brief discussion following the KIP call. KIP-35
> > >> > <
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> > >> >
> > >> > wiki has been updated accordingly. Please review the KIP and vote on
> > the
> > >> > corresponding vote thread.
> > >> >
> > >> > On Mon, Mar 14, 2016 at 11:27 PM, Ashish Singh 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

I do not have a preference for 1 or 2. 1 basically makes us view supported
versions for all brokers

On Tuesday, March 15, 2016, Jay Kreps  wrote:

> Yeah I think there are two possible approaches:
> 1. You get the versions in the metadata request and cache them and
> invalidate that cache if you get a version mismatch error (basically
> as we do with leadership information).
> 2. You check each connection
>
> I think combining metadata request and version check only makes sense
> in (1), right? If it is (2) I don't see how you save anything and the
> requests don't really make sense because you're mixing cluster wide
> state about partitions with info about the answering broker.


> -Jay
>
> On Tue, Mar 15, 2016 at 4:25 PM, Magnus Edenhill  > wrote:
> > Hey Jay,
> >
> > as discussed earlier it is not safe to cache/relay a broker's version or
> > its supported API versions,
> > by the time the client connects the broker might have upgraded to another
> > version which effectively
> > makes this information useless in a cached form.
> >
> > The complexity of querying for protocol verion is very implementation
> > dependent and
> > hard to generalize on, I dont foresee any bigger problems adding support
> > for an extra protocol version
> > querying state in librdkafka, but other client devs should chime in.
> > There are already post-connect,pre-operation states for dealing with SSL
> > and SASL.
> >
> > The reason for putting the API versioning stuff in the Metadata request
> is
> > that it is already used
> > for bootstrapping a client and/or connection and thus saves us a
> round-trip
> > (and possibly a state).
> >
> >
> > For how this will be used; I can't speak for other client devs but aim to
> > make a mapping between
> > the features my client exposes to a set of specific APIs and their
> minimum
> > version..
> > E.g.: Balanced consumer groups requires JoinGroup >= V0, LeaveGroup >=
> V0,
> > SyncGroup >= V0, and so on.
> > If those requirements can be fullfilled then the feature is enabled,
> > otherwise an error is returned to the user.
> >
> > /Magnus
> >
> >
> > 2016-03-15 23:35 GMT+01:00 Jay Kreps >:
> >
> >> Hey Ashish,
> >>
> >> Can you expand in the proposal on how this would be used by clients?
> >> This proposal only has one slot for api versions, though in fact there
> >> is potentially a different version on each broker. I think the
> >> proposal is that every single time the client establishes a connection
> >> it would then need to issue a metadata request on that connection to
> >> check supported versions. Is that correct?
> >>
> >> The point of merging version information with metadata request was
> >> that the client wouldn't have to manage this additional state for each
> >> connection, but rather the broker would gather the information and
> >> give a summary of all brokers in the cluster. (Managing the state
> >> doesn't seem complex but actually since the full state machine for a
> >> request is something like begin connecting=>connection complete=>begin
> >> sending request=>do work sending=>await response=>do work reading
> >> response adding to the state machine around this is not as simple as
> >> it seems...you can see the code in the java client around this).
> >>
> >> It sounds like in this proposal you are proposing merging with the
> >> metadata request but not summarizing across the cluster? Can you
> >> explain the thinking vs a separate request?
> >>
> >> It would really be good if the KIP can summarize the whole interaction
> >> and how clients will work.
> >>
> >> -Jay
> >>
> >> On Tue, Mar 15, 2016 at 3:24 PM, Ashish Singh  > wrote:
> >> > Magnus and I had a brief discussion following the KIP call. KIP-35
> >> > <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> >> >
> >> > wiki has been updated accordingly. Please review the KIP and vote on
> the
> >> > corresponding vote thread.
> >> >
> >> > On Mon, Mar 14, 2016 at 11:27 PM, Ashish Singh  >
> >> wrote:
> >> >
> >> >> I think there is a bit of misunderstanding going on here regarding
> >> >> protocol documentation and its versioning. It could be that I am the
> one
> >> >> who misunderstood it, please correct me if so.
> >> >>
> >> >> Taking Gwen's example.
> >> >>
> >> >> 1. 0.10.0 (protocol v4)  is released with current KIP-35
> >> >> 2. On trunk, modify produce requests and bump to v5
> >> >> 3. On trunk, we modify metadata requests and bump to v6
> >> >> 4. Now we decide that the metadata change fixes a super critical
> issue
> >> and
> >> >> want to backport the change. What's the protocol version of the next
> >> >> release of 0.10.0 - which supports v6 protocol only partially?
> >> >>
> >> >> As per my understanding, this will be v7. When we say a broker is on
> >> >> ApiVersion 7, we do not 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jay Kreps]

Yeah I think there are two possible approaches:
1. You get the versions in the metadata request and cache them and
invalidate that cache if you get a version mismatch error (basically
as we do with leadership information).
2. You check each connection

I think combining metadata request and version check only makes sense
in (1), right? If it is (2) I don't see how you save anything and the
requests don't really make sense because you're mixing cluster wide
state about partitions with info about the answering broker.

-Jay

On Tue, Mar 15, 2016 at 4:25 PM, Magnus Edenhill  wrote:
> Hey Jay,
>
> as discussed earlier it is not safe to cache/relay a broker's version or
> its supported API versions,
> by the time the client connects the broker might have upgraded to another
> version which effectively
> makes this information useless in a cached form.
>
> The complexity of querying for protocol verion is very implementation
> dependent and
> hard to generalize on, I dont foresee any bigger problems adding support
> for an extra protocol version
> querying state in librdkafka, but other client devs should chime in.
> There are already post-connect,pre-operation states for dealing with SSL
> and SASL.
>
> The reason for putting the API versioning stuff in the Metadata request is
> that it is already used
> for bootstrapping a client and/or connection and thus saves us a round-trip
> (and possibly a state).
>
>
> For how this will be used; I can't speak for other client devs but aim to
> make a mapping between
> the features my client exposes to a set of specific APIs and their minimum
> version..
> E.g.: Balanced consumer groups requires JoinGroup >= V0, LeaveGroup >= V0,
> SyncGroup >= V0, and so on.
> If those requirements can be fullfilled then the feature is enabled,
> otherwise an error is returned to the user.
>
> /Magnus
>
>
> 2016-03-15 23:35 GMT+01:00 Jay Kreps :
>
>> Hey Ashish,
>>
>> Can you expand in the proposal on how this would be used by clients?
>> This proposal only has one slot for api versions, though in fact there
>> is potentially a different version on each broker. I think the
>> proposal is that every single time the client establishes a connection
>> it would then need to issue a metadata request on that connection to
>> check supported versions. Is that correct?
>>
>> The point of merging version information with metadata request was
>> that the client wouldn't have to manage this additional state for each
>> connection, but rather the broker would gather the information and
>> give a summary of all brokers in the cluster. (Managing the state
>> doesn't seem complex but actually since the full state machine for a
>> request is something like begin connecting=>connection complete=>begin
>> sending request=>do work sending=>await response=>do work reading
>> response adding to the state machine around this is not as simple as
>> it seems...you can see the code in the java client around this).
>>
>> It sounds like in this proposal you are proposing merging with the
>> metadata request but not summarizing across the cluster? Can you
>> explain the thinking vs a separate request?
>>
>> It would really be good if the KIP can summarize the whole interaction
>> and how clients will work.
>>
>> -Jay
>>
>> On Tue, Mar 15, 2016 at 3:24 PM, Ashish Singh  wrote:
>> > Magnus and I had a brief discussion following the KIP call. KIP-35
>> > <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
>> >
>> > wiki has been updated accordingly. Please review the KIP and vote on the
>> > corresponding vote thread.
>> >
>> > On Mon, Mar 14, 2016 at 11:27 PM, Ashish Singh 
>> wrote:
>> >
>> >> I think there is a bit of misunderstanding going on here regarding
>> >> protocol documentation and its versioning. It could be that I am the one
>> >> who misunderstood it, please correct me if so.
>> >>
>> >> Taking Gwen's example.
>> >>
>> >> 1. 0.10.0 (protocol v4)  is released with current KIP-35
>> >> 2. On trunk, modify produce requests and bump to v5
>> >> 3. On trunk, we modify metadata requests and bump to v6
>> >> 4. Now we decide that the metadata change fixes a super critical issue
>> and
>> >> want to backport the change. What's the protocol version of the next
>> >> release of 0.10.0 - which supports v6 protocol only partially?
>> >>
>> >> As per my understanding, this will be v7. When we say a broker is on
>> >> ApiVersion 7, we do not necessarily mean that it also supports
>> ApiVersion
>> >> up to v7. A broker on ApiVersion v7 should probably mean, please refer
>> v7
>> >> of protocol documentation to find out supported protocol versions of
>> this
>> >> broker.
>> >>
>> >> I just added an example on the KIP wiki to elaborate more on protocol
>> >> documentation versioning. Below is the excerpt.
>> >>
>> >> For instance say we have two brokers, BrokerA has ApiVersion 4 and
>> BrokerB
>> 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Magnus Edenhill]

Hey Jay,

as discussed earlier it is not safe to cache/relay a broker's version or
its supported API versions,
by the time the client connects the broker might have upgraded to another
version which effectively
makes this information useless in a cached form.

The complexity of querying for protocol verion is very implementation
dependent and
hard to generalize on, I dont foresee any bigger problems adding support
for an extra protocol version
querying state in librdkafka, but other client devs should chime in.
There are already post-connect,pre-operation states for dealing with SSL
and SASL.

The reason for putting the API versioning stuff in the Metadata request is
that it is already used
for bootstrapping a client and/or connection and thus saves us a round-trip
(and possibly a state).


For how this will be used; I can't speak for other client devs but aim to
make a mapping between
the features my client exposes to a set of specific APIs and their minimum
version..
E.g.: Balanced consumer groups requires JoinGroup >= V0, LeaveGroup >= V0,
SyncGroup >= V0, and so on.
If those requirements can be fullfilled then the feature is enabled,
otherwise an error is returned to the user.

/Magnus


2016-03-15 23:35 GMT+01:00 Jay Kreps :

> Hey Ashish,
>
> Can you expand in the proposal on how this would be used by clients?
> This proposal only has one slot for api versions, though in fact there
> is potentially a different version on each broker. I think the
> proposal is that every single time the client establishes a connection
> it would then need to issue a metadata request on that connection to
> check supported versions. Is that correct?
>
> The point of merging version information with metadata request was
> that the client wouldn't have to manage this additional state for each
> connection, but rather the broker would gather the information and
> give a summary of all brokers in the cluster. (Managing the state
> doesn't seem complex but actually since the full state machine for a
> request is something like begin connecting=>connection complete=>begin
> sending request=>do work sending=>await response=>do work reading
> response adding to the state machine around this is not as simple as
> it seems...you can see the code in the java client around this).
>
> It sounds like in this proposal you are proposing merging with the
> metadata request but not summarizing across the cluster? Can you
> explain the thinking vs a separate request?
>
> It would really be good if the KIP can summarize the whole interaction
> and how clients will work.
>
> -Jay
>
> On Tue, Mar 15, 2016 at 3:24 PM, Ashish Singh  wrote:
> > Magnus and I had a brief discussion following the KIP call. KIP-35
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> >
> > wiki has been updated accordingly. Please review the KIP and vote on the
> > corresponding vote thread.
> >
> > On Mon, Mar 14, 2016 at 11:27 PM, Ashish Singh 
> wrote:
> >
> >> I think there is a bit of misunderstanding going on here regarding
> >> protocol documentation and its versioning. It could be that I am the one
> >> who misunderstood it, please correct me if so.
> >>
> >> Taking Gwen's example.
> >>
> >> 1. 0.10.0 (protocol v4)  is released with current KIP-35
> >> 2. On trunk, modify produce requests and bump to v5
> >> 3. On trunk, we modify metadata requests and bump to v6
> >> 4. Now we decide that the metadata change fixes a super critical issue
> and
> >> want to backport the change. What's the protocol version of the next
> >> release of 0.10.0 - which supports v6 protocol only partially?
> >>
> >> As per my understanding, this will be v7. When we say a broker is on
> >> ApiVersion 7, we do not necessarily mean that it also supports
> ApiVersion
> >> up to v7. A broker on ApiVersion v7 should probably mean, please refer
> v7
> >> of protocol documentation to find out supported protocol versions of
> this
> >> broker.
> >>
> >> I just added an example on the KIP wiki to elaborate more on protocol
> >> documentation versioning. Below is the excerpt.
> >>
> >> For instance say we have two brokers, BrokerA has ApiVersion 4 and
> BrokerB
> >> has ApiVersion 5. This means we should have protocol documentations for
> >> ApiVersions 4 and 5. Say we have the following as protocol documentation
> >> for these two versions.
> >>
> >> Sample Protocol Documentation V4
> >> Version: 4 // Comes from ApiVersion
> >> REQ_A_0: ...
> >> REQ_A_1: ...
> >> RESP_A_0: ...
> >> RESP_A_1: ...
> >>
> >> Sample Protocol Documentation V5
> >> Version: 5 // Comes from ApiVersion
> >> REQ_A_1: ...
> >> REQ_A_2: ...
> >> RESP_A_1: ...
> >> RESP_A_2: ...
> >>
> >> All a client needs to know to be able to successfully communicate with a
> >> broker is what is the supported ApiVersion of the broker. Say via some
> >> mechanism, discussed below, client gets to know that BrokerA has
> 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Hello Jay,

On Tue, Mar 15, 2016 at 3:35 PM, Jay Kreps  wrote:

> Hey Ashish,
>
> Can you expand in the proposal on how this would be used by clients?
>

Will add some info along this line.

This proposal only has one slot for api versions, though in fact there
> is potentially a different version on each broker. I think the
> proposal is that every single time the client establishes a connection
> it would then need to issue a metadata request on that connection to
> check supported versions. Is that correct?
>
> The point of merging version information with metadata request was
> that the client wouldn't have to manage this additional state for each
> connection, but rather the broker would gather the information and
> give a summary of all brokers in the cluster. (Managing the state
> doesn't seem complex but actually since the full state machine for a
> request is something like begin connecting=>connection complete=>begin
> sending request=>do work sending=>await response=>do work reading
> response adding to the state machine around this is not as simple as
> it seems...you can see the code in the java client around this).
>
> It sounds like in this proposal you are proposing merging with the
> metadata request but not summarizing across the cluster? Can you
> explain the thinking vs a separate request?
>

You are right that this proposal is suggesting that client will have to ask
each broker it is talking to, every time a new connection is established
for supported protocol versions.

The other option, as you suggested, is to have broker return information on
supported protocol versions for all brokers. However, that will be a bit
hard to manage. For instance, lets consider following scenario.

1. Client sends MetadataRequest to BrokerA.
2. BrokerA replies following state to Client.
2.a. BrokerA [(0, 0, 2)] // (api, min_version, max_version)
2.b. BrokerB [(0, 0, 3)]
3. Client stores this info and decides to talk to BrokerB using version 3
for Api 0.
4. Meanwhile BrokerB was downgraded and now it actually only supports [(0,
0, 1)]. Not expecting a v3, broker will close connection to the hopeful
Client.

Yes, at this point client can re-send MetadataRequest and update its view,
but a scenario on rolling upgrade or downgrade this can get messy.

As per the current suggestion this scenario will look something like this.

1. Client sends MetadataRequest to BrokerB.
2. BrokerB replies with its supported versions [(0, 0, 3)]
3. Client stores this info and decides to talk to BrokerB using version 3
for Api 0.
4. Meanwhile BrokerB was downgraded and now it actually only supports [(0,
0, 1)]. While doing so, it will close connection to BrokerB.
5. BrokerB re-connects and sends the MetadataRequest.
6. BrokerB replies with its supported versions [(0, 0, 1)]
7. Client and BrokerB talked happily ever after.

AFAIK, we moved away from separate request as having to send a separate
request will need clients to wait for two network cycles instead of one.

>
> It would really be good if the KIP can summarize the whole interaction
> and how clients will work.
>
> -Jay
>
> On Tue, Mar 15, 2016 at 3:24 PM, Ashish Singh  wrote:
> > Magnus and I had a brief discussion following the KIP call. KIP-35
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version
> >
> > wiki has been updated accordingly. Please review the KIP and vote on the
> > corresponding vote thread.
> >
> > On Mon, Mar 14, 2016 at 11:27 PM, Ashish Singh 
> wrote:
> >
> >> I think there is a bit of misunderstanding going on here regarding
> >> protocol documentation and its versioning. It could be that I am the one
> >> who misunderstood it, please correct me if so.
> >>
> >> Taking Gwen's example.
> >>
> >> 1. 0.10.0 (protocol v4)  is released with current KIP-35
> >> 2. On trunk, modify produce requests and bump to v5
> >> 3. On trunk, we modify metadata requests and bump to v6
> >> 4. Now we decide that the metadata change fixes a super critical issue
> and
> >> want to backport the change. What's the protocol version of the next
> >> release of 0.10.0 - which supports v6 protocol only partially?
> >>
> >> As per my understanding, this will be v7. When we say a broker is on
> >> ApiVersion 7, we do not necessarily mean that it also supports
> ApiVersion
> >> up to v7. A broker on ApiVersion v7 should probably mean, please refer
> v7
> >> of protocol documentation to find out supported protocol versions of
> this
> >> broker.
> >>
> >> I just added an example on the KIP wiki to elaborate more on protocol
> >> documentation versioning. Below is the excerpt.
> >>
> >> For instance say we have two brokers, BrokerA has ApiVersion 4 and
> BrokerB
> >> has ApiVersion 5. This means we should have protocol documentations for
> >> ApiVersions 4 and 5. Say we have the following as protocol documentation
> >> for these two versions.
> >>
> >> Sample Protocol 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jay Kreps]

Hey Ashish,

Can you expand in the proposal on how this would be used by clients?
This proposal only has one slot for api versions, though in fact there
is potentially a different version on each broker. I think the
proposal is that every single time the client establishes a connection
it would then need to issue a metadata request on that connection to
check supported versions. Is that correct?

The point of merging version information with metadata request was
that the client wouldn't have to manage this additional state for each
connection, but rather the broker would gather the information and
give a summary of all brokers in the cluster. (Managing the state
doesn't seem complex but actually since the full state machine for a
request is something like begin connecting=>connection complete=>begin
sending request=>do work sending=>await response=>do work reading
response adding to the state machine around this is not as simple as
it seems...you can see the code in the java client around this).

It sounds like in this proposal you are proposing merging with the
metadata request but not summarizing across the cluster? Can you
explain the thinking vs a separate request?

It would really be good if the KIP can summarize the whole interaction
and how clients will work.

-Jay

On Tue, Mar 15, 2016 at 3:24 PM, Ashish Singh  wrote:
> Magnus and I had a brief discussion following the KIP call. KIP-35
> 
> wiki has been updated accordingly. Please review the KIP and vote on the
> corresponding vote thread.
>
> On Mon, Mar 14, 2016 at 11:27 PM, Ashish Singh  wrote:
>
>> I think there is a bit of misunderstanding going on here regarding
>> protocol documentation and its versioning. It could be that I am the one
>> who misunderstood it, please correct me if so.
>>
>> Taking Gwen's example.
>>
>> 1. 0.10.0 (protocol v4)  is released with current KIP-35
>> 2. On trunk, modify produce requests and bump to v5
>> 3. On trunk, we modify metadata requests and bump to v6
>> 4. Now we decide that the metadata change fixes a super critical issue and
>> want to backport the change. What's the protocol version of the next
>> release of 0.10.0 - which supports v6 protocol only partially?
>>
>> As per my understanding, this will be v7. When we say a broker is on
>> ApiVersion 7, we do not necessarily mean that it also supports ApiVersion
>> up to v7. A broker on ApiVersion v7 should probably mean, please refer v7
>> of protocol documentation to find out supported protocol versions of this
>> broker.
>>
>> I just added an example on the KIP wiki to elaborate more on protocol
>> documentation versioning. Below is the excerpt.
>>
>> For instance say we have two brokers, BrokerA has ApiVersion 4 and BrokerB
>> has ApiVersion 5. This means we should have protocol documentations for
>> ApiVersions 4 and 5. Say we have the following as protocol documentation
>> for these two versions.
>>
>> Sample Protocol Documentation V4
>> Version: 4 // Comes from ApiVersion
>> REQ_A_0: ...
>> REQ_A_1: ...
>> RESP_A_0: ...
>> RESP_A_1: ...
>>
>> Sample Protocol Documentation V5
>> Version: 5 // Comes from ApiVersion
>> REQ_A_1: ...
>> REQ_A_2: ...
>> RESP_A_1: ...
>> RESP_A_2: ...
>>
>> All a client needs to know to be able to successfully communicate with a
>> broker is what is the supported ApiVersion of the broker. Say via some
>> mechanism, discussed below, client gets to know that BrokerA has ApiVersion
>> 4 and BrokerB has ApiVersion 5. With that information, and the available
>> protocol documentations for those ApiVersions client can deduce what
>> protocol versions does the broker supports. In this case client will deduce
>> that it can use v0 and v1 of REQ_A and RESP_A while talking to BrokerA,
>> while it can use v1 and v2 of REQ_A and RESP_A while talking to BrokerB.
>>
>> On Mon, Mar 14, 2016 at 10:50 PM, Ewen Cheslack-Postava > > wrote:
>>
>>> Yeah, Gwen's example is a good one. And it doesn't even have to be thought
>>> of in terms of the implementation -- you can think of the protocol itself
>>> as effectively being possible to branch and have changes cherry-picked.
>>> Given the way some changes interact and that only some may be feasible to
>>> backport, this may be important.
>>>
>>> Similarly, it's difficult to make that definition . In practice, we
>>> sometimes branch and effectively merge the protocol -- i.e. we develop 2
>>> KIPs with independent changes at the same time. If you force a linear
>>> model, you also *force* the ordering of implementation, which will be a
>>> pretty serious constraint in a lot of cases. Two protocol-changing KIPs
>>> may
>>> occur near in time, but one may be a much larger change.
>>>
>>> Finally, it might be worth noting that from a client developer's
>>> perspective, the linear order may not be all that intuitive when we pile
>>> on
>>> a bunch of 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Magnus and I had a brief discussion following the KIP call. KIP-35

wiki has been updated accordingly. Please review the KIP and vote on the
corresponding vote thread.

On Mon, Mar 14, 2016 at 11:27 PM, Ashish Singh  wrote:

> I think there is a bit of misunderstanding going on here regarding
> protocol documentation and its versioning. It could be that I am the one
> who misunderstood it, please correct me if so.
>
> Taking Gwen's example.
>
> 1. 0.10.0 (protocol v4)  is released with current KIP-35
> 2. On trunk, modify produce requests and bump to v5
> 3. On trunk, we modify metadata requests and bump to v6
> 4. Now we decide that the metadata change fixes a super critical issue and
> want to backport the change. What's the protocol version of the next
> release of 0.10.0 - which supports v6 protocol only partially?
>
> As per my understanding, this will be v7. When we say a broker is on
> ApiVersion 7, we do not necessarily mean that it also supports ApiVersion
> up to v7. A broker on ApiVersion v7 should probably mean, please refer v7
> of protocol documentation to find out supported protocol versions of this
> broker.
>
> I just added an example on the KIP wiki to elaborate more on protocol
> documentation versioning. Below is the excerpt.
>
> For instance say we have two brokers, BrokerA has ApiVersion 4 and BrokerB
> has ApiVersion 5. This means we should have protocol documentations for
> ApiVersions 4 and 5. Say we have the following as protocol documentation
> for these two versions.
>
> Sample Protocol Documentation V4
> Version: 4 // Comes from ApiVersion
> REQ_A_0: ...
> REQ_A_1: ...
> RESP_A_0: ...
> RESP_A_1: ...
>
> Sample Protocol Documentation V5
> Version: 5 // Comes from ApiVersion
> REQ_A_1: ...
> REQ_A_2: ...
> RESP_A_1: ...
> RESP_A_2: ...
>
> All a client needs to know to be able to successfully communicate with a
> broker is what is the supported ApiVersion of the broker. Say via some
> mechanism, discussed below, client gets to know that BrokerA has ApiVersion
> 4 and BrokerB has ApiVersion 5. With that information, and the available
> protocol documentations for those ApiVersions client can deduce what
> protocol versions does the broker supports. In this case client will deduce
> that it can use v0 and v1 of REQ_A and RESP_A while talking to BrokerA,
> while it can use v1 and v2 of REQ_A and RESP_A while talking to BrokerB.
>
> On Mon, Mar 14, 2016 at 10:50 PM, Ewen Cheslack-Postava  > wrote:
>
>> Yeah, Gwen's example is a good one. And it doesn't even have to be thought
>> of in terms of the implementation -- you can think of the protocol itself
>> as effectively being possible to branch and have changes cherry-picked.
>> Given the way some changes interact and that only some may be feasible to
>> backport, this may be important.
>>
>> Similarly, it's difficult to make that definition . In practice, we
>> sometimes branch and effectively merge the protocol -- i.e. we develop 2
>> KIPs with independent changes at the same time. If you force a linear
>> model, you also *force* the ordering of implementation, which will be a
>> pretty serious constraint in a lot of cases. Two protocol-changing KIPs
>> may
>> occur near in time, but one may be a much larger change.
>>
>> Finally, it might be worth noting that from a client developer's
>> perspective, the linear order may not be all that intuitive when we pile
>> on
>> a bunch of protocol changes in one release. They probably don't actually
>> care about that global protocol version. They'll care more about the types
>> of things Dana was talking about previously: LZ4 support (which isn't even
>> a protocol change, but an important feature clients might need to know
>> about!), Kafka-backed offset storage (requires 2 protocol changes), etc.
>> While we want to encourage supporting all features, we should be realistic
>> about how client developers tackle feature development and limited
>> bandwidth. They are probably more feature driven than version driven.
>>
>> This is what Gwen was saying I had mentioned. The idea of features is
>> actually separate from what has been described so far and *does* require a
>> mapping to protocol versions, but also allows you to capture more than
>> that
>> and at more flexible granularity (single request type protocol version
>> bump
>> or the whole set of requests could change). The idea isn't quite the same
>> as browser feature detection, but that's my frame of reference for it (see
>> https://developer.mozilla.org/en-US/docs/Browser_Feature_Detection), the
>> process of trying to sort out supported features and protocols based on
>> browser version IDs (sort of equivalent to broker implementation versions
>> here) is a huge mess. Going entirely the other route (say, only enabling a
>> feature in CSS3 if *all* CSS3 features are implemented) is really
>> restrictive.

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

I think there is a bit of misunderstanding going on here regarding protocol
documentation and its versioning. It could be that I am the one who
misunderstood it, please correct me if so.

Taking Gwen's example.

1. 0.10.0 (protocol v4)  is released with current KIP-35
2. On trunk, modify produce requests and bump to v5
3. On trunk, we modify metadata requests and bump to v6
4. Now we decide that the metadata change fixes a super critical issue and
want to backport the change. What's the protocol version of the next
release of 0.10.0 - which supports v6 protocol only partially?

As per my understanding, this will be v7. When we say a broker is on
ApiVersion 7, we do not necessarily mean that it also supports ApiVersion
up to v7. A broker on ApiVersion v7 should probably mean, please refer v7
of protocol documentation to find out supported protocol versions of this
broker.

I just added an example on the KIP wiki to elaborate more on protocol
documentation versioning. Below is the excerpt.

For instance say we have two brokers, BrokerA has ApiVersion 4 and BrokerB
has ApiVersion 5. This means we should have protocol documentations for
ApiVersions 4 and 5. Say we have the following as protocol documentation
for these two versions.

Sample Protocol Documentation V4
Version: 4 // Comes from ApiVersion
REQ_A_0: ...
REQ_A_1: ...
RESP_A_0: ...
RESP_A_1: ...

Sample Protocol Documentation V5
Version: 5 // Comes from ApiVersion
REQ_A_1: ...
REQ_A_2: ...
RESP_A_1: ...
RESP_A_2: ...

All a client needs to know to be able to successfully communicate with a
broker is what is the supported ApiVersion of the broker. Say via some
mechanism, discussed below, client gets to know that BrokerA has ApiVersion
4 and BrokerB has ApiVersion 5. With that information, and the available
protocol documentations for those ApiVersions client can deduce what
protocol versions does the broker supports. In this case client will deduce
that it can use v0 and v1 of REQ_A and RESP_A while talking to BrokerA,
while it can use v1 and v2 of REQ_A and RESP_A while talking to BrokerB.

On Mon, Mar 14, 2016 at 10:50 PM, Ewen Cheslack-Postava 
wrote:

> Yeah, Gwen's example is a good one. And it doesn't even have to be thought
> of in terms of the implementation -- you can think of the protocol itself
> as effectively being possible to branch and have changes cherry-picked.
> Given the way some changes interact and that only some may be feasible to
> backport, this may be important.
>
> Similarly, it's difficult to make that definition . In practice, we
> sometimes branch and effectively merge the protocol -- i.e. we develop 2
> KIPs with independent changes at the same time. If you force a linear
> model, you also *force* the ordering of implementation, which will be a
> pretty serious constraint in a lot of cases. Two protocol-changing KIPs may
> occur near in time, but one may be a much larger change.
>
> Finally, it might be worth noting that from a client developer's
> perspective, the linear order may not be all that intuitive when we pile on
> a bunch of protocol changes in one release. They probably don't actually
> care about that global protocol version. They'll care more about the types
> of things Dana was talking about previously: LZ4 support (which isn't even
> a protocol change, but an important feature clients might need to know
> about!), Kafka-backed offset storage (requires 2 protocol changes), etc.
> While we want to encourage supporting all features, we should be realistic
> about how client developers tackle feature development and limited
> bandwidth. They are probably more feature driven than version driven.
>
> This is what Gwen was saying I had mentioned. The idea of features is
> actually separate from what has been described so far and *does* require a
> mapping to protocol versions, but also allows you to capture more than that
> and at more flexible granularity (single request type protocol version bump
> or the whole set of requests could change). The idea isn't quite the same
> as browser feature detection, but that's my frame of reference for it (see
> https://developer.mozilla.org/en-US/docs/Browser_Feature_Detection), the
> process of trying to sort out supported features and protocols based on
> browser version IDs (sort of equivalent to broker implementation versions
> here) is a huge mess. Going entirely the other route (say, only enabling a
> feature in CSS3 if *all* CSS3 features are implemented) is really
> restrictive.
>
> I don't have a concrete proposal right now, but something like "features"
> that sit somewhere between a global protocol version number and only
> individual request versions more accurately captures what we want to
> express, in my opinion.
>
> -Ewen
>
> On Mon, Mar 14, 2016 at 6:45 PM, Gwen Shapira  wrote:
>
> > Jay,
> >
> > Ewen had a good example:
> >
> > 1. 0.10.0 (protocol v4)  is released with current KIP-35
> > 2. On trunk, modify produce 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ewen Cheslack-Postava]

Yeah, Gwen's example is a good one. And it doesn't even have to be thought
of in terms of the implementation -- you can think of the protocol itself
as effectively being possible to branch and have changes cherry-picked.
Given the way some changes interact and that only some may be feasible to
backport, this may be important.

Similarly, it's difficult to make that definition . In practice, we
sometimes branch and effectively merge the protocol -- i.e. we develop 2
KIPs with independent changes at the same time. If you force a linear
model, you also *force* the ordering of implementation, which will be a
pretty serious constraint in a lot of cases. Two protocol-changing KIPs may
occur near in time, but one may be a much larger change.

Finally, it might be worth noting that from a client developer's
perspective, the linear order may not be all that intuitive when we pile on
a bunch of protocol changes in one release. They probably don't actually
care about that global protocol version. They'll care more about the types
of things Dana was talking about previously: LZ4 support (which isn't even
a protocol change, but an important feature clients might need to know
about!), Kafka-backed offset storage (requires 2 protocol changes), etc.
While we want to encourage supporting all features, we should be realistic
about how client developers tackle feature development and limited
bandwidth. They are probably more feature driven than version driven.

This is what Gwen was saying I had mentioned. The idea of features is
actually separate from what has been described so far and *does* require a
mapping to protocol versions, but also allows you to capture more than that
and at more flexible granularity (single request type protocol version bump
or the whole set of requests could change). The idea isn't quite the same
as browser feature detection, but that's my frame of reference for it (see
https://developer.mozilla.org/en-US/docs/Browser_Feature_Detection), the
process of trying to sort out supported features and protocols based on
browser version IDs (sort of equivalent to broker implementation versions
here) is a huge mess. Going entirely the other route (say, only enabling a
feature in CSS3 if *all* CSS3 features are implemented) is really
restrictive.

I don't have a concrete proposal right now, but something like "features"
that sit somewhere between a global protocol version number and only
individual request versions more accurately captures what we want to
express, in my opinion.

-Ewen

On Mon, Mar 14, 2016 at 6:45 PM, Gwen Shapira  wrote:

> Jay,
>
> Ewen had a good example:
>
> 1. 0.10.0 (protocol v4)  is released with current KIP-35
> 2. On trunk, modify produce requests and bump to v5
> 3. On trunk, we modify metadata requests and bump to v6
> 4. Now we decide that the metadata change fixes a super critical issue and
> want to backport the change. What's the protocol version of the next
> release of 0.10.0 - which supports v6 protocol only partially?
>
> Gwen
>
> On Mon, Mar 14, 2016 at 6:38 PM, Jay Kreps  wrote:
>
> > Hey Dana,
> >
> > I am actually thinking about it differently. Basically I think you are
> > imagining a world in which the Kafka code is the source of truth, and
> > the Kafka developers make random changes that inflict pain on you at
> > will. The protocol documentation is basically just some semi-accurate
> > description of what the code does. It sounds like this isn't too far
> > from the actual world. :-) In that world I agree that the best we
> > could do would be to assign some id to versions (the md5 of the Kafka
> > jar, maybe) and put in various checks around that in clients to try to
> > keep things working.
> >
> > But imagine a different approach where we try to really treat the
> > protocol as a document and treat that as the source of truth. We try
> > to make this document cover what is and isn't specified and make it
> > cover enough to support client implementations and a given Kafka
> > version covers some range of protocols explicitly. There is a version
> > of this document for each protocol version. The code implements the
> > protocol rather than vice versa.
> >
> > So in other words protocol changes are totally ordered and separate
> > from code development (we might develop them together but the protocol
> > assignment would come when you checked in the new protocol version
> > which would happen with your code).
> >
> > This was really the intention with the protocol originally (though we
> > were doing it on a per-api basis), but I think that understanding was
> > not shared by the full team and we have not done a great job of
> > important things like documentation or explaining how this are
> > supposed to work so we fall back on the "the protocol is whatever the
> > code does" thing.
> >
> > Does that make sense? In that sense think one of the more important
> > things we could get out of this would not be more 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Joel Koshy]

I have been following this thread with some interest over the past couple
of days especially since we run off trunk for the most part. I have been on
the fence on the approach of a global API version (which we did come up in
the original KIP discussion). I think what everyone felt at the time was a
global API version would also be tricky to use in practice as you would
need to associate that with a vector of per-API request/response pairs or
associate the global API version with a set of features (which is what I
think this thread has converged onto).

I still prefer the earlier approach of an explicit per-API version although
I totally get the Jason and Jay's skepticism wrt how it may be used in
practice - and I think the new consumer plays a big factor in that. It's
just that most other requests are one-shot unlike the sequence of requests
that consumers are expected to participate in so perhaps more complex
features such as the consumer's wire protocol needs some kind of an
umbrella request-group version that needs to be embedded in the metadata
response.

On Mon, Mar 14, 2016 at 6:45 PM, Gwen Shapira  wrote:

> Jay,
>
> Ewen had a good example:
>
> 1. 0.10.0 (protocol v4)  is released with current KIP-35
> 2. On trunk, modify produce requests and bump to v5
> 3. On trunk, we modify metadata requests and bump to v6
> 4. Now we decide that the metadata change fixes a super critical issue and
> want to backport the change. What's the protocol version of the next
> release of 0.10.0 - which supports v6 protocol only partially?
>
> Gwen
>
> On Mon, Mar 14, 2016 at 6:38 PM, Jay Kreps  wrote:
>
> > Hey Dana,
> >
> > I am actually thinking about it differently. Basically I think you are
> > imagining a world in which the Kafka code is the source of truth, and
> > the Kafka developers make random changes that inflict pain on you at
> > will. The protocol documentation is basically just some semi-accurate
> > description of what the code does. It sounds like this isn't too far
> > from the actual world. :-) In that world I agree that the best we
> > could do would be to assign some id to versions (the md5 of the Kafka
> > jar, maybe) and put in various checks around that in clients to try to
> > keep things working.
> >
> > But imagine a different approach where we try to really treat the
> > protocol as a document and treat that as the source of truth. We try
> > to make this document cover what is and isn't specified and make it
> > cover enough to support client implementations and a given Kafka
> > version covers some range of protocols explicitly. There is a version
> > of this document for each protocol version. The code implements the
> > protocol rather than vice versa.
> >
> > So in other words protocol changes are totally ordered and separate
> > from code development (we might develop them together but the protocol
> > assignment would come when you checked in the new protocol version
> > which would happen with your code).
> >
> > This was really the intention with the protocol originally (though we
> > were doing it on a per-api basis), but I think that understanding was
> > not shared by the full team and we have not done a great job of
> > important things like documentation or explaining how this are
> > supposed to work so we fall back on the "the protocol is whatever the
> > code does" thing.
> >
> > Does that make sense? In that sense think one of the more important
> > things we could get out of this would not be more versioning features
> > so much as clear docs and processes around protocol versioning.
> >
> > -Jay
> >
> > On Mon, Mar 14, 2016 at 6:22 PM, Dana Powers 
> > wrote:
> > > Is a linear protocol int consistent with the current release model? It
> > > seems like that would break down w/ the multiple release branches that
> > are
> > > all simultaneously maintained? Or is it implicit that no patch release
> > can
> > > ever bump the protocol int? Or maybe the protocol int gets some extra
> > > "wiggle" on minor / major releases to create unallocated version ints
> > that
> > > could be used on future patch releases / backports?
> > >
> > > I think the protocol version int does make sense for folks deploying
> from
> > > trunk.
> > >
> > > -Dana
> > >
> > > On Mon, Mar 14, 2016 at 6:13 PM, Jay Kreps  wrote:
> > >
> > >> Yeah I think that is the point--we have a proposal for a new protocol
> > >> versioning scheme and a vote on it but it doesn't actually describe
> > >> how versioning will work yet! I gave my vague impression based on this
> > >> thread, but I want to make sure that is correct and get it written
> > >> down before we adopt it.
> > >>
> > >> -Jay
> > >>
> > >> On Mon, Mar 14, 2016 at 5:58 PM, Gwen Shapira 
> > wrote:
> > >> > On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps 
> wrote:
> > >> >
> > >> >> Couple of missing things:
> > >> >>
> > >> >> This KIP 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jay Kreps]

Again I think that implies a world view where the code is the source of
truth. What I'm proposing, though is that there are only protocol versions
5 and version 6 as defined by the protocol spec and you either support 6 or
you don't, supporting part of 6 is just being an incorrect implementation
of the protocol.

Is that sequencing good? Dunno. Alternately you could maintain the status
quo where we version APIs independently...

-Jay

On Monday, March 14, 2016, Gwen Shapira  wrote:

> Jay,
>
> Ewen had a good example:
>
> 1. 0.10.0 (protocol v4)  is released with current KIP-35
> 2. On trunk, modify produce requests and bump to v5
> 3. On trunk, we modify metadata requests and bump to v6
> 4. Now we decide that the metadata change fixes a super critical issue and
> want to backport the change. What's the protocol version of the next
> release of 0.10.0 - which supports v6 protocol only partially?
>
> Gwen
>
> On Mon, Mar 14, 2016 at 6:38 PM, Jay Kreps  > wrote:
>
> > Hey Dana,
> >
> > I am actually thinking about it differently. Basically I think you are
> > imagining a world in which the Kafka code is the source of truth, and
> > the Kafka developers make random changes that inflict pain on you at
> > will. The protocol documentation is basically just some semi-accurate
> > description of what the code does. It sounds like this isn't too far
> > from the actual world. :-) In that world I agree that the best we
> > could do would be to assign some id to versions (the md5 of the Kafka
> > jar, maybe) and put in various checks around that in clients to try to
> > keep things working.
> >
> > But imagine a different approach where we try to really treat the
> > protocol as a document and treat that as the source of truth. We try
> > to make this document cover what is and isn't specified and make it
> > cover enough to support client implementations and a given Kafka
> > version covers some range of protocols explicitly. There is a version
> > of this document for each protocol version. The code implements the
> > protocol rather than vice versa.
> >
> > So in other words protocol changes are totally ordered and separate
> > from code development (we might develop them together but the protocol
> > assignment would come when you checked in the new protocol version
> > which would happen with your code).
> >
> > This was really the intention with the protocol originally (though we
> > were doing it on a per-api basis), but I think that understanding was
> > not shared by the full team and we have not done a great job of
> > important things like documentation or explaining how this are
> > supposed to work so we fall back on the "the protocol is whatever the
> > code does" thing.
> >
> > Does that make sense? In that sense think one of the more important
> > things we could get out of this would not be more versioning features
> > so much as clear docs and processes around protocol versioning.
> >
> > -Jay
> >
> > On Mon, Mar 14, 2016 at 6:22 PM, Dana Powers  >
> > wrote:
> > > Is a linear protocol int consistent with the current release model? It
> > > seems like that would break down w/ the multiple release branches that
> > are
> > > all simultaneously maintained? Or is it implicit that no patch release
> > can
> > > ever bump the protocol int? Or maybe the protocol int gets some extra
> > > "wiggle" on minor / major releases to create unallocated version ints
> > that
> > > could be used on future patch releases / backports?
> > >
> > > I think the protocol version int does make sense for folks deploying
> from
> > > trunk.
> > >
> > > -Dana
> > >
> > > On Mon, Mar 14, 2016 at 6:13 PM, Jay Kreps  > wrote:
> > >
> > >> Yeah I think that is the point--we have a proposal for a new protocol
> > >> versioning scheme and a vote on it but it doesn't actually describe
> > >> how versioning will work yet! I gave my vague impression based on this
> > >> thread, but I want to make sure that is correct and get it written
> > >> down before we adopt it.
> > >>
> > >> -Jay
> > >>
> > >> On Mon, Mar 14, 2016 at 5:58 PM, Gwen Shapira  >
> > wrote:
> > >> > On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps  > wrote:
> > >> >
> > >> >> Couple of missing things:
> > >> >>
> > >> >> This KIP doesn't have a proposal on versioning it just gives
> > different
> > >> >> options, it'd be good to get a concrete proposal in the KIP. Here
> is
> > my
> > >> >> understanding of what we are proposing (can someone sanity check
> and
> > if
> > >> >> correct, update the kip):
> > >> >>
> > >> >>1. We will augment the existing api_version field in the header
> > with
> > >> a
> > >> >>protocol_version that will begin at some initial value and
> > increment
> > >> by
> > >> >> 1
> > >> >>every time we make a changes to any of the api_versions
> 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

Jay,

Ewen had a good example:

1. 0.10.0 (protocol v4)  is released with current KIP-35
2. On trunk, modify produce requests and bump to v5
3. On trunk, we modify metadata requests and bump to v6
4. Now we decide that the metadata change fixes a super critical issue and
want to backport the change. What's the protocol version of the next
release of 0.10.0 - which supports v6 protocol only partially?

Gwen

On Mon, Mar 14, 2016 at 6:38 PM, Jay Kreps  wrote:

> Hey Dana,
>
> I am actually thinking about it differently. Basically I think you are
> imagining a world in which the Kafka code is the source of truth, and
> the Kafka developers make random changes that inflict pain on you at
> will. The protocol documentation is basically just some semi-accurate
> description of what the code does. It sounds like this isn't too far
> from the actual world. :-) In that world I agree that the best we
> could do would be to assign some id to versions (the md5 of the Kafka
> jar, maybe) and put in various checks around that in clients to try to
> keep things working.
>
> But imagine a different approach where we try to really treat the
> protocol as a document and treat that as the source of truth. We try
> to make this document cover what is and isn't specified and make it
> cover enough to support client implementations and a given Kafka
> version covers some range of protocols explicitly. There is a version
> of this document for each protocol version. The code implements the
> protocol rather than vice versa.
>
> So in other words protocol changes are totally ordered and separate
> from code development (we might develop them together but the protocol
> assignment would come when you checked in the new protocol version
> which would happen with your code).
>
> This was really the intention with the protocol originally (though we
> were doing it on a per-api basis), but I think that understanding was
> not shared by the full team and we have not done a great job of
> important things like documentation or explaining how this are
> supposed to work so we fall back on the "the protocol is whatever the
> code does" thing.
>
> Does that make sense? In that sense think one of the more important
> things we could get out of this would not be more versioning features
> so much as clear docs and processes around protocol versioning.
>
> -Jay
>
> On Mon, Mar 14, 2016 at 6:22 PM, Dana Powers 
> wrote:
> > Is a linear protocol int consistent with the current release model? It
> > seems like that would break down w/ the multiple release branches that
> are
> > all simultaneously maintained? Or is it implicit that no patch release
> can
> > ever bump the protocol int? Or maybe the protocol int gets some extra
> > "wiggle" on minor / major releases to create unallocated version ints
> that
> > could be used on future patch releases / backports?
> >
> > I think the protocol version int does make sense for folks deploying from
> > trunk.
> >
> > -Dana
> >
> > On Mon, Mar 14, 2016 at 6:13 PM, Jay Kreps  wrote:
> >
> >> Yeah I think that is the point--we have a proposal for a new protocol
> >> versioning scheme and a vote on it but it doesn't actually describe
> >> how versioning will work yet! I gave my vague impression based on this
> >> thread, but I want to make sure that is correct and get it written
> >> down before we adopt it.
> >>
> >> -Jay
> >>
> >> On Mon, Mar 14, 2016 at 5:58 PM, Gwen Shapira 
> wrote:
> >> > On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps  wrote:
> >> >
> >> >> Couple of missing things:
> >> >>
> >> >> This KIP doesn't have a proposal on versioning it just gives
> different
> >> >> options, it'd be good to get a concrete proposal in the KIP. Here is
> my
> >> >> understanding of what we are proposing (can someone sanity check and
> if
> >> >> correct, update the kip):
> >> >>
> >> >>1. We will augment the existing api_version field in the header
> with
> >> a
> >> >>protocol_version that will begin at some initial value and
> increment
> >> by
> >> >> 1
> >> >>every time we make a changes to any of the api_versions (question:
> >> >>including internal apis?).
> >> >>
> >> >
> >> > Jay, this part was not in the KIP and was never discussed.
> >> > Are you proposing adding this? Or is it just an assumption you made?
> >> >
> >> >
> >> >
> >> >>2. The protocol_version will be added to the metadata request
> >> >>3. We will also add a string that this proposal is calling
> >> VersionString
> >> >>which will describe the build of kafka in some way. The clients
> >> should
> >> >> not
> >> >>under any circumstances do anything with this string other than
> >> print it
> >> >>out to the user.
> >> >>
> >> >> One thing I'm not sure about: I think currently metadata sits in the
> >> client
> >> >> for 10 mins by default. Say a client bootstraps and then a server is
> >> >> downgraded 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jay Kreps]

Hey Dana,

I am actually thinking about it differently. Basically I think you are
imagining a world in which the Kafka code is the source of truth, and
the Kafka developers make random changes that inflict pain on you at
will. The protocol documentation is basically just some semi-accurate
description of what the code does. It sounds like this isn't too far
from the actual world. :-) In that world I agree that the best we
could do would be to assign some id to versions (the md5 of the Kafka
jar, maybe) and put in various checks around that in clients to try to
keep things working.

But imagine a different approach where we try to really treat the
protocol as a document and treat that as the source of truth. We try
to make this document cover what is and isn't specified and make it
cover enough to support client implementations and a given Kafka
version covers some range of protocols explicitly. There is a version
of this document for each protocol version. The code implements the
protocol rather than vice versa.

So in other words protocol changes are totally ordered and separate
from code development (we might develop them together but the protocol
assignment would come when you checked in the new protocol version
which would happen with your code).

This was really the intention with the protocol originally (though we
were doing it on a per-api basis), but I think that understanding was
not shared by the full team and we have not done a great job of
important things like documentation or explaining how this are
supposed to work so we fall back on the "the protocol is whatever the
code does" thing.

Does that make sense? In that sense think one of the more important
things we could get out of this would not be more versioning features
so much as clear docs and processes around protocol versioning.

-Jay

On Mon, Mar 14, 2016 at 6:22 PM, Dana Powers  wrote:
> Is a linear protocol int consistent with the current release model? It
> seems like that would break down w/ the multiple release branches that are
> all simultaneously maintained? Or is it implicit that no patch release can
> ever bump the protocol int? Or maybe the protocol int gets some extra
> "wiggle" on minor / major releases to create unallocated version ints that
> could be used on future patch releases / backports?
>
> I think the protocol version int does make sense for folks deploying from
> trunk.
>
> -Dana
>
> On Mon, Mar 14, 2016 at 6:13 PM, Jay Kreps  wrote:
>
>> Yeah I think that is the point--we have a proposal for a new protocol
>> versioning scheme and a vote on it but it doesn't actually describe
>> how versioning will work yet! I gave my vague impression based on this
>> thread, but I want to make sure that is correct and get it written
>> down before we adopt it.
>>
>> -Jay
>>
>> On Mon, Mar 14, 2016 at 5:58 PM, Gwen Shapira  wrote:
>> > On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps  wrote:
>> >
>> >> Couple of missing things:
>> >>
>> >> This KIP doesn't have a proposal on versioning it just gives different
>> >> options, it'd be good to get a concrete proposal in the KIP. Here is my
>> >> understanding of what we are proposing (can someone sanity check and if
>> >> correct, update the kip):
>> >>
>> >>1. We will augment the existing api_version field in the header with
>> a
>> >>protocol_version that will begin at some initial value and increment
>> by
>> >> 1
>> >>every time we make a changes to any of the api_versions (question:
>> >>including internal apis?).
>> >>
>> >
>> > Jay, this part was not in the KIP and was never discussed.
>> > Are you proposing adding this? Or is it just an assumption you made?
>> >
>> >
>> >
>> >>2. The protocol_version will be added to the metadata request
>> >>3. We will also add a string that this proposal is calling
>> VersionString
>> >>which will describe the build of kafka in some way. The clients
>> should
>> >> not
>> >>under any circumstances do anything with this string other than
>> print it
>> >>out to the user.
>> >>
>> >> One thing I'm not sure about: I think currently metadata sits in the
>> client
>> >> for 10 mins by default. Say a client bootstraps and then a server is
>> >> downgraded to an earlier version, won't the client's metadata version
>> >> indicate that that client handles a version it doesn't actually handle
>> any
>> >> more? We need to document how clients will handle this.
>> >>
>> >> Here are some comments on other details:
>> >>
>> >>1. As a minor thing I think we should avoid naming the fields
>> VersionId
>> >>and VersionString which sort of implies they are both used for
>> >> versioning.
>> >>I think we should call them something like ProtocolVersion and
>> >>BuildDescription, with BuildDescription being totally unspecified
>> other
>> >>than that it is some kind of human readable string describing a
>> >> particular
>> >>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

Thats a great point, Dana. Thanks for bringing this up.

Ewen raised the same concern and suggested something called "feature
detection" where the broker would advertise support for specific features
that clients may need and by that avoid a linear model (it also allows for
advertising features such as compression codecs that are not part of the
protocol at all). I hope he'll jump in to explain :)

Do you have other suggestions?

Gwen



On Mon, Mar 14, 2016 at 6:22 PM, Dana Powers  wrote:

> Is a linear protocol int consistent with the current release model? It
> seems like that would break down w/ the multiple release branches that are
> all simultaneously maintained? Or is it implicit that no patch release can
> ever bump the protocol int? Or maybe the protocol int gets some extra
> "wiggle" on minor / major releases to create unallocated version ints that
> could be used on future patch releases / backports?
>
> I think the protocol version int does make sense for folks deploying from
> trunk.
>
> -Dana
>
> On Mon, Mar 14, 2016 at 6:13 PM, Jay Kreps  wrote:
>
> > Yeah I think that is the point--we have a proposal for a new protocol
> > versioning scheme and a vote on it but it doesn't actually describe
> > how versioning will work yet! I gave my vague impression based on this
> > thread, but I want to make sure that is correct and get it written
> > down before we adopt it.
> >
> > -Jay
> >
> > On Mon, Mar 14, 2016 at 5:58 PM, Gwen Shapira  wrote:
> > > On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps  wrote:
> > >
> > >> Couple of missing things:
> > >>
> > >> This KIP doesn't have a proposal on versioning it just gives different
> > >> options, it'd be good to get a concrete proposal in the KIP. Here is
> my
> > >> understanding of what we are proposing (can someone sanity check and
> if
> > >> correct, update the kip):
> > >>
> > >>1. We will augment the existing api_version field in the header
> with
> > a
> > >>protocol_version that will begin at some initial value and
> increment
> > by
> > >> 1
> > >>every time we make a changes to any of the api_versions (question:
> > >>including internal apis?).
> > >>
> > >
> > > Jay, this part was not in the KIP and was never discussed.
> > > Are you proposing adding this? Or is it just an assumption you made?
> > >
> > >
> > >
> > >>2. The protocol_version will be added to the metadata request
> > >>3. We will also add a string that this proposal is calling
> > VersionString
> > >>which will describe the build of kafka in some way. The clients
> > should
> > >> not
> > >>under any circumstances do anything with this string other than
> > print it
> > >>out to the user.
> > >>
> > >> One thing I'm not sure about: I think currently metadata sits in the
> > client
> > >> for 10 mins by default. Say a client bootstraps and then a server is
> > >> downgraded to an earlier version, won't the client's metadata version
> > >> indicate that that client handles a version it doesn't actually handle
> > any
> > >> more? We need to document how clients will handle this.
> > >>
> > >> Here are some comments on other details:
> > >>
> > >>1. As a minor thing I think we should avoid naming the fields
> > VersionId
> > >>and VersionString which sort of implies they are both used for
> > >> versioning.
> > >>I think we should call them something like ProtocolVersion and
> > >>BuildDescription, with BuildDescription being totally unspecified
> > other
> > >>than that it is some kind of human readable string describing a
> > >> particular
> > >>Kafka build. We really don't want a client attempting to use this
> > >> string in
> > >>any way as that would always be the wrong thing to do in the
> > versioning
> > >>scheme we are proposing, you should always use the protocol
> version.
> > >>2. Does making the topics field in the metadata request nullable
> > >>actually make sense? We have a history of wanting to add magical
> > values
> > >>rather than fields. Currently topics=[a] means give me information
> > about
> > >>topic a, topics=[] means give me information about all topics, and
> we
> > >> are
> > >>proposing topics=null would mean don't give me topics. I don't
> have a
> > >>strong opinion.
> > >>3. I prefer Jason's proposal on using a conservative metadata
> version
> > >>versus the empty response hack. However I think that may actually
> > >>exacerbate the downgrade scenario I described.
> > >>4. I agree with Jason that we should really look at the details of
> > the
> > >>implementation so we know it works--implementing server support
> > without
> > >>actually trying it is kind of risky.
> > >>
> > >> As a meta comment: I'd really like to encourage us to think of the
> > protocol
> > >> as a document that includes the following things:
> > >>
> > >>- The binary format, 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Dana Powers]

Is a linear protocol int consistent with the current release model? It
seems like that would break down w/ the multiple release branches that are
all simultaneously maintained? Or is it implicit that no patch release can
ever bump the protocol int? Or maybe the protocol int gets some extra
"wiggle" on minor / major releases to create unallocated version ints that
could be used on future patch releases / backports?

I think the protocol version int does make sense for folks deploying from
trunk.

-Dana

On Mon, Mar 14, 2016 at 6:13 PM, Jay Kreps  wrote:

> Yeah I think that is the point--we have a proposal for a new protocol
> versioning scheme and a vote on it but it doesn't actually describe
> how versioning will work yet! I gave my vague impression based on this
> thread, but I want to make sure that is correct and get it written
> down before we adopt it.
>
> -Jay
>
> On Mon, Mar 14, 2016 at 5:58 PM, Gwen Shapira  wrote:
> > On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps  wrote:
> >
> >> Couple of missing things:
> >>
> >> This KIP doesn't have a proposal on versioning it just gives different
> >> options, it'd be good to get a concrete proposal in the KIP. Here is my
> >> understanding of what we are proposing (can someone sanity check and if
> >> correct, update the kip):
> >>
> >>1. We will augment the existing api_version field in the header with
> a
> >>protocol_version that will begin at some initial value and increment
> by
> >> 1
> >>every time we make a changes to any of the api_versions (question:
> >>including internal apis?).
> >>
> >
> > Jay, this part was not in the KIP and was never discussed.
> > Are you proposing adding this? Or is it just an assumption you made?
> >
> >
> >
> >>2. The protocol_version will be added to the metadata request
> >>3. We will also add a string that this proposal is calling
> VersionString
> >>which will describe the build of kafka in some way. The clients
> should
> >> not
> >>under any circumstances do anything with this string other than
> print it
> >>out to the user.
> >>
> >> One thing I'm not sure about: I think currently metadata sits in the
> client
> >> for 10 mins by default. Say a client bootstraps and then a server is
> >> downgraded to an earlier version, won't the client's metadata version
> >> indicate that that client handles a version it doesn't actually handle
> any
> >> more? We need to document how clients will handle this.
> >>
> >> Here are some comments on other details:
> >>
> >>1. As a minor thing I think we should avoid naming the fields
> VersionId
> >>and VersionString which sort of implies they are both used for
> >> versioning.
> >>I think we should call them something like ProtocolVersion and
> >>BuildDescription, with BuildDescription being totally unspecified
> other
> >>than that it is some kind of human readable string describing a
> >> particular
> >>Kafka build. We really don't want a client attempting to use this
> >> string in
> >>any way as that would always be the wrong thing to do in the
> versioning
> >>scheme we are proposing, you should always use the protocol version.
> >>2. Does making the topics field in the metadata request nullable
> >>actually make sense? We have a history of wanting to add magical
> values
> >>rather than fields. Currently topics=[a] means give me information
> about
> >>topic a, topics=[] means give me information about all topics, and we
> >> are
> >>proposing topics=null would mean don't give me topics. I don't have a
> >>strong opinion.
> >>3. I prefer Jason's proposal on using a conservative metadata version
> >>versus the empty response hack. However I think that may actually
> >>exacerbate the downgrade scenario I described.
> >>4. I agree with Jason that we should really look at the details of
> the
> >>implementation so we know it works--implementing server support
> without
> >>actually trying it is kind of risky.
> >>
> >> As a meta comment: I'd really like to encourage us to think of the
> protocol
> >> as a document that includes the following things:
> >>
> >>- The binary format, error codes, etc
> >>- The request/response interaction
> >>- The semantics of each request in different cases
> >>- Instructions on how to use this to implement a client
> >>
> >> This document is versioned with the protocol number and is the source of
> >> truth for the protocol.
> >>
> >> Part of any protocol change needs to be an update to the instructions on
> >> how to use that part of the protocol. We should be opinionated. If there
> >> are two options there should be a reason, and then we need to document
> both
> >> and say exactly when to use each.
> >>
> >> I think we also need to get a "how to" document on protocol changes
> just so
> >> people know what they need to do to add a new protocol feature.
> >>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jay Kreps]

Yeah I think that is the point--we have a proposal for a new protocol
versioning scheme and a vote on it but it doesn't actually describe
how versioning will work yet! I gave my vague impression based on this
thread, but I want to make sure that is correct and get it written
down before we adopt it.

-Jay

On Mon, Mar 14, 2016 at 5:58 PM, Gwen Shapira  wrote:
> On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps  wrote:
>
>> Couple of missing things:
>>
>> This KIP doesn't have a proposal on versioning it just gives different
>> options, it'd be good to get a concrete proposal in the KIP. Here is my
>> understanding of what we are proposing (can someone sanity check and if
>> correct, update the kip):
>>
>>1. We will augment the existing api_version field in the header with a
>>protocol_version that will begin at some initial value and increment by
>> 1
>>every time we make a changes to any of the api_versions (question:
>>including internal apis?).
>>
>
> Jay, this part was not in the KIP and was never discussed.
> Are you proposing adding this? Or is it just an assumption you made?
>
>
>
>>2. The protocol_version will be added to the metadata request
>>3. We will also add a string that this proposal is calling VersionString
>>which will describe the build of kafka in some way. The clients should
>> not
>>under any circumstances do anything with this string other than print it
>>out to the user.
>>
>> One thing I'm not sure about: I think currently metadata sits in the client
>> for 10 mins by default. Say a client bootstraps and then a server is
>> downgraded to an earlier version, won't the client's metadata version
>> indicate that that client handles a version it doesn't actually handle any
>> more? We need to document how clients will handle this.
>>
>> Here are some comments on other details:
>>
>>1. As a minor thing I think we should avoid naming the fields VersionId
>>and VersionString which sort of implies they are both used for
>> versioning.
>>I think we should call them something like ProtocolVersion and
>>BuildDescription, with BuildDescription being totally unspecified other
>>than that it is some kind of human readable string describing a
>> particular
>>Kafka build. We really don't want a client attempting to use this
>> string in
>>any way as that would always be the wrong thing to do in the versioning
>>scheme we are proposing, you should always use the protocol version.
>>2. Does making the topics field in the metadata request nullable
>>actually make sense? We have a history of wanting to add magical values
>>rather than fields. Currently topics=[a] means give me information about
>>topic a, topics=[] means give me information about all topics, and we
>> are
>>proposing topics=null would mean don't give me topics. I don't have a
>>strong opinion.
>>3. I prefer Jason's proposal on using a conservative metadata version
>>versus the empty response hack. However I think that may actually
>>exacerbate the downgrade scenario I described.
>>4. I agree with Jason that we should really look at the details of the
>>implementation so we know it works--implementing server support without
>>actually trying it is kind of risky.
>>
>> As a meta comment: I'd really like to encourage us to think of the protocol
>> as a document that includes the following things:
>>
>>- The binary format, error codes, etc
>>- The request/response interaction
>>- The semantics of each request in different cases
>>- Instructions on how to use this to implement a client
>>
>> This document is versioned with the protocol number and is the source of
>> truth for the protocol.
>>
>> Part of any protocol change needs to be an update to the instructions on
>> how to use that part of the protocol. We should be opinionated. If there
>> are two options there should be a reason, and then we need to document both
>> and say exactly when to use each.
>>
>> I think we also need to get a "how to" document on protocol changes just so
>> people know what they need to do to add a new protocol feature.
>>
>> -Jay
>>
>> On Mon, Mar 14, 2016 at 4:51 PM, Jason Gustafson 
>> wrote:
>>
>> > >
>> > > Are you suggesting this as a solution for the problem where a KIP-35
>> > aware
>> > > client sends a higher version of metadata req, say v2, to a KIP-35
>> aware
>> > > broker that only supports up to v1.
>> >
>> >
>> > Yes, that's right. In that case, the client first sends v1, finds out
>> that
>> > the broker supports v2, and then sends v2 (if it has any reason to do
>> so).
>> >
>> > We had a bit of discussion on such scenarios, and it seemed to be a
>> chicken
>> > > and egg problem that is hard to avoid. Your suggestion definitely makes
>> > > sense, however it falls under the purview of clients.
>> >
>> >
>> > That basically means clients should figure it out 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps  wrote:

> Couple of missing things:
>
> This KIP doesn't have a proposal on versioning it just gives different
> options, it'd be good to get a concrete proposal in the KIP. Here is my
> understanding of what we are proposing (can someone sanity check and if
> correct, update the kip):
>
>1. We will augment the existing api_version field in the header with a
>protocol_version that will begin at some initial value and increment by
> 1
>every time we make a changes to any of the api_versions (question:
>including internal apis?).
>

Jay, this part was not in the KIP and was never discussed.
Are you proposing adding this? Or is it just an assumption you made?



>2. The protocol_version will be added to the metadata request
>3. We will also add a string that this proposal is calling VersionString
>which will describe the build of kafka in some way. The clients should
> not
>under any circumstances do anything with this string other than print it
>out to the user.
>
> One thing I'm not sure about: I think currently metadata sits in the client
> for 10 mins by default. Say a client bootstraps and then a server is
> downgraded to an earlier version, won't the client's metadata version
> indicate that that client handles a version it doesn't actually handle any
> more? We need to document how clients will handle this.
>
> Here are some comments on other details:
>
>1. As a minor thing I think we should avoid naming the fields VersionId
>and VersionString which sort of implies they are both used for
> versioning.
>I think we should call them something like ProtocolVersion and
>BuildDescription, with BuildDescription being totally unspecified other
>than that it is some kind of human readable string describing a
> particular
>Kafka build. We really don't want a client attempting to use this
> string in
>any way as that would always be the wrong thing to do in the versioning
>scheme we are proposing, you should always use the protocol version.
>2. Does making the topics field in the metadata request nullable
>actually make sense? We have a history of wanting to add magical values
>rather than fields. Currently topics=[a] means give me information about
>topic a, topics=[] means give me information about all topics, and we
> are
>proposing topics=null would mean don't give me topics. I don't have a
>strong opinion.
>3. I prefer Jason's proposal on using a conservative metadata version
>versus the empty response hack. However I think that may actually
>exacerbate the downgrade scenario I described.
>4. I agree with Jason that we should really look at the details of the
>implementation so we know it works--implementing server support without
>actually trying it is kind of risky.
>
> As a meta comment: I'd really like to encourage us to think of the protocol
> as a document that includes the following things:
>
>- The binary format, error codes, etc
>- The request/response interaction
>- The semantics of each request in different cases
>- Instructions on how to use this to implement a client
>
> This document is versioned with the protocol number and is the source of
> truth for the protocol.
>
> Part of any protocol change needs to be an update to the instructions on
> how to use that part of the protocol. We should be opinionated. If there
> are two options there should be a reason, and then we need to document both
> and say exactly when to use each.
>
> I think we also need to get a "how to" document on protocol changes just so
> people know what they need to do to add a new protocol feature.
>
> -Jay
>
> On Mon, Mar 14, 2016 at 4:51 PM, Jason Gustafson 
> wrote:
>
> > >
> > > Are you suggesting this as a solution for the problem where a KIP-35
> > aware
> > > client sends a higher version of metadata req, say v2, to a KIP-35
> aware
> > > broker that only supports up to v1.
> >
> >
> > Yes, that's right. In that case, the client first sends v1, finds out
> that
> > the broker supports v2, and then sends v2 (if it has any reason to do
> so).
> >
> > We had a bit of discussion on such scenarios, and it seemed to be a
> chicken
> > > and egg problem that is hard to avoid. Your suggestion definitely makes
> > > sense, however it falls under the purview of clients.
> >
> >
> > That basically means clients should figure it out for themselves? Might
> be
> > nice to have a better answer.
> >
> > KIP-35 only aims on adding support for getting the version info from a
> > > broker. This definitely can be utilized by our clients. However, that
> can
> > > follow KIP-35 changes. Does this sound reasonable to you?
> >
> >
> > It may be OK, but I'm a little concerned about offering a feature that we
> > don't support ourselves. Sometimes it's not until implementation that we
> > find out whether it really works as 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

On Mon, Mar 14, 2016 at 4:51 PM, Jason Gustafson  wrote:

> >
> > Are you suggesting this as a solution for the problem where a KIP-35
> aware
> > client sends a higher version of metadata req, say v2, to a KIP-35 aware
> > broker that only supports up to v1.
>
>
> Yes, that's right. In that case, the client first sends v1, finds out that
> the broker supports v2, and then sends v2 (if it has any reason to do so).
>
> We had a bit of discussion on such scenarios, and it seemed to be a chicken
> > and egg problem that is hard to avoid. Your suggestion definitely makes
> > sense, however it falls under the purview of clients.
>
>
> That basically means clients should figure it out for themselves? Might be
> nice to have a better answer.
>
You mean to provide guidance for clients? I think that would makes sense.
However, I do not see how server can alleviate this issue.

>
> KIP-35 only aims on adding support for getting the version info from a
> > broker. This definitely can be utilized by our clients. However, that can
> > follow KIP-35 changes. Does this sound reasonable to you?
>
>
> It may be OK, but I'm a little concerned about offering a feature that we
> don't support ourselves. Sometimes it's not until implementation that we
> find out whether it really works as expected. And if we're eventually
> planning to support it, I feel we should think through some of the cases a
> bit more. For example, the upgrade and downgrade cases that Becket
> mentioned earlier. It doesn't feel too great to support this feature unless
> we can offer guidance on how to use it.
>
> Totally agreed with implementation is required to make sure the feature is
working as expected. Magnus and I had some offline chat and he is willing
to test this out with librdkafka.

Making our client utilize KIP-35 changes will be super awesome. My only
concern is that if we decide to make those changes as part of this KIP, we
will have to move KIP-35 out of 0.10 release scope.

If testing is the only concern, would librdkafka validation be good enough?

> -Jason
>
>
>
> On Mon, Mar 14, 2016 at 4:20 PM, Ashish Singh  wrote:
>
> > Hi Jason,
> >
> > On Mon, Mar 14, 2016 at 4:04 PM, Jason Gustafson 
> > wrote:
> >
> > > Perhaps clients should always send the oldest version of the metadata
> > > request which supports KIP-35 when initially connecting to the cluster.
> > > Depending on the versions in the response, it can upgrade to a more
> > recent
> > > version. Then maybe we don't need the empty response hack?
> > >
> > Are you suggesting this as a solution for the problem where a KIP-35
> aware
> > client sends a higher version of metadata req, say v2, to a KIP-35 aware
> > broker that only supports up to v1.
> >
> > We had a bit of discussion on such scenarios, and it seemed to be a
> chicken
> > and egg problem that is hard to avoid. Your suggestion definitely makes
> > sense, however it falls under the purview of clients.
> >
> > >
> > > One thing that's not clear to me is whether the ultimate goal of this
> KIP
> > > is to have our clients support multiple broker versions. It would be a
> > > little weird to have this feature if our own clients don't use it.
> > >
> > KIP-35 only aims on adding support for getting the version info from a
> > broker. This definitely can be utilized by our clients. However, that can
> > follow KIP-35 changes. Does this sound reasonable to you?
> >
> > >
> > > -Jason
> > >
> > > On Mon, Mar 14, 2016 at 3:34 PM, Ashish Singh 
> > wrote:
> > >
> > > > On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma 
> > wrote:
> > > >
> > > > > On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira 
> > > wrote:
> > > > > >
> > > > > > > I don't see how it helps. If the client is communicating with a
> > > > broker
> > > > > > that
> > > > > > > does not support KIP-35, that broker will simply close the
> > > > connection.
> > > > > If
> > > > > > > the broker supports KIP-35, then it will provide the broker
> > > version.
> > > > I
> > > > > > > don't envisage a scenario where a broker does not support
> KIP-35,
> > > but
> > > > > > > implements the new behaviour of sending an empty response. Do
> > you?
> > > > > > >
> > > > > > > Are you sure about that? Per KIP-35, the broker supplies the
> > > version
> > > > in
> > > > > > response to Metadata request, not in response to anything else.
> > > > > > If the client sends producer request version 42 (accidentally or
> > due
> > > to
> > > > > > premature upgrade) to KIP-35-compactible broker - we want to see
> an
> > > > empty
> > > > > > packet and not a connection close.
> > > > > > Sending a broker version was deemed impractical IIRC.
> > > > > >
> > > > >
> > > > > OK, so this is a different case than the one Ashish described
> > ("client
> > > > that
> > > > > wants to support broker versions that do not provide broker version
> > in
> > > > > metadata and broker 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jason Gustafson]

>
> Are you suggesting this as a solution for the problem where a KIP-35 aware
> client sends a higher version of metadata req, say v2, to a KIP-35 aware
> broker that only supports up to v1.


Yes, that's right. In that case, the client first sends v1, finds out that
the broker supports v2, and then sends v2 (if it has any reason to do so).

We had a bit of discussion on such scenarios, and it seemed to be a chicken
> and egg problem that is hard to avoid. Your suggestion definitely makes
> sense, however it falls under the purview of clients.


That basically means clients should figure it out for themselves? Might be
nice to have a better answer.

KIP-35 only aims on adding support for getting the version info from a
> broker. This definitely can be utilized by our clients. However, that can
> follow KIP-35 changes. Does this sound reasonable to you?


It may be OK, but I'm a little concerned about offering a feature that we
don't support ourselves. Sometimes it's not until implementation that we
find out whether it really works as expected. And if we're eventually
planning to support it, I feel we should think through some of the cases a
bit more. For example, the upgrade and downgrade cases that Becket
mentioned earlier. It doesn't feel too great to support this feature unless
we can offer guidance on how to use it.

-Jason



On Mon, Mar 14, 2016 at 4:20 PM, Ashish Singh  wrote:

> Hi Jason,
>
> On Mon, Mar 14, 2016 at 4:04 PM, Jason Gustafson 
> wrote:
>
> > Perhaps clients should always send the oldest version of the metadata
> > request which supports KIP-35 when initially connecting to the cluster.
> > Depending on the versions in the response, it can upgrade to a more
> recent
> > version. Then maybe we don't need the empty response hack?
> >
> Are you suggesting this as a solution for the problem where a KIP-35 aware
> client sends a higher version of metadata req, say v2, to a KIP-35 aware
> broker that only supports up to v1.
>
> We had a bit of discussion on such scenarios, and it seemed to be a chicken
> and egg problem that is hard to avoid. Your suggestion definitely makes
> sense, however it falls under the purview of clients.
>
> >
> > One thing that's not clear to me is whether the ultimate goal of this KIP
> > is to have our clients support multiple broker versions. It would be a
> > little weird to have this feature if our own clients don't use it.
> >
> KIP-35 only aims on adding support for getting the version info from a
> broker. This definitely can be utilized by our clients. However, that can
> follow KIP-35 changes. Does this sound reasonable to you?
>
> >
> > -Jason
> >
> > On Mon, Mar 14, 2016 at 3:34 PM, Ashish Singh 
> wrote:
> >
> > > On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma 
> wrote:
> > >
> > > > On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira 
> > wrote:
> > > > >
> > > > > > I don't see how it helps. If the client is communicating with a
> > > broker
> > > > > that
> > > > > > does not support KIP-35, that broker will simply close the
> > > connection.
> > > > If
> > > > > > the broker supports KIP-35, then it will provide the broker
> > version.
> > > I
> > > > > > don't envisage a scenario where a broker does not support KIP-35,
> > but
> > > > > > implements the new behaviour of sending an empty response. Do
> you?
> > > > > >
> > > > > > Are you sure about that? Per KIP-35, the broker supplies the
> > version
> > > in
> > > > > response to Metadata request, not in response to anything else.
> > > > > If the client sends producer request version 42 (accidentally or
> due
> > to
> > > > > premature upgrade) to KIP-35-compactible broker - we want to see an
> > > empty
> > > > > packet and not a connection close.
> > > > > Sending a broker version was deemed impractical IIRC.
> > > > >
> > > >
> > > > OK, so this is a different case than the one Ashish described
> ("client
> > > that
> > > > wants to support broker versions that do not provide broker version
> in
> > > > metadata and broker versions that provides version info in
> metadata").
> > > So,
> > > > you are suggesting that if a client is communicating with a broker
> that
> > > > implements KIP-35 and it receives an empty response, it will assume
> > that
> > > > the broker doesn't support the request version and it won't try to
> > parse
> > > > the response? I think it would be good to explain this kind of thing
> in
> > > > detail in the KIP.
> > > >
> > > Actually even in this case and the case I mentioned, closing connection
> > > should be fine. Lets think about possible reasons that could lead to
> this
> > > issue.
> > >
> > > 1. Client has incorrect mapping of supported protocols for a broker
> > > version.
> > > 2. Client misread broker version from metadata response.
> > > 3. Client constructed unsupported protocol version by mistake.
> > >
> > > In all the above cases irrespective of what 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Hi Jason,

On Mon, Mar 14, 2016 at 4:04 PM, Jason Gustafson  wrote:

> Perhaps clients should always send the oldest version of the metadata
> request which supports KIP-35 when initially connecting to the cluster.
> Depending on the versions in the response, it can upgrade to a more recent
> version. Then maybe we don't need the empty response hack?
>
Are you suggesting this as a solution for the problem where a KIP-35 aware
client sends a higher version of metadata req, say v2, to a KIP-35 aware
broker that only supports up to v1.

We had a bit of discussion on such scenarios, and it seemed to be a chicken
and egg problem that is hard to avoid. Your suggestion definitely makes
sense, however it falls under the purview of clients.

>
> One thing that's not clear to me is whether the ultimate goal of this KIP
> is to have our clients support multiple broker versions. It would be a
> little weird to have this feature if our own clients don't use it.
>
KIP-35 only aims on adding support for getting the version info from a
broker. This definitely can be utilized by our clients. However, that can
follow KIP-35 changes. Does this sound reasonable to you?

>
> -Jason
>
> On Mon, Mar 14, 2016 at 3:34 PM, Ashish Singh  wrote:
>
> > On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma  wrote:
> >
> > > On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira 
> wrote:
> > > >
> > > > > I don't see how it helps. If the client is communicating with a
> > broker
> > > > that
> > > > > does not support KIP-35, that broker will simply close the
> > connection.
> > > If
> > > > > the broker supports KIP-35, then it will provide the broker
> version.
> > I
> > > > > don't envisage a scenario where a broker does not support KIP-35,
> but
> > > > > implements the new behaviour of sending an empty response. Do you?
> > > > >
> > > > > Are you sure about that? Per KIP-35, the broker supplies the
> version
> > in
> > > > response to Metadata request, not in response to anything else.
> > > > If the client sends producer request version 42 (accidentally or due
> to
> > > > premature upgrade) to KIP-35-compactible broker - we want to see an
> > empty
> > > > packet and not a connection close.
> > > > Sending a broker version was deemed impractical IIRC.
> > > >
> > >
> > > OK, so this is a different case than the one Ashish described ("client
> > that
> > > wants to support broker versions that do not provide broker version in
> > > metadata and broker versions that provides version info in metadata").
> > So,
> > > you are suggesting that if a client is communicating with a broker that
> > > implements KIP-35 and it receives an empty response, it will assume
> that
> > > the broker doesn't support the request version and it won't try to
> parse
> > > the response? I think it would be good to explain this kind of thing in
> > > detail in the KIP.
> > >
> > Actually even in this case and the case I mentioned, closing connection
> > should be fine. Lets think about possible reasons that could lead to this
> > issue.
> >
> > 1. Client has incorrect mapping of supported protocols for a broker
> > version.
> > 2. Client misread broker version from metadata response.
> > 3. Client constructed unsupported protocol version by mistake.
> >
> > In all the above cases irrespective of what broker does, client will keep
> > sending wrong request version.
> >
> > At this point, I think sending an empty packet instead of closing
> > connection is a nice to have and not mandatory requirement. Like in the
> > above case, a client can catch parsing error and be sure that there is
> > something wrong in the protocol version it is sending. However, a generic
> > connection close does not really provide any information on probable
> cause.
> >
> > What do you guys suggest?
> >
> > >
> > > Ismael
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jason Gustafson]

Perhaps clients should always send the oldest version of the metadata
request which supports KIP-35 when initially connecting to the cluster.
Depending on the versions in the response, it can upgrade to a more recent
version. Then maybe we don't need the empty response hack?

One thing that's not clear to me is whether the ultimate goal of this KIP
is to have our clients support multiple broker versions. It would be a
little weird to have this feature if our own clients don't use it.

-Jason

On Mon, Mar 14, 2016 at 3:34 PM, Ashish Singh  wrote:

> On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma  wrote:
>
> > On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira  wrote:
> > >
> > > > I don't see how it helps. If the client is communicating with a
> broker
> > > that
> > > > does not support KIP-35, that broker will simply close the
> connection.
> > If
> > > > the broker supports KIP-35, then it will provide the broker version.
> I
> > > > don't envisage a scenario where a broker does not support KIP-35, but
> > > > implements the new behaviour of sending an empty response. Do you?
> > > >
> > > > Are you sure about that? Per KIP-35, the broker supplies the version
> in
> > > response to Metadata request, not in response to anything else.
> > > If the client sends producer request version 42 (accidentally or due to
> > > premature upgrade) to KIP-35-compactible broker - we want to see an
> empty
> > > packet and not a connection close.
> > > Sending a broker version was deemed impractical IIRC.
> > >
> >
> > OK, so this is a different case than the one Ashish described ("client
> that
> > wants to support broker versions that do not provide broker version in
> > metadata and broker versions that provides version info in metadata").
> So,
> > you are suggesting that if a client is communicating with a broker that
> > implements KIP-35 and it receives an empty response, it will assume that
> > the broker doesn't support the request version and it won't try to parse
> > the response? I think it would be good to explain this kind of thing in
> > detail in the KIP.
> >
> Actually even in this case and the case I mentioned, closing connection
> should be fine. Lets think about possible reasons that could lead to this
> issue.
>
> 1. Client has incorrect mapping of supported protocols for a broker
> version.
> 2. Client misread broker version from metadata response.
> 3. Client constructed unsupported protocol version by mistake.
>
> In all the above cases irrespective of what broker does, client will keep
> sending wrong request version.
>
> At this point, I think sending an empty packet instead of closing
> connection is a nice to have and not mandatory requirement. Like in the
> above case, a client can catch parsing error and be sure that there is
> something wrong in the protocol version it is sending. However, a generic
> connection close does not really provide any information on probable cause.
>
> What do you guys suggest?
>
> >
> > Ismael
> >
>
>
>
> --
>
> Regards,
> Ashish
>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

I have updated the KIP based on the discussions so far. Will initiate a
VOTE thread soon. There are some minor details that are still under
discussion, but nothing major to stop us from voting.

On Mon, Mar 14, 2016 at 3:39 PM, Ashish Singh  wrote:

>
>
> On Mon, Mar 14, 2016 at 3:37 PM, Ismael Juma  wrote:
>
>> On Mon, Mar 14, 2016 at 10:35 PM, Ashish Singh 
>> wrote:
>>
>> > > Also, I think it's a bit odd to say a `single null topic with size
>> -1`.
>> > Do
>> > > we mean an array of topics with size -1 and no elements? That would
>> imply
>> > > introducing a NULLABLE_ARRAY type (we currently have NULLABLE_STRING
>> and
>> > > NULLABLE_BYTES).
>> > >
>> > Missed this point. I was thinking of an array with one null string,
>> > ArrayOf[NULLABLE_STRING]. However, we can add NullableArrayOf and use
>> > NullableArrayOf[STRING] as well. What do you think is better here?
>> >
>>
>> I prefer the latter.
>>
>> So be it. Updating KIP. Thanks!
>
>> Ismael
>>
>
>
>
> --
>
> Regards,
> Ashish
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

On Mon, Mar 14, 2016 at 3:37 PM, Ismael Juma  wrote:

> On Mon, Mar 14, 2016 at 10:35 PM, Ashish Singh 
> wrote:
>
> > > Also, I think it's a bit odd to say a `single null topic with size -1`.
> > Do
> > > we mean an array of topics with size -1 and no elements? That would
> imply
> > > introducing a NULLABLE_ARRAY type (we currently have NULLABLE_STRING
> and
> > > NULLABLE_BYTES).
> > >
> > Missed this point. I was thinking of an array with one null string,
> > ArrayOf[NULLABLE_STRING]. However, we can add NullableArrayOf and use
> > NullableArrayOf[STRING] as well. What do you think is better here?
> >
>
> I prefer the latter.
>
> So be it. Updating KIP. Thanks!

> Ismael
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

On Mon, Mar 14, 2016 at 10:35 PM, Ashish Singh  wrote:

> > Also, I think it's a bit odd to say a `single null topic with size -1`.
> Do
> > we mean an array of topics with size -1 and no elements? That would imply
> > introducing a NULLABLE_ARRAY type (we currently have NULLABLE_STRING and
> > NULLABLE_BYTES).
> >
> Missed this point. I was thinking of an array with one null string,
> ArrayOf[NULLABLE_STRING]. However, we can add NullableArrayOf and use
> NullableArrayOf[STRING] as well. What do you think is better here?
>

I prefer the latter.

Ismael

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma  wrote:

> On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira  wrote:
> >
> > > I don't see how it helps. If the client is communicating with a broker
> > that
> > > does not support KIP-35, that broker will simply close the connection.
> If
> > > the broker supports KIP-35, then it will provide the broker version. I
> > > don't envisage a scenario where a broker does not support KIP-35, but
> > > implements the new behaviour of sending an empty response. Do you?
> > >
> > > Are you sure about that? Per KIP-35, the broker supplies the version in
> > response to Metadata request, not in response to anything else.
> > If the client sends producer request version 42 (accidentally or due to
> > premature upgrade) to KIP-35-compactible broker - we want to see an empty
> > packet and not a connection close.
> > Sending a broker version was deemed impractical IIRC.
> >
>
> OK, so this is a different case than the one Ashish described ("client that
> wants to support broker versions that do not provide broker version in
> metadata and broker versions that provides version info in metadata"). So,
> you are suggesting that if a client is communicating with a broker that
> implements KIP-35 and it receives an empty response, it will assume that
> the broker doesn't support the request version and it won't try to parse
> the response? I think it would be good to explain this kind of thing in
> detail in the KIP.
>
Actually even in this case and the case I mentioned, closing connection
should be fine. Lets think about possible reasons that could lead to this
issue.

1. Client has incorrect mapping of supported protocols for a broker version.
2. Client misread broker version from metadata response.
3. Client constructed unsupported protocol version by mistake.

In all the above cases irrespective of what broker does, client will keep
sending wrong request version.

At this point, I think sending an empty packet instead of closing
connection is a nice to have and not mandatory requirement. Like in the
above case, a client can catch parsing error and be sure that there is
something wrong in the protocol version it is sending. However, a generic
connection close does not really provide any information on probable cause.

What do you guys suggest?

>
> Ismael
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

Agree. Regardless of which one of us understood the KIP correctly, this
kind of thing should be crystal clear in the KIP.

On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma  wrote:

> On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira  wrote:
> >
> > > I don't see how it helps. If the client is communicating with a broker
> > that
> > > does not support KIP-35, that broker will simply close the connection.
> If
> > > the broker supports KIP-35, then it will provide the broker version. I
> > > don't envisage a scenario where a broker does not support KIP-35, but
> > > implements the new behaviour of sending an empty response. Do you?
> > >
> > > Are you sure about that? Per KIP-35, the broker supplies the version in
> > response to Metadata request, not in response to anything else.
> > If the client sends producer request version 42 (accidentally or due to
> > premature upgrade) to KIP-35-compactible broker - we want to see an empty
> > packet and not a connection close.
> > Sending a broker version was deemed impractical IIRC.
> >
>
> OK, so this is a different case than the one Ashish described ("client that
> wants to support broker versions that do not provide broker version in
> metadata and broker versions that provides version info in metadata"). So,
> you are suggesting that if a client is communicating with a broker that
> implements KIP-35 and it receives an empty response, it will assume that
> the broker doesn't support the request version and it won't try to parse
> the response? I think it would be good to explain this kind of thing in
> detail in the KIP.
>
> Ismael
>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira  wrote:
>
> > I don't see how it helps. If the client is communicating with a broker
> that
> > does not support KIP-35, that broker will simply close the connection. If
> > the broker supports KIP-35, then it will provide the broker version. I
> > don't envisage a scenario where a broker does not support KIP-35, but
> > implements the new behaviour of sending an empty response. Do you?
> >
> > Are you sure about that? Per KIP-35, the broker supplies the version in
> response to Metadata request, not in response to anything else.
> If the client sends producer request version 42 (accidentally or due to
> premature upgrade) to KIP-35-compactible broker - we want to see an empty
> packet and not a connection close.
> Sending a broker version was deemed impractical IIRC.
>

OK, so this is a different case than the one Ashish described ("client that
wants to support broker versions that do not provide broker version in
metadata and broker versions that provides version info in metadata"). So,
you are suggesting that if a client is communicating with a broker that
implements KIP-35 and it receives an empty response, it will assume that
the broker doesn't support the request version and it won't try to parse
the response? I think it would be good to explain this kind of thing in
detail in the KIP.

Ismael

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

>
>
> I don't see how it helps. If the client is communicating with a broker that
> does not support KIP-35, that broker will simply close the connection. If
> the broker supports KIP-35, then it will provide the broker version. I
> don't envisage a scenario where a broker does not support KIP-35, but
> implements the new behaviour of sending an empty response. Do you?
>
> Are you sure about that? Per KIP-35, the broker supplies the version in
response to Metadata request, not in response to anything else.
If the client sends producer request version 42 (accidentally or due to
premature upgrade) to KIP-35-compactible broker - we want to see an empty
packet and not a connection close.
Sending a broker version was deemed impractical IIRC.


> Ismael
>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

Hi Ashish,

Comments inline.

On Mon, Mar 14, 2016 at 6:15 PM, Ashish Singh  wrote:

> I think ApiVersion aims to identify protocol version changes, more than
> release change,
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L30
> .
>

That is something that has changed as part of KIP-31/32 (as I mentioned),
it wasn't like that in 0.9.0.x for example:

https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/api/ApiVersion.scala#L21


> I do agree that having an automated check to ensure this happens will be
> really useful.
>

Great.

> My understanding is that this doesn't help clients that support KIP-35
> > since they will know the broker version. And for older clients, they will
> > fail with a parsing exception, which is a bit better, but not much
> better.
> > So, is it really worth doing? In the KIP call we had about this months
> ago,
> > there was no consensus on this one from what I remember.
> >
> That is a good point! However, what about a client that wants to support
> broker versions that do not provide broker version in metadata and broker
> versions that provides version info in metadata. I think having this does
> not cost us anything, but enables such clients to be smart.
>

I don't see how it helps. If the client is communicating with a broker that
does not support KIP-35, that broker will simply close the connection. If
the broker supports KIP-35, then it will provide the broker version. I
don't envisage a scenario where a broker does not support KIP-35, but
implements the new behaviour of sending an empty response. Do you?

Ismael

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Dana Powers]

There seems to be a lot of tension between support for release-deploys vs.
trunk-deploys. Is there a better way to handle this?

In my experience the vast majority of third-party client code is written
managing compatibility on the first case (release-deploys). I would prefer
a simple approach that is optimized for third-party clients that rarely
connect to trunk-deploys. As Magnus mentioned, the approach we take in
kafka-python is to attempt to identify the broker version -- 0.9, 0.8.2,
0.8.1, 0.8.0 -- and gate "blocks" of features based on that information.

Would it be reasonable to put the onus on the user to manage connecting to
trunk-deploys? If the broker always returns 'trunk' and the client is
configured to manually override the "broker version" via configuration,
would that work for people running trunk-deploys? For example, I might run
a trunk-deploy broker and configure my client to assume broker version
'0.10-dev' and write some client code to support that.

To be honest, I do not plan to release any code publicly (i.e., to pypi)
that is intended to support trunk-deploys. That really sounds like a
maintenance nightmare. I would expect anyone running a server pulled from
trunk to also run clients that aren't officially released / are in active
development.

-Dana


On Mon, Mar 14, 2016 at 11:19 AM, Ashish Singh  wrote:

> On Mon, Mar 14, 2016 at 9:37 AM, Gwen Shapira  wrote:
>
> > On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma  wrote:
> > > Hi Ashish,
> > >
> > > A few comments below.
> > >
> > > On Fri, Mar 11, 2016 at 9:59 PM, Ashish Singh 
> > wrote:
> > >
> > >> Sounds like we are mostly in agreement. Following are the key points.
> > >>
> > >>1. Every time a protocol version changes, for any request/response,
> > >>broker version, ApiVersion, will be bumped up.
> > >>
> > >
> > > Any thoughts on how we will enforce this?
> >
> > Code reviews? :)
> >
> > We are already doing it in ApiVersion (and have been since
> > 0.8.2.0-SNAPSHOT). Enforcing is awesome, but not necessarily part of
> > this KIP.
> >
> > >
> > >
> > >>2. Protocol documentation will be versioned with broker version.
> > Every
> > >>time there is a broker version change, protocol documentation
> version
> > >> needs
> > >>to be updated and linked to main documentation page.
> > >>3. Deprecation of protocol version will be done via marking the
> > version
> > >>as deprecated on the protocol documentation.
> > >>
> > >
> > > I think this is fine for the first version. We may consider doing more
> in
> > > the future (logging a warning perhaps).
> > >
> > >
> > >>4. On getting unknown protocol version, broker will send an empty
> > >>response, instead of simply closing client connection.
> > >>
> > >
> > > I am not sure about this one. It's an unusual pattern and feels like a
> > hack.
> >
> > We discussed this and failed to come up with a better solution that
> > doesn't break compatibility.
> > Improvements can be added in the future - nothing can be worse than
> > current state (where the broker silently closes the connection)
> >
> > >
> > >5. Metadata response will be enhanced to also contain broker
> version,
> > >>VersionInt and VersionString. VersionString will contain internal
> > >>version information.
> > >>
> > >
> > > Even though Magnus suggested that it's OK for clients to parse
> > > `VersionString`, I personally would rather avoid that. Do we really
> need
> > 3
> > > separate versions or could we get away with 2? I think it would be good
> > to
> > > elaborate on this a bit and explain how each of these versions would be
> > > used (both from the broker and clients perspective).
> >
> > Agree! I'm also confused.
> >
> I am working on updating KIP and hopefully that will be less confusing.
> What I meant was metadata response will have broker-version, which will be
> made up of VersionInt and VersionString. For example, (4, "0.10.0-IV0"),
> this will be based on respective ApiVersions,
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L100
> .
>
> >
> > >
> > >>6. Metadata request with single null topic and size of -1 can be
> > used to
> > >>fetch metadata response with only broker version information and no
> > >>topic/broker info.
> > >
> > >7. On receiving a metadata request with single null topic with size
> of
> > >>-1, broker will respond with only broker version.
> > >>
> > >
> > > As Magnus says, the broker information should be returned. This would
> > also
> > > help us reduce unnecessary data transfer during NetworkClient's
> metadata
> > > updates (KAFKA-3358). At the moment, we get information for all topics
> in
> > > situations where we actually want no topics.
> > >
> > > Also, I think it's a bit odd to say a `single null topic with size -1`.
> > Do
> > > we mean an array of topics with size -1 and no 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

On Mon, Mar 14, 2016 at 9:37 AM, Gwen Shapira  wrote:

> On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma  wrote:
> > Hi Ashish,
> >
> > A few comments below.
> >
> > On Fri, Mar 11, 2016 at 9:59 PM, Ashish Singh 
> wrote:
> >
> >> Sounds like we are mostly in agreement. Following are the key points.
> >>
> >>1. Every time a protocol version changes, for any request/response,
> >>broker version, ApiVersion, will be bumped up.
> >>
> >
> > Any thoughts on how we will enforce this?
>
> Code reviews? :)
>
> We are already doing it in ApiVersion (and have been since
> 0.8.2.0-SNAPSHOT). Enforcing is awesome, but not necessarily part of
> this KIP.
>
> >
> >
> >>2. Protocol documentation will be versioned with broker version.
> Every
> >>time there is a broker version change, protocol documentation version
> >> needs
> >>to be updated and linked to main documentation page.
> >>3. Deprecation of protocol version will be done via marking the
> version
> >>as deprecated on the protocol documentation.
> >>
> >
> > I think this is fine for the first version. We may consider doing more in
> > the future (logging a warning perhaps).
> >
> >
> >>4. On getting unknown protocol version, broker will send an empty
> >>response, instead of simply closing client connection.
> >>
> >
> > I am not sure about this one. It's an unusual pattern and feels like a
> hack.
>
> We discussed this and failed to come up with a better solution that
> doesn't break compatibility.
> Improvements can be added in the future - nothing can be worse than
> current state (where the broker silently closes the connection)
>
> >
> >5. Metadata response will be enhanced to also contain broker version,
> >>VersionInt and VersionString. VersionString will contain internal
> >>version information.
> >>
> >
> > Even though Magnus suggested that it's OK for clients to parse
> > `VersionString`, I personally would rather avoid that. Do we really need
> 3
> > separate versions or could we get away with 2? I think it would be good
> to
> > elaborate on this a bit and explain how each of these versions would be
> > used (both from the broker and clients perspective).
>
> Agree! I'm also confused.
>
I am working on updating KIP and hopefully that will be less confusing.
What I meant was metadata response will have broker-version, which will be
made up of VersionInt and VersionString. For example, (4, "0.10.0-IV0"),
this will be based on respective ApiVersions,
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L100
.

>
> >
> >>6. Metadata request with single null topic and size of -1 can be
> used to
> >>fetch metadata response with only broker version information and no
> >>topic/broker info.
> >
> >7. On receiving a metadata request with single null topic with size of
> >>-1, broker will respond with only broker version.
> >>
> >
> > As Magnus says, the broker information should be returned. This would
> also
> > help us reduce unnecessary data transfer during NetworkClient's metadata
> > updates (KAFKA-3358). At the moment, we get information for all topics in
> > situations where we actually want no topics.
> >
> > Also, I think it's a bit odd to say a `single null topic with size -1`.
> Do
> > we mean an array of topics with size -1 and no elements? That would imply
> > introducing a NULLABLE_ARRAY type (we currently have NULLABLE_STRING and
> > NULLABLE_BYTES).
> >
> > Ismael
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Hello Gwen/ Ismael,

On Mon, Mar 14, 2016 at 9:55 AM, Ismael Juma  wrote:

> Hi Gwen,
>
> On Mon, Mar 14, 2016 at 4:37 PM, Gwen Shapira  wrote:
>
> > On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma  wrote:>
> > >>1. Every time a protocol version changes, for any request/response,
> > >>broker version, ApiVersion, will be bumped up.
> > >>
> > >
> > > Any thoughts on how we will enforce this?
> >
> > Code reviews? :)
> >
> > We are already doing it in ApiVersion (and have been since
> > 0.8.2.0-SNAPSHOT). Enforcing is awesome, but not necessarily part of
> > this KIP.
> >
>
> What we have been doing since 0.8.2.0-SNAPSHOT is to create a new
> `ApiVersion` for each new release. What is being proposed here is to create
> a new `ApiVersion` every time a protocol version changes for any
> request/response. This is much easier to miss. Admittedly, this approach
> was introduced as part of KIP-31/32, but if we are going to expose this
> version to clients, I think it is good to think about ways to ensure
> correctness. It may be that we decide that it's out of scope or that we can
> do it later, but I don't think we should just dismiss it without even
> thinking about it.
>
I think ApiVersion aims to identify protocol version changes, more than
release change,
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L30.
I do agree that having an automated check to ensure this happens will be
really useful.

>
> >>4. On getting unknown protocol version, broker will send an empty
> > >>response, instead of simply closing client connection.
> > >>
> > >
> > > I am not sure about this one. It's an unusual pattern and feels like a
> > hack.
> >
> > We discussed this and failed to come up with a better solution that
> > doesn't break compatibility.
> > Improvements can be added in the future - nothing can be worse than
> > current state (where the broker silently closes the connection)
> >
>
> My understanding is that this doesn't help clients that support KIP-35
> since they will know the broker version. And for older clients, they will
> fail with a parsing exception, which is a bit better, but not much better.
> So, is it really worth doing? In the KIP call we had about this months ago,
> there was no consensus on this one from what I remember.
>
That is a good point! However, what about a client that wants to support
broker versions that do not provide broker version in metadata and broker
versions that provides version info in metadata. I think having this does
not cost us anything, but enables such clients to be smart.

>
> Ismael
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

Hi Gwen,

On Mon, Mar 14, 2016 at 4:37 PM, Gwen Shapira  wrote:

> On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma  wrote:>
> >>1. Every time a protocol version changes, for any request/response,
> >>broker version, ApiVersion, will be bumped up.
> >>
> >
> > Any thoughts on how we will enforce this?
>
> Code reviews? :)
>
> We are already doing it in ApiVersion (and have been since
> 0.8.2.0-SNAPSHOT). Enforcing is awesome, but not necessarily part of
> this KIP.
>

What we have been doing since 0.8.2.0-SNAPSHOT is to create a new
`ApiVersion` for each new release. What is being proposed here is to create
a new `ApiVersion` every time a protocol version changes for any
request/response. This is much easier to miss. Admittedly, this approach
was introduced as part of KIP-31/32, but if we are going to expose this
version to clients, I think it is good to think about ways to ensure
correctness. It may be that we decide that it's out of scope or that we can
do it later, but I don't think we should just dismiss it without even
thinking about it.

>>4. On getting unknown protocol version, broker will send an empty
> >>response, instead of simply closing client connection.
> >>
> >
> > I am not sure about this one. It's an unusual pattern and feels like a
> hack.
>
> We discussed this and failed to come up with a better solution that
> doesn't break compatibility.
> Improvements can be added in the future - nothing can be worse than
> current state (where the broker silently closes the connection)
>

My understanding is that this doesn't help clients that support KIP-35
since they will know the broker version. And for older clients, they will
fail with a parsing exception, which is a bit better, but not much better.
So, is it really worth doing? In the KIP call we had about this months ago,
there was no consensus on this one from what I remember.

Ismael

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma  wrote:
> Hi Ashish,
>
> A few comments below.
>
> On Fri, Mar 11, 2016 at 9:59 PM, Ashish Singh  wrote:
>
>> Sounds like we are mostly in agreement. Following are the key points.
>>
>>1. Every time a protocol version changes, for any request/response,
>>broker version, ApiVersion, will be bumped up.
>>
>
> Any thoughts on how we will enforce this?

Code reviews? :)

We are already doing it in ApiVersion (and have been since
0.8.2.0-SNAPSHOT). Enforcing is awesome, but not necessarily part of
this KIP.

>
>
>>2. Protocol documentation will be versioned with broker version. Every
>>time there is a broker version change, protocol documentation version
>> needs
>>to be updated and linked to main documentation page.
>>3. Deprecation of protocol version will be done via marking the version
>>as deprecated on the protocol documentation.
>>
>
> I think this is fine for the first version. We may consider doing more in
> the future (logging a warning perhaps).
>
>
>>4. On getting unknown protocol version, broker will send an empty
>>response, instead of simply closing client connection.
>>
>
> I am not sure about this one. It's an unusual pattern and feels like a hack.

We discussed this and failed to come up with a better solution that
doesn't break compatibility.
Improvements can be added in the future - nothing can be worse than
current state (where the broker silently closes the connection)

>
>5. Metadata response will be enhanced to also contain broker version,
>>VersionInt and VersionString. VersionString will contain internal
>>version information.
>>
>
> Even though Magnus suggested that it's OK for clients to parse
> `VersionString`, I personally would rather avoid that. Do we really need 3
> separate versions or could we get away with 2? I think it would be good to
> elaborate on this a bit and explain how each of these versions would be
> used (both from the broker and clients perspective).

Agree! I'm also confused.

>
>>6. Metadata request with single null topic and size of -1 can be used to
>>fetch metadata response with only broker version information and no
>>topic/broker info.
>
>7. On receiving a metadata request with single null topic with size of
>>-1, broker will respond with only broker version.
>>
>
> As Magnus says, the broker information should be returned. This would also
> help us reduce unnecessary data transfer during NetworkClient's metadata
> updates (KAFKA-3358). At the moment, we get information for all topics in
> situations where we actually want no topics.
>
> Also, I think it's a bit odd to say a `single null topic with size -1`. Do
> we mean an array of topics with size -1 and no elements? That would imply
> introducing a NULLABLE_ARRAY type (we currently have NULLABLE_STRING and
> NULLABLE_BYTES).
>
> Ismael

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

Hi Ashish,

A few comments below.

On Fri, Mar 11, 2016 at 9:59 PM, Ashish Singh  wrote:

> Sounds like we are mostly in agreement. Following are the key points.
>
>1. Every time a protocol version changes, for any request/response,
>broker version, ApiVersion, will be bumped up.
>

Any thoughts on how we will enforce this?


>2. Protocol documentation will be versioned with broker version. Every
>time there is a broker version change, protocol documentation version
> needs
>to be updated and linked to main documentation page.
>3. Deprecation of protocol version will be done via marking the version
>as deprecated on the protocol documentation.
>

I think this is fine for the first version. We may consider doing more in
the future (logging a warning perhaps).


>4. On getting unknown protocol version, broker will send an empty
>response, instead of simply closing client connection.
>

I am not sure about this one. It's an unusual pattern and feels like a hack.

   5. Metadata response will be enhanced to also contain broker version,
>VersionInt and VersionString. VersionString will contain internal
>version information.
>

Even though Magnus suggested that it's OK for clients to parse
`VersionString`, I personally would rather avoid that. Do we really need 3
separate versions or could we get away with 2? I think it would be good to
elaborate on this a bit and explain how each of these versions would be
used (both from the broker and clients perspective).


>6. Metadata request with single null topic and size of -1 can be used to
>fetch metadata response with only broker version information and no
>topic/broker info.

   7. On receiving a metadata request with single null topic with size of
>-1, broker will respond with only broker version.
>

As Magnus says, the broker information should be returned. This would also
help us reduce unnecessary data transfer during NetworkClient's metadata
updates (KAFKA-3358). At the moment, we get information for all topics in
situations where we actually want no topics.

Also, I think it's a bit odd to say a `single null topic with size -1`. Do
we mean an array of topics with size -1 and no elements? That would imply
introducing a NULLABLE_ARRAY type (we currently have NULLABLE_STRING and
NULLABLE_BYTES).

Ismael

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Magnus Edenhill]

Good summary!

Some comments inline:


2016-03-11 22:59 GMT+01:00 Ashish Singh :

> Sounds like we are mostly in agreement. Following are the key points.
>
>1. Every time a protocol version changes, for any request/response,
>broker version, ApiVersion, will be bumped up.
>

Maybe clarify that the protocol version (=broker version) bump is implicit
by a release (formal/final or interim).



>2. Protocol documentation will be versioned with broker version. Every
>time there is a broker version change, protocol documentation version
> needs
>to be updated and linked to main documentation page.
>3. Deprecation of protocol version will be done via marking the version
>as deprecated on the protocol documentation.
>4. On getting unknown protocol version, broker will send an empty
>response, instead of simply closing client connection.
>5. Metadata response will be enhanced to also contain broker version,
>VersionInt and VersionString. VersionString will contain internal
>version information.
>

Clarification: only needs to include version of the responding broker, not
for all returned brokers.


>6. Metadata request with single null topic and size of -1 can be used to
>fetch metadata response with only broker version information and no
>topic/broker info.

   7. On receiving a metadata request with single null topic with size of
>-1, broker will respond with only broker version.
>

Clarification: It should just skip the topic enumeration, broker
enumeration should still be included
since this is used to seed the broker list in the client.


>
> Please correct/add, if I missed out something. If the aforementioned
> changes sound good, I can update the KIP-35 wiki, WIP PR and start a Vote
> thread.
>
> On Fri, Mar 11, 2016 at 12:48 PM, Magnus Edenhill 
> wrote:
>
> > I'm not sure supporting specific interim versions between releases are
> > really that big of a concern,
> > for a start the protocol changes may be in flux and not settle until the
> > formal release, secondly
> > the 3rd party clients typically lag behind at least until the formal
> > release before they implement support (for the first stated reason..).
> > But this is still a good point and if we could use the version fields to
> > specify a point between
> > two formal releases then that would be useful to ease client development
> > during that period.
> > Grabbing 0.10.0 from versionInt and "IV" from versionString is an
> > acceptable solution as long
> > as there is some way for a client to distinguish the formal release.
> >
> >
> > /Magnus
> >
> >
> >
> >
> > 2016-03-11 20:27 GMT+01:00 Gwen Shapira :
> >
> > > Yeah, I'm not sure that 0.10.0-IV1 and 0.10.0-IV2 is what Magnus had
> > > in mind when he was advocating for release versions in the protocol.
> > >
> > > But - if we serialize both the string and the integer Id of ApiVersion
> > > into the Metadata object, I think both Magnus and Jay will be happy :)
> > >
> > > Gwen
> > >
> > > On Fri, Mar 11, 2016 at 11:22 AM, Ismael Juma 
> wrote:
> > > > We introduced a way to bump the API version in between releases as
> part
> > > of
> > > > the KIP-31/KIP-32 by the way. Extending that could maybe work. Take a
> > > look
> > > > at the ApiVersion class and its documentation.
> > > >
> > > > Ismael
> > > > On 11 Mar 2016 19:06, "Gwen Shapira"  wrote:
> > > >
> > > >> Magnus,
> > > >>
> > > >> If we go with release version as protocol version (which I agree is
> > > >> much more user-friendly) - what will be the release version on
> trunk?
> > > >> 0.10.0-SNAPSHOT?
> > > >> How will clients handle the fact that some 0.10.0-SNAPSHOT will have
> > > >> different protocol than others (because we modify the protocol
> > > >> multiple times between releases)?
> > > >>
> > > >> Gwen
> > > >>
> > > >> On Thu, Mar 10, 2016 at 1:52 PM, Magnus Edenhill <
> mag...@edenhill.se>
> > > >> wrote:
> > > >> > Hi all,
> > > >> >
> > > >> > sorry for joining late in the game, the carribean got in the way.
> > > >> >
> > > >> > My thoughts:
> > > >> >
> > > >> > There is no way around the chicken problem, so the sooner we
> can
> > > >> > add protocol versioning functionality the better and we'll add
> > > heuristics
> > > >> > in clients to
> > > >> > handle the migration period (e.g, what Dana has done in
> > kafka-python).
> > > >> > The focus at this point should be to mitigate the core issue
> (allow
> > > >> clients
> > > >> > to know what is supported)
> > > >> > in the least intrusive way. Hopefully we can redesign the protocol
> > in
> > > the
> > > >> > future to add proper
> > > >> > response headers, etc.
> > > >> >
> > > >> > I'm with Data that reusing the broker version as a protocol
> version
> > > will
> > > >> > work just fine and
> > > >> > saves us from administrating another version.
> > > >> > From a client's perspective an explicit 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jay Kreps]

Yeah I agree with that characterization of the tradeoff. I think what that
would imply would be that evolution of the metadata request (or the
protocol version request) would remain server-first, whereas other apis
would be independent. Not sure if I've fully thought it through, though.

-Jay

On Mon, Mar 7, 2016 at 10:58 AM, Ashish Singh  wrote:

> On Fri, Mar 4, 2016 at 5:46 PM, Jay Kreps  wrote:
>
> > Hey Ashish,
> >
> > Both good points.
> >
> > I think the issue with the general metadata request is the same as the
> > issue with a version-specific metadata request from the other
> > proposal--basically it's a chicken and egg problem, to find out anything
> > about the cluster you have to be able to communicate something in a
> format
> > the server can understand without knowing a priori what version it's on.
> I
> > guess the question is how can you continue to evolve the metadata request
> > (whether it is the existing metadata or a protocol-version specific
> > metadata request) given that you need this information to bootstrap you
> > have to be more careful in how that request evolves.
> >
> You are correct. It's just that protocol version request would be very
> specific to retrieve the protocol versions. Changes to protocol version
> request itself should be very rare, if at all. However, the general
> metadata request carries a lot more information and its format is more
> probable to evolve. This boils down to higher probability of change vs a
> definite network round-trip for each re/connect. It does sound like, it is
> better to avoid a definite penalty than to avoid a probable rare issue.
>
> >
> > I think deprecation/removal may be okay. Ultimately clients will always
> use
> > the highest possible version of the protocol the server supports so if
> > we've already deprecated and removed your highest version then you are
> > screwed and you're going to get an error no matter what, right? Basically
> > there is nothing dynamic you can do in that case.
> >
> Sure, this should be expected. Just wanted to make sure deprecation is
> still on the table.
>
> >
> > -Jay
> >
> > On Fri, Mar 4, 2016 at 4:05 PM, Ashish Singh 
> wrote:
> >
> > > Hello Jay,
> > >
> > > The overall approach sounds good. I do realize that this discussion has
> > > gotten too lengthy and is starting to shoot tangents. Maybe a KIP call
> > will
> > > help us getting to a decision faster. I do have a few questions though.
> > >
> > > On Fri, Mar 4, 2016 at 9:52 AM, Jay Kreps  wrote:
> > >
> > > > Yeah here is my summary of my take:
> > > >
> > > > 1. Negotiating a per-connection protocol actually does add a lot of
> > > > complexity to clients (many more failure states to get right).
> > > >
> > > > 2. Having the client configure the protocol version manually is
> doable
> > > now
> > > > but probably a worse state. I suspect this will lead to more not less
> > > > confusion.
> > > >
> > > > 3. I don't think the current state is actually that bad. Integrators
> > > pick a
> > > > conservative version and build against that. There is a tradeoff
> > between
> > > > getting the new features and being compatible with old Kafka
> versions.
> > > But
> > > > a large part of this tradeoff is essential since new features aren't
> > > going
> > > > to magically appear on old servers, so even if you upgrade your
> client
> > > you
> > > > likely aren't going to get the new stuff (since we will end up
> > > dynamically
> > > > turning it off). Having client features that are there but don't work
> > > > because you're on an old cluster may actually be a worse experience
> if
> > > not
> > > > handled very carefully..
> > > >
> > > > 4. The problems Dana brought up are totally orthogonal to the problem
> > of
> > > > having per-api versions or overall versions. The problem was that we
> > > > changed behavior subtly without changing the version. This will be an
> > > issue
> > > > regardless of whether the version is global or not.
> > > >
> > > > 5. Using the broker release as the version is strictly worse than
> > using a
> > > > global protocol version (0, 1, 2, ...) that increments any time any
> api
> > > > changes but doesn't increment just because non-protocol code is
> > changed.
> > > > The problem with using the broker release version is we want to be
> able
> > > to
> > > > keep Kafka releasable from any commit which means there isn't as
> clear
> > a
> > > > sequencing of releases as you would think.
> > > >
> > > > 6. We need to consider the case of mixed version clusters during the
> > time
> > > > period when you are upgrading Kafka.
> > > >
> > > > So overall I think this is not a critical thing to do right now, but
> if
> > > we
> > > > are going to do it we should do it in a way that actually improves
> > > things.
> > > >
> > > > Here would be one proposal for that:
> > > > a. Add a global protocol version that increments with any api 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Magnus Edenhill]

I'm not sure supporting specific interim versions between releases are
really that big of a concern,
for a start the protocol changes may be in flux and not settle until the
formal release, secondly
the 3rd party clients typically lag behind at least until the formal
release before they implement support (for the first stated reason..).
But this is still a good point and if we could use the version fields to
specify a point between
two formal releases then that would be useful to ease client development
during that period.
Grabbing 0.10.0 from versionInt and "IV" from versionString is an
acceptable solution as long
as there is some way for a client to distinguish the formal release.


/Magnus




2016-03-11 20:27 GMT+01:00 Gwen Shapira :

> Yeah, I'm not sure that 0.10.0-IV1 and 0.10.0-IV2 is what Magnus had
> in mind when he was advocating for release versions in the protocol.
>
> But - if we serialize both the string and the integer Id of ApiVersion
> into the Metadata object, I think both Magnus and Jay will be happy :)
>
> Gwen
>
> On Fri, Mar 11, 2016 at 11:22 AM, Ismael Juma  wrote:
> > We introduced a way to bump the API version in between releases as part
> of
> > the KIP-31/KIP-32 by the way. Extending that could maybe work. Take a
> look
> > at the ApiVersion class and its documentation.
> >
> > Ismael
> > On 11 Mar 2016 19:06, "Gwen Shapira"  wrote:
> >
> >> Magnus,
> >>
> >> If we go with release version as protocol version (which I agree is
> >> much more user-friendly) - what will be the release version on trunk?
> >> 0.10.0-SNAPSHOT?
> >> How will clients handle the fact that some 0.10.0-SNAPSHOT will have
> >> different protocol than others (because we modify the protocol
> >> multiple times between releases)?
> >>
> >> Gwen
> >>
> >> On Thu, Mar 10, 2016 at 1:52 PM, Magnus Edenhill 
> >> wrote:
> >> > Hi all,
> >> >
> >> > sorry for joining late in the game, the carribean got in the way.
> >> >
> >> > My thoughts:
> >> >
> >> > There is no way around the chicken problem, so the sooner we can
> >> > add protocol versioning functionality the better and we'll add
> heuristics
> >> > in clients to
> >> > handle the migration period (e.g, what Dana has done in kafka-python).
> >> > The focus at this point should be to mitigate the core issue (allow
> >> clients
> >> > to know what is supported)
> >> > in the least intrusive way. Hopefully we can redesign the protocol in
> the
> >> > future to add proper
> >> > response headers, etc.
> >> >
> >> > I'm with Data that reusing the broker version as a protocol version
> will
> >> > work just fine and
> >> > saves us from administrating another version.
> >> > From a client's perspective an explicit protocol version doesn't
> really
> >> add
> >> > any value.
> >> > I'd rather maintain a mapping of actual broker versions to supported
> >> > protocol requests rather than
> >> > some independent protocol version that still needs to be translated
> to a
> >> > broker version for
> >> > proper code maintainability / error messages / etc.
> >> >
> >> >
> >> > Thus my suggestion is in line with some of the previous speakers,
> that is
> >> > is to keep things
> >> > simple and bump the MetadataRequest version to 1 by adding a
> >> VersionString
> >> > ("0.9.1.0")
> >> > and VersionInt (0x00090100) field to the response.
> >> > These fields return version information for the current connection's
> >> broker
> >> > only, not for other broker's
> >> > in the cluster:
> >> > Providing version information for other brokers doesn't really serve
> any
> >> > purpose:
> >> >  a) the information is cached by the responding broker so it might be
> >> > outdated ( = cant be trusted)
> >> >  b) by the time the client connects to a given broker it might have
> >> upgraded
> >> >
> >> > This means that a client (that is interested in protocol versioning)
> will
> >> > need to query each
> >> > connection's version any way. Since MetadataRequets are typically
> already
> >> > sent on connection set up
> >> > this seems to be the proper place to put it.
> >> >
> >> > The MetadataRequest semantics should also be extended to allow asking
> >> only
> >> > for cluster and version information,
> >> > but not the topic list since this might have negative performance
> impact
> >> on
> >> > large clusters with many topics.
> >> > One way to achieve this would be to provide one single Null topic in
> the
> >> > request (length=-1).
> >> >
> >> > Sending a new Metadata V1 request to an old broker will cause the
> >> > connection to be closed and
> >> > the client will need to use this as a heuristic to downgrade its
> protocol
> >> > ambitions to an older version
> >> > (either by some default value or by user configuration).
> >> >
> >> >
> >> > /Magnus
> >> >
> >> >
> >> > 2016-03-10 20:04 GMT+01:00 Ashish Singh :
> >> >
> >> >> @Magnus,
> >> >>
> >> >> Does the latest suggestion 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

We introduced a way to bump the API version in between releases as part of
the KIP-31/KIP-32 by the way. Extending that could maybe work. Take a look
at the ApiVersion class and its documentation.

Ismael
On 11 Mar 2016 19:06, "Gwen Shapira"  wrote:

> Magnus,
>
> If we go with release version as protocol version (which I agree is
> much more user-friendly) - what will be the release version on trunk?
> 0.10.0-SNAPSHOT?
> How will clients handle the fact that some 0.10.0-SNAPSHOT will have
> different protocol than others (because we modify the protocol
> multiple times between releases)?
>
> Gwen
>
> On Thu, Mar 10, 2016 at 1:52 PM, Magnus Edenhill 
> wrote:
> > Hi all,
> >
> > sorry for joining late in the game, the carribean got in the way.
> >
> > My thoughts:
> >
> > There is no way around the chicken problem, so the sooner we can
> > add protocol versioning functionality the better and we'll add heuristics
> > in clients to
> > handle the migration period (e.g, what Dana has done in kafka-python).
> > The focus at this point should be to mitigate the core issue (allow
> clients
> > to know what is supported)
> > in the least intrusive way. Hopefully we can redesign the protocol in the
> > future to add proper
> > response headers, etc.
> >
> > I'm with Data that reusing the broker version as a protocol version will
> > work just fine and
> > saves us from administrating another version.
> > From a client's perspective an explicit protocol version doesn't really
> add
> > any value.
> > I'd rather maintain a mapping of actual broker versions to supported
> > protocol requests rather than
> > some independent protocol version that still needs to be translated to a
> > broker version for
> > proper code maintainability / error messages / etc.
> >
> >
> > Thus my suggestion is in line with some of the previous speakers, that is
> > is to keep things
> > simple and bump the MetadataRequest version to 1 by adding a
> VersionString
> > ("0.9.1.0")
> > and VersionInt (0x00090100) field to the response.
> > These fields return version information for the current connection's
> broker
> > only, not for other broker's
> > in the cluster:
> > Providing version information for other brokers doesn't really serve any
> > purpose:
> >  a) the information is cached by the responding broker so it might be
> > outdated ( = cant be trusted)
> >  b) by the time the client connects to a given broker it might have
> upgraded
> >
> > This means that a client (that is interested in protocol versioning) will
> > need to query each
> > connection's version any way. Since MetadataRequets are typically already
> > sent on connection set up
> > this seems to be the proper place to put it.
> >
> > The MetadataRequest semantics should also be extended to allow asking
> only
> > for cluster and version information,
> > but not the topic list since this might have negative performance impact
> on
> > large clusters with many topics.
> > One way to achieve this would be to provide one single Null topic in the
> > request (length=-1).
> >
> > Sending a new Metadata V1 request to an old broker will cause the
> > connection to be closed and
> > the client will need to use this as a heuristic to downgrade its protocol
> > ambitions to an older version
> > (either by some default value or by user configuration).
> >
> >
> > /Magnus
> >
> >
> > 2016-03-10 20:04 GMT+01:00 Ashish Singh :
> >
> >> @Magnus,
> >>
> >> Does the latest suggestion sound OK to you. I am planning to update PR
> >> based on latest suggestion.
> >>
> >> On Mon, Mar 7, 2016 at 10:58 AM, Ashish Singh 
> wrote:
> >>
> >> >
> >> >
> >> > On Fri, Mar 4, 2016 at 5:46 PM, Jay Kreps  wrote:
> >> >
> >> >> Hey Ashish,
> >> >>
> >> >> Both good points.
> >> >>
> >> >> I think the issue with the general metadata request is the same as
> the
> >> >> issue with a version-specific metadata request from the other
> >> >> proposal--basically it's a chicken and egg problem, to find out
> anything
> >> >> about the cluster you have to be able to communicate something in a
> >> format
> >> >> the server can understand without knowing a priori what version it's
> >> on. I
> >> >> guess the question is how can you continue to evolve the metadata
> >> request
> >> >> (whether it is the existing metadata or a protocol-version specific
> >> >> metadata request) given that you need this information to bootstrap
> you
> >> >> have to be more careful in how that request evolves.
> >> >>
> >> > You are correct. It's just that protocol version request would be very
> >> > specific to retrieve the protocol versions. Changes to protocol
> version
> >> > request itself should be very rare, if at all. However, the general
> >> > metadata request carries a lot more information and its format is more
> >> > probable to evolve. This boils down to higher probability of change
> vs a
> >> > definite network 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

Magnus,

If we go with release version as protocol version (which I agree is
much more user-friendly) - what will be the release version on trunk?
0.10.0-SNAPSHOT?
How will clients handle the fact that some 0.10.0-SNAPSHOT will have
different protocol than others (because we modify the protocol
multiple times between releases)?

Gwen

On Thu, Mar 10, 2016 at 1:52 PM, Magnus Edenhill  wrote:
> Hi all,
>
> sorry for joining late in the game, the carribean got in the way.
>
> My thoughts:
>
> There is no way around the chicken problem, so the sooner we can
> add protocol versioning functionality the better and we'll add heuristics
> in clients to
> handle the migration period (e.g, what Dana has done in kafka-python).
> The focus at this point should be to mitigate the core issue (allow clients
> to know what is supported)
> in the least intrusive way. Hopefully we can redesign the protocol in the
> future to add proper
> response headers, etc.
>
> I'm with Data that reusing the broker version as a protocol version will
> work just fine and
> saves us from administrating another version.
> From a client's perspective an explicit protocol version doesn't really add
> any value.
> I'd rather maintain a mapping of actual broker versions to supported
> protocol requests rather than
> some independent protocol version that still needs to be translated to a
> broker version for
> proper code maintainability / error messages / etc.
>
>
> Thus my suggestion is in line with some of the previous speakers, that is
> is to keep things
> simple and bump the MetadataRequest version to 1 by adding a VersionString
> ("0.9.1.0")
> and VersionInt (0x00090100) field to the response.
> These fields return version information for the current connection's broker
> only, not for other broker's
> in the cluster:
> Providing version information for other brokers doesn't really serve any
> purpose:
>  a) the information is cached by the responding broker so it might be
> outdated ( = cant be trusted)
>  b) by the time the client connects to a given broker it might have upgraded
>
> This means that a client (that is interested in protocol versioning) will
> need to query each
> connection's version any way. Since MetadataRequets are typically already
> sent on connection set up
> this seems to be the proper place to put it.
>
> The MetadataRequest semantics should also be extended to allow asking only
> for cluster and version information,
> but not the topic list since this might have negative performance impact on
> large clusters with many topics.
> One way to achieve this would be to provide one single Null topic in the
> request (length=-1).
>
> Sending a new Metadata V1 request to an old broker will cause the
> connection to be closed and
> the client will need to use this as a heuristic to downgrade its protocol
> ambitions to an older version
> (either by some default value or by user configuration).
>
>
> /Magnus
>
>
> 2016-03-10 20:04 GMT+01:00 Ashish Singh :
>
>> @Magnus,
>>
>> Does the latest suggestion sound OK to you. I am planning to update PR
>> based on latest suggestion.
>>
>> On Mon, Mar 7, 2016 at 10:58 AM, Ashish Singh  wrote:
>>
>> >
>> >
>> > On Fri, Mar 4, 2016 at 5:46 PM, Jay Kreps  wrote:
>> >
>> >> Hey Ashish,
>> >>
>> >> Both good points.
>> >>
>> >> I think the issue with the general metadata request is the same as the
>> >> issue with a version-specific metadata request from the other
>> >> proposal--basically it's a chicken and egg problem, to find out anything
>> >> about the cluster you have to be able to communicate something in a
>> format
>> >> the server can understand without knowing a priori what version it's
>> on. I
>> >> guess the question is how can you continue to evolve the metadata
>> request
>> >> (whether it is the existing metadata or a protocol-version specific
>> >> metadata request) given that you need this information to bootstrap you
>> >> have to be more careful in how that request evolves.
>> >>
>> > You are correct. It's just that protocol version request would be very
>> > specific to retrieve the protocol versions. Changes to protocol version
>> > request itself should be very rare, if at all. However, the general
>> > metadata request carries a lot more information and its format is more
>> > probable to evolve. This boils down to higher probability of change vs a
>> > definite network round-trip for each re/connect. It does sound like, it
>> is
>> > better to avoid a definite penalty than to avoid a probable rare issue.
>> >
>> >>
>> >> I think deprecation/removal may be okay. Ultimately clients will always
>> >> use
>> >> the highest possible version of the protocol the server supports so if
>> >> we've already deprecated and removed your highest version then you are
>> >> screwed and you're going to get an error no matter what, right?
>> Basically
>> >> there is nothing dynamic you can do in 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Magnus Edenhill]

Hey Jay,

some thoughts in-line:


2016-03-04 18:52 GMT+01:00 Jay Kreps :

> Yeah here is my summary of my take:
>
> 1. Negotiating a per-connection protocol actually does add a lot of
> complexity to clients (many more failure states to get right).
>

I'm not sure about this.
A client that needs to support multiple versions still has this complexity.
Having the broker provide an API to query the broker/protocol version at
least makes that
part easier. Not having such an API and needing to probe is a lot more
complex
and error prone (with possible side effects).



>
> 2. Having the client configure the protocol version manually is doable now
> but probably a worse state. I suspect this will lead to more not less
> confusion.
>

Agree, this should be as a rare manual fallback.



>
> 3. I don't think the current state is actually that bad. Integrators pick a
> conservative version and build against that. There is a tradeoff between
> getting the new features and being compatible with old Kafka versions. But
> a large part of this tradeoff is essential since new features aren't going
> to magically appear on old servers, so even if you upgrade your client you
> likely aren't going to get the new stuff (since we will end up dynamically
> turning it off). Having client features that are there but don't work
> because you're on an old cluster may actually be a worse experience if not
> handled very carefully..
>

Upgrading brokers and clients in lock-step is a luxury of the official
client, 3rd party clients
has no version or distribution coupling with the broker and thus needs to
support
a wider range of broker versions than the Java client.
>From a user perspective I want to install the latest version of client XYZ
and
connect to my Kafka cluster, whatever verison it has, and things should
just work.
No digging up old client versions to support a specific broker, no feature
compat matrices,
no excessive manual configuration.
Things should just work, and they can, with a little help from the broker.
I'm sure most 3rd party client developers are willing to make the effort on
their side.


> 4. The problems Dana brought up are totally orthogonal to the problem of
> having per-api versions or overall versions. The problem was that we
> changed behavior subtly without changing the version. This will be an issue
> regardless of whether the version is global or not.
>

This is actually a good point for using the broker version rather than an
explicit
protocol version: Mistakenly changing an API without bumping the request or
protocol version
makes it impossible for a client to provide a workaround for the new
unversioned functionality,
but a new release will always have a new broker version which a client may
use.



> 5. Using the broker release as the version is strictly worse than using a
> global protocol version (0, 1, 2, ...) that increments any time any api
> changes but doesn't increment just because non-protocol code is changed.
> The problem with using the broker release version is we want to be able to
> keep Kafka releasable from any commit which means there isn't as clear a
> sequencing of releases as you would think.
>

I'm not really sure I follow.
Are you saying that 0.9.1.1 may have lesser protocol support than 0.9.1.0?
I would imagine that any broker version V2 would support at least V1's
protocol requests and versions.
(until things are deprecated/removed, but that is another story).


>
> 6. We need to consider the case of mixed version clusters during the time
> period when you are upgrading Kafka.
>
> So overall I think this is not a critical thing to do right now, but if we
> are going to do it we should do it in a way that actually improves things.
>

I disagree, the sooner we get this in the lesser the future headaches will
be.
Otherwise this discussion will blossom up for each new protocol request,
or version.


/Magnus


> On Thu, Mar 3, 2016 at 9:38 PM, Jason Gustafson 
> wrote:
>
> > I talked with Jay about this KIP briefly this morning, so let me try to
> > summarize the discussion (I'm sure he'll jump in if I get anything
> wrong).
> > Apologies in advance for the length.
> >
> > I think we both share some skepticism that a request with all the
> supported
> > versions of all the request APIs is going to be a useful primitive to try
> > and build client compatibility around. In practice I think people would
> end
> > up checking for particular request versions in order to determine if the
> > broker is 0.8 or 0.9 or whatever, and then change behavior accordingly.
> I'm
> > wondering if there's a reasonable way to handle the version responses
> that
> > doesn't amount to that. Maybe you could try to capture feature
> > compatibility by checking the versions for a subset of request types? For
> > example, to ensure that you can use the new consumer API, you check that
> > the group coordinator request is present, the offset commit request
> 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Magnus Edenhill]

Hi all,

sorry for joining late in the game, the carribean got in the way.

My thoughts:

There is no way around the chicken problem, so the sooner we can
add protocol versioning functionality the better and we'll add heuristics
in clients to
handle the migration period (e.g, what Dana has done in kafka-python).
The focus at this point should be to mitigate the core issue (allow clients
to know what is supported)
in the least intrusive way. Hopefully we can redesign the protocol in the
future to add proper
response headers, etc.

I'm with Data that reusing the broker version as a protocol version will
work just fine and
saves us from administrating another version.
>From a client's perspective an explicit protocol version doesn't really add
any value.
I'd rather maintain a mapping of actual broker versions to supported
protocol requests rather than
some independent protocol version that still needs to be translated to a
broker version for
proper code maintainability / error messages / etc.


Thus my suggestion is in line with some of the previous speakers, that is
is to keep things
simple and bump the MetadataRequest version to 1 by adding a VersionString
("0.9.1.0")
and VersionInt (0x00090100) field to the response.
These fields return version information for the current connection's broker
only, not for other broker's
in the cluster:
Providing version information for other brokers doesn't really serve any
purpose:
 a) the information is cached by the responding broker so it might be
outdated ( = cant be trusted)
 b) by the time the client connects to a given broker it might have upgraded

This means that a client (that is interested in protocol versioning) will
need to query each
connection's version any way. Since MetadataRequets are typically already
sent on connection set up
this seems to be the proper place to put it.

The MetadataRequest semantics should also be extended to allow asking only
for cluster and version information,
but not the topic list since this might have negative performance impact on
large clusters with many topics.
One way to achieve this would be to provide one single Null topic in the
request (length=-1).

Sending a new Metadata V1 request to an old broker will cause the
connection to be closed and
the client will need to use this as a heuristic to downgrade its protocol
ambitions to an older version
(either by some default value or by user configuration).


/Magnus


2016-03-10 20:04 GMT+01:00 Ashish Singh :

> @Magnus,
>
> Does the latest suggestion sound OK to you. I am planning to update PR
> based on latest suggestion.
>
> On Mon, Mar 7, 2016 at 10:58 AM, Ashish Singh  wrote:
>
> >
> >
> > On Fri, Mar 4, 2016 at 5:46 PM, Jay Kreps  wrote:
> >
> >> Hey Ashish,
> >>
> >> Both good points.
> >>
> >> I think the issue with the general metadata request is the same as the
> >> issue with a version-specific metadata request from the other
> >> proposal--basically it's a chicken and egg problem, to find out anything
> >> about the cluster you have to be able to communicate something in a
> format
> >> the server can understand without knowing a priori what version it's
> on. I
> >> guess the question is how can you continue to evolve the metadata
> request
> >> (whether it is the existing metadata or a protocol-version specific
> >> metadata request) given that you need this information to bootstrap you
> >> have to be more careful in how that request evolves.
> >>
> > You are correct. It's just that protocol version request would be very
> > specific to retrieve the protocol versions. Changes to protocol version
> > request itself should be very rare, if at all. However, the general
> > metadata request carries a lot more information and its format is more
> > probable to evolve. This boils down to higher probability of change vs a
> > definite network round-trip for each re/connect. It does sound like, it
> is
> > better to avoid a definite penalty than to avoid a probable rare issue.
> >
> >>
> >> I think deprecation/removal may be okay. Ultimately clients will always
> >> use
> >> the highest possible version of the protocol the server supports so if
> >> we've already deprecated and removed your highest version then you are
> >> screwed and you're going to get an error no matter what, right?
> Basically
> >> there is nothing dynamic you can do in that case.
> >>
> > Sure, this should be expected. Just wanted to make sure deprecation is
> > still on the table.
> >
> >>
> >> -Jay
> >>
> >> On Fri, Mar 4, 2016 at 4:05 PM, Ashish Singh 
> wrote:
> >>
> >> > Hello Jay,
> >> >
> >> > The overall approach sounds good. I do realize that this discussion
> has
> >> > gotten too lengthy and is starting to shoot tangents. Maybe a KIP call
> >> will
> >> > help us getting to a decision faster. I do have a few questions
> though.
> >> >
> >> > On Fri, Mar 4, 2016 at 9:52 AM, Jay Kreps  

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

@Magnus,

Does the latest suggestion sound OK to you. I am planning to update PR
based on latest suggestion.

On Mon, Mar 7, 2016 at 10:58 AM, Ashish Singh  wrote:

>
>
> On Fri, Mar 4, 2016 at 5:46 PM, Jay Kreps  wrote:
>
>> Hey Ashish,
>>
>> Both good points.
>>
>> I think the issue with the general metadata request is the same as the
>> issue with a version-specific metadata request from the other
>> proposal--basically it's a chicken and egg problem, to find out anything
>> about the cluster you have to be able to communicate something in a format
>> the server can understand without knowing a priori what version it's on. I
>> guess the question is how can you continue to evolve the metadata request
>> (whether it is the existing metadata or a protocol-version specific
>> metadata request) given that you need this information to bootstrap you
>> have to be more careful in how that request evolves.
>>
> You are correct. It's just that protocol version request would be very
> specific to retrieve the protocol versions. Changes to protocol version
> request itself should be very rare, if at all. However, the general
> metadata request carries a lot more information and its format is more
> probable to evolve. This boils down to higher probability of change vs a
> definite network round-trip for each re/connect. It does sound like, it is
> better to avoid a definite penalty than to avoid a probable rare issue.
>
>>
>> I think deprecation/removal may be okay. Ultimately clients will always
>> use
>> the highest possible version of the protocol the server supports so if
>> we've already deprecated and removed your highest version then you are
>> screwed and you're going to get an error no matter what, right? Basically
>> there is nothing dynamic you can do in that case.
>>
> Sure, this should be expected. Just wanted to make sure deprecation is
> still on the table.
>
>>
>> -Jay
>>
>> On Fri, Mar 4, 2016 at 4:05 PM, Ashish Singh  wrote:
>>
>> > Hello Jay,
>> >
>> > The overall approach sounds good. I do realize that this discussion has
>> > gotten too lengthy and is starting to shoot tangents. Maybe a KIP call
>> will
>> > help us getting to a decision faster. I do have a few questions though.
>> >
>> > On Fri, Mar 4, 2016 at 9:52 AM, Jay Kreps  wrote:
>> >
>> > > Yeah here is my summary of my take:
>> > >
>> > > 1. Negotiating a per-connection protocol actually does add a lot of
>> > > complexity to clients (many more failure states to get right).
>> > >
>> > > 2. Having the client configure the protocol version manually is doable
>> > now
>> > > but probably a worse state. I suspect this will lead to more not less
>> > > confusion.
>> > >
>> > > 3. I don't think the current state is actually that bad. Integrators
>> > pick a
>> > > conservative version and build against that. There is a tradeoff
>> between
>> > > getting the new features and being compatible with old Kafka versions.
>> > But
>> > > a large part of this tradeoff is essential since new features aren't
>> > going
>> > > to magically appear on old servers, so even if you upgrade your client
>> > you
>> > > likely aren't going to get the new stuff (since we will end up
>> > dynamically
>> > > turning it off). Having client features that are there but don't work
>> > > because you're on an old cluster may actually be a worse experience if
>> > not
>> > > handled very carefully..
>> > >
>> > > 4. The problems Dana brought up are totally orthogonal to the problem
>> of
>> > > having per-api versions or overall versions. The problem was that we
>> > > changed behavior subtly without changing the version. This will be an
>> > issue
>> > > regardless of whether the version is global or not.
>> > >
>> > > 5. Using the broker release as the version is strictly worse than
>> using a
>> > > global protocol version (0, 1, 2, ...) that increments any time any
>> api
>> > > changes but doesn't increment just because non-protocol code is
>> changed.
>> > > The problem with using the broker release version is we want to be
>> able
>> > to
>> > > keep Kafka releasable from any commit which means there isn't as
>> clear a
>> > > sequencing of releases as you would think.
>> > >
>> > > 6. We need to consider the case of mixed version clusters during the
>> time
>> > > period when you are upgrading Kafka.
>> > >
>> > > So overall I think this is not a critical thing to do right now, but
>> if
>> > we
>> > > are going to do it we should do it in a way that actually improves
>> > things.
>> > >
>> > > Here would be one proposal for that:
>> > > a. Add a global protocol version that increments with any api version
>> > > update. Move the documentation so that the docs are by version. This
>> is
>> > > basically just a short-hand for a complete set of supported api
>> versions.
>> > > b. Include a field in the metadata response for each broker that adds
>> the
>> > > protocol version.
>> 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

On Fri, Mar 4, 2016 at 5:46 PM, Jay Kreps  wrote:

> Hey Ashish,
>
> Both good points.
>
> I think the issue with the general metadata request is the same as the
> issue with a version-specific metadata request from the other
> proposal--basically it's a chicken and egg problem, to find out anything
> about the cluster you have to be able to communicate something in a format
> the server can understand without knowing a priori what version it's on. I
> guess the question is how can you continue to evolve the metadata request
> (whether it is the existing metadata or a protocol-version specific
> metadata request) given that you need this information to bootstrap you
> have to be more careful in how that request evolves.
>
You are correct. It's just that protocol version request would be very
specific to retrieve the protocol versions. Changes to protocol version
request itself should be very rare, if at all. However, the general
metadata request carries a lot more information and its format is more
probable to evolve. This boils down to higher probability of change vs a
definite network round-trip for each re/connect. It does sound like, it is
better to avoid a definite penalty than to avoid a probable rare issue.

>
> I think deprecation/removal may be okay. Ultimately clients will always use
> the highest possible version of the protocol the server supports so if
> we've already deprecated and removed your highest version then you are
> screwed and you're going to get an error no matter what, right? Basically
> there is nothing dynamic you can do in that case.
>
Sure, this should be expected. Just wanted to make sure deprecation is
still on the table.

>
> -Jay
>
> On Fri, Mar 4, 2016 at 4:05 PM, Ashish Singh  wrote:
>
> > Hello Jay,
> >
> > The overall approach sounds good. I do realize that this discussion has
> > gotten too lengthy and is starting to shoot tangents. Maybe a KIP call
> will
> > help us getting to a decision faster. I do have a few questions though.
> >
> > On Fri, Mar 4, 2016 at 9:52 AM, Jay Kreps  wrote:
> >
> > > Yeah here is my summary of my take:
> > >
> > > 1. Negotiating a per-connection protocol actually does add a lot of
> > > complexity to clients (many more failure states to get right).
> > >
> > > 2. Having the client configure the protocol version manually is doable
> > now
> > > but probably a worse state. I suspect this will lead to more not less
> > > confusion.
> > >
> > > 3. I don't think the current state is actually that bad. Integrators
> > pick a
> > > conservative version and build against that. There is a tradeoff
> between
> > > getting the new features and being compatible with old Kafka versions.
> > But
> > > a large part of this tradeoff is essential since new features aren't
> > going
> > > to magically appear on old servers, so even if you upgrade your client
> > you
> > > likely aren't going to get the new stuff (since we will end up
> > dynamically
> > > turning it off). Having client features that are there but don't work
> > > because you're on an old cluster may actually be a worse experience if
> > not
> > > handled very carefully..
> > >
> > > 4. The problems Dana brought up are totally orthogonal to the problem
> of
> > > having per-api versions or overall versions. The problem was that we
> > > changed behavior subtly without changing the version. This will be an
> > issue
> > > regardless of whether the version is global or not.
> > >
> > > 5. Using the broker release as the version is strictly worse than
> using a
> > > global protocol version (0, 1, 2, ...) that increments any time any api
> > > changes but doesn't increment just because non-protocol code is
> changed.
> > > The problem with using the broker release version is we want to be able
> > to
> > > keep Kafka releasable from any commit which means there isn't as clear
> a
> > > sequencing of releases as you would think.
> > >
> > > 6. We need to consider the case of mixed version clusters during the
> time
> > > period when you are upgrading Kafka.
> > >
> > > So overall I think this is not a critical thing to do right now, but if
> > we
> > > are going to do it we should do it in a way that actually improves
> > things.
> > >
> > > Here would be one proposal for that:
> > > a. Add a global protocol version that increments with any api version
> > > update. Move the documentation so that the docs are by version. This is
> > > basically just a short-hand for a complete set of supported api
> versions.
> > > b. Include a field in the metadata response for each broker that adds
> the
> > > protocol version.
> > >
> > There might be an issue here where the metadata request version sent by
> > client is not supported by broker, an older broker. However, if we are
> > clearly stating that a client is not guaranteed to work with an older
> > broker then this becomes expected. This will potentially limit us in
> terms
> > of