Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Ewen Cheslack-Postava]

On Sat, Dec 10, 2016 at 5:23 PM, Sven Ludwig  wrote:

> Hi,
>
> as a user I find this concept interesting, but not important enough. I
> just want to share my personal thoughts, potentially leaning myself
> somewhat out of the window, that came to my mind during reading:
>
> - I think it is of utmost importance for the Kafka project to continue the
> good work on easing and really securing the upgrading of the broker version
> via rolling upgrade. It is really cool for users when they can rely on
> that, allowing for continuous tech refresh, even on brokers with really
> large and/or really many partitions. I would always prefer this to stay in
> front.
>
> - Secondly, I, as most users, like a clear, easy to understand, yet
> powerful API contract together with producer/consumer configuration,
> including also aspects such as load-balancing and availability (consumer
> groups and alternative solutions via external partitioning and
> schedulers/supervisors respectively etc.).
>
> - Third, I find it important for the Kafka community to continue support
> for Scala-based APIs on the client side with the same devotion as seen in
> the support for the Java-based Kafka Streams API.
>
> - Fourth, I need production readiness features.
>
> Only after all these, I would look at a possibility to use a new client
> together with an old broker. Even though the concept seems to be sound on
> first read, I am somewhat afraid of the additional complexity, resulting
> complications and thus additional work for people. Think of database
> drivers. It can potentially become really complicated to support feature
> handshaking. Also, runtime exceptions that could have been avoided by a
> clear and simple policy would not be appealing. I really wonder if there
> are actually enough users who would need the ability to use a newer client
> with an older broker, to justify the increase in complexity.
>

This is something we see come up very regularly on the mailing list. It's
almost always a problem when you have one team that manages the broker
infrastructure and other teams that write apps against it -- the former
tend to be conservative with upgrades, the latter want new features even if
they are client-side only. There are a ton of folks looking for a) bug
fixes in producer/consumer that are newer than their broker (but unrelated
to broker-side fixes), b) support for connectors that are written against
newer versions of the Connect API than their broker (which rarely have any
coupled changes), and c) support using streams apps against older versions
(which also rarely have coupled changes).

This KIP covers the core clients, which actually solves that vast majority
of problems for Connect & Streams as well.

In terms of effort, the delta isn't exactly the same because librdkafka was
already setup for some compatibility, but KIP-35 support there was a
relatively small patch given what it was adding:
https://github.com/edenhill/librdkafka/commit/763b5220905246987c08fc86ea65c1df7d1cc9a9
There have been minimal follow ups required after that patch. Kafka also
has the benefit of substantial cross-version system testing already being
in place, so getting decent realistic validation of compatibility will
actually be pretty low cost (as described in the KIP) and will provide
ongoing validation.


>
> I would agree however if someone said that it should better be possible to
> use two different client versions within one JVM, in order to talk to
> several brokers of different versions, which could become easier to achieve
> at the client side under Java 9 (Jigsaw).
>

With this KIP you don't need to support that because a single newer client
will be able to talk to a variety of broker versions anyway.

-Ewen


>
> Kind Regards
> Sven
>
>

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Sven Ludwig]

Hi,
 
as a user I find this concept interesting, but not important enough. I just 
want to share my personal thoughts, potentially leaning myself somewhat out of 
the window, that came to my mind during reading:
 
- I think it is of utmost importance for the Kafka project to continue the good 
work on easing and really securing the upgrading of the broker version via 
rolling upgrade. It is really cool for users when they can rely on that, 
allowing for continuous tech refresh, even on brokers with really large and/or 
really many partitions. I would always prefer this to stay in front.
 
- Secondly, I, as most users, like a clear, easy to understand, yet powerful 
API contract together with producer/consumer configuration, including also 
aspects such as load-balancing and availability (consumer groups and 
alternative solutions via external partitioning and schedulers/supervisors 
respectively etc.).
 
- Third, I find it important for the Kafka community to continue support for 
Scala-based APIs on the client side with the same devotion as seen in the 
support for the Java-based Kafka Streams API.
 
- Fourth, I need production readiness features.
 
Only after all these, I would look at a possibility to use a new client 
together with an old broker. Even though the concept seems to be sound on first 
read, I am somewhat afraid of the additional complexity, resulting 
complications and thus additional work for people. Think of database drivers. 
It can potentially become really complicated to support feature handshaking. 
Also, runtime exceptions that could have been avoided by a clear and simple 
policy would not be appealing. I really wonder if there are actually enough 
users who would need the ability to use a newer client with an older broker, to 
justify the increase in complexity.
 
I would agree however if someone said that it should better be possible to use 
two different client versions within one JVM, in order to talk to several 
brokers of different versions, which could become easier to achieve at the 
client side under Java 9 (Jigsaw).
 
Kind Regards
Sven
 

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Colin McCabe]

On Fri, Dec 2, 2016, at 13:41, Ashish Singh wrote:
> Colin,
> 
> I just rebased KAFKA-3600's PR on trunk.
> 

Thanks, Ashish!

> KAFKA-4457 is a good idea, however it is probably doing some redundant
> work. AdminClient has a network client that will already have api
> versions
> info (after KAFKA-3600 goes in), so I do not think we need to send out
> another ApiVersions request over wire.
> 

That's a good point.  As long as we can talk to all brokers.  Let's
continue the discussion on Jira.

I'll start a vote thread for KIP-97.

Best,
Colin

> On Wed, Nov 30, 2016 at 5:33 PM, Colin McCabe  wrote:
> 
> > Hi all,
> >
> > I updated the KIP to include a command to print out the version
> > information of brokers (KAFKA-4457).  This will be a useful command for
> > administrators.
> >
> > best,
> > Colin
> >
> >
> > On Wed, Nov 30, 2016, at 11:17, Colin McCabe wrote:
> > > Thanks, Ashish.  I think the idea of having the client make an
> > > ApiVersionRequest call when it starts up is a good one.  This idea is
> > > described in both KIP-97, and the KAFKA-3600 patches.  I also think we
> > > ought to maintain per-node version information.  It would be good to get
> > > that in so that we can use it as a building block for the stuff
> > > described in this KIP.  I'd be happy to review it.
> > >
> > > Colin
> > >
> > >
> > > On Wed, Nov 30, 2016, at 10:40, Ashish Singh wrote:
> > > > Hello Ismael,
> > > >
> > > > It is good to know that you are willing to review KAFKA-3600 again. As
> > > > before, we at Cloudera are highly in support of client compatibility,
> > and
> > > > KAFKA-3600 has always been a building block for that. Now that client
> > > > compatibility is at forefront again, thanks to Colin, I will be happy
> > to
> > > > rebase KAFKA-3600 on trunk. However, any comments/ feedback on
> > > > KAFKA-3600's
> > > > approach on maintaining api version info for each connected broker is
> > > > highly welcome.
> > > >
> > > > On Wed, Nov 30, 2016 at 5:47 AM, Ismael Juma 
> > wrote:
> > > >
> > > > > Thanks Colin, I think this is a good improvement.
> > > > >
> > > > > Ashish, some of the concerns with regards to KAFKA-3600 were related
> > to the
> > > > > cost versus benefit. Once one adds client compatibility, the benefit
> > is
> > > > > much higher. I would be happy to review and merge KAFKA-3600 if we
> > think it
> > > > > serves as a good first step towards client compatibility (if the
> > vote for
> > > > > this passes). Colin, maybe you can review the PR for KAFKA-3600 and
> > see if
> > > > > you can build on that? Ashish, it may be worth merging trunk into
> > your
> > > > > branch and fixing the conflicts.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh 
> > wrote:
> > > > >
> > > > > > Hello Colin,
> > > > > >
> > > > > > In the KIP you mentioned that currently the client uses supported
> > api
> > > > > > versions information to check if the server supports its desired
> > > > > versions.
> > > > > > Not sure, if that is true. I had put together a PR for KAFKA-3600,
> > to do
> > > > > > that, but it never went in. Also, I could not find how you plan to
> > > > > perform
> > > > > > version check on client side. In KAFKA-3600, I am maintaining api
> > version
> > > > > > for each live connection, and that made a few folks think it is
> > too big
> > > > > of
> > > > > > a change.
> > > > > >
> > > > > > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe  > >
> > > > > wrote:
> > > > > >
> > > > > > > Sorry, that link should be:
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I've been thinking about a KIP to improve the Kafka client's
> > > > > > > > compatibility policy.  If you're interested, please check out:
> > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > > > > > >
> > > > > > > > cheers,
> > > > > > > > Colin
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Regards,
> > > > > > Ashish
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashish
> >
> 
> 
> 
> -- 
> 
> Regards,
> Ashish

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Ashish Singh]

Colin,

I just rebased KAFKA-3600's PR on trunk.

KAFKA-4457 is a good idea, however it is probably doing some redundant
work. AdminClient has a network client that will already have api versions
info (after KAFKA-3600 goes in), so I do not think we need to send out
another ApiVersions request over wire.

On Wed, Nov 30, 2016 at 5:33 PM, Colin McCabe  wrote:

> Hi all,
>
> I updated the KIP to include a command to print out the version
> information of brokers (KAFKA-4457).  This will be a useful command for
> administrators.
>
> best,
> Colin
>
>
> On Wed, Nov 30, 2016, at 11:17, Colin McCabe wrote:
> > Thanks, Ashish.  I think the idea of having the client make an
> > ApiVersionRequest call when it starts up is a good one.  This idea is
> > described in both KIP-97, and the KAFKA-3600 patches.  I also think we
> > ought to maintain per-node version information.  It would be good to get
> > that in so that we can use it as a building block for the stuff
> > described in this KIP.  I'd be happy to review it.
> >
> > Colin
> >
> >
> > On Wed, Nov 30, 2016, at 10:40, Ashish Singh wrote:
> > > Hello Ismael,
> > >
> > > It is good to know that you are willing to review KAFKA-3600 again. As
> > > before, we at Cloudera are highly in support of client compatibility,
> and
> > > KAFKA-3600 has always been a building block for that. Now that client
> > > compatibility is at forefront again, thanks to Colin, I will be happy
> to
> > > rebase KAFKA-3600 on trunk. However, any comments/ feedback on
> > > KAFKA-3600's
> > > approach on maintaining api version info for each connected broker is
> > > highly welcome.
> > >
> > > On Wed, Nov 30, 2016 at 5:47 AM, Ismael Juma 
> wrote:
> > >
> > > > Thanks Colin, I think this is a good improvement.
> > > >
> > > > Ashish, some of the concerns with regards to KAFKA-3600 were related
> to the
> > > > cost versus benefit. Once one adds client compatibility, the benefit
> is
> > > > much higher. I would be happy to review and merge KAFKA-3600 if we
> think it
> > > > serves as a good first step towards client compatibility (if the
> vote for
> > > > this passes). Colin, maybe you can review the PR for KAFKA-3600 and
> see if
> > > > you can build on that? Ashish, it may be worth merging trunk into
> your
> > > > branch and fixing the conflicts.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh 
> wrote:
> > > >
> > > > > Hello Colin,
> > > > >
> > > > > In the KIP you mentioned that currently the client uses supported
> api
> > > > > versions information to check if the server supports its desired
> > > > versions.
> > > > > Not sure, if that is true. I had put together a PR for KAFKA-3600,
> to do
> > > > > that, but it never went in. Also, I could not find how you plan to
> > > > perform
> > > > > version check on client side. In KAFKA-3600, I am maintaining api
> version
> > > > > for each live connection, and that made a few folks think it is
> too big
> > > > of
> > > > > a change.
> > > > >
> > > > > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe  >
> > > > wrote:
> > > > >
> > > > > > Sorry, that link should be:
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I've been thinking about a KIP to improve the Kafka client's
> > > > > > > compatibility policy.  If you're interested, please check out:
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > > > > >
> > > > > > > cheers,
> > > > > > > Colin
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Regards,
> > > > > Ashish
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashish
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Colin McCabe]

Hi all,

I updated the KIP to include a command to print out the version
information of brokers (KAFKA-4457).  This will be a useful command for
administrators.

best,
Colin


On Wed, Nov 30, 2016, at 11:17, Colin McCabe wrote:
> Thanks, Ashish.  I think the idea of having the client make an
> ApiVersionRequest call when it starts up is a good one.  This idea is
> described in both KIP-97, and the KAFKA-3600 patches.  I also think we
> ought to maintain per-node version information.  It would be good to get
> that in so that we can use it as a building block for the stuff
> described in this KIP.  I'd be happy to review it.
> 
> Colin
> 
> 
> On Wed, Nov 30, 2016, at 10:40, Ashish Singh wrote:
> > Hello Ismael,
> > 
> > It is good to know that you are willing to review KAFKA-3600 again. As
> > before, we at Cloudera are highly in support of client compatibility, and
> > KAFKA-3600 has always been a building block for that. Now that client
> > compatibility is at forefront again, thanks to Colin, I will be happy to
> > rebase KAFKA-3600 on trunk. However, any comments/ feedback on
> > KAFKA-3600's
> > approach on maintaining api version info for each connected broker is
> > highly welcome.
> > 
> > On Wed, Nov 30, 2016 at 5:47 AM, Ismael Juma  wrote:
> > 
> > > Thanks Colin, I think this is a good improvement.
> > >
> > > Ashish, some of the concerns with regards to KAFKA-3600 were related to 
> > > the
> > > cost versus benefit. Once one adds client compatibility, the benefit is
> > > much higher. I would be happy to review and merge KAFKA-3600 if we think 
> > > it
> > > serves as a good first step towards client compatibility (if the vote for
> > > this passes). Colin, maybe you can review the PR for KAFKA-3600 and see if
> > > you can build on that? Ashish, it may be worth merging trunk into your
> > > branch and fixing the conflicts.
> > >
> > > Ismael
> > >
> > > On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh  wrote:
> > >
> > > > Hello Colin,
> > > >
> > > > In the KIP you mentioned that currently the client uses supported api
> > > > versions information to check if the server supports its desired
> > > versions.
> > > > Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> > > > that, but it never went in. Also, I could not find how you plan to
> > > perform
> > > > version check on client side. In KAFKA-3600, I am maintaining api 
> > > > version
> > > > for each live connection, and that made a few folks think it is too big
> > > of
> > > > a change.
> > > >
> > > > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe 
> > > wrote:
> > > >
> > > > > Sorry, that link should be:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I've been thinking about a KIP to improve the Kafka client's
> > > > > > compatibility policy.  If you're interested, please check out:
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > > > >
> > > > > > cheers,
> > > > > > Colin
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashish
> > > >
> > >
> > 
> > 
> > 
> > -- 
> > 
> > Regards,
> > Ashish

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Colin McCabe]

Thanks, Ashish.  I think the idea of having the client make an
ApiVersionRequest call when it starts up is a good one.  This idea is
described in both KIP-97, and the KAFKA-3600 patches.  I also think we
ought to maintain per-node version information.  It would be good to get
that in so that we can use it as a building block for the stuff
described in this KIP.  I'd be happy to review it.

Colin


On Wed, Nov 30, 2016, at 10:40, Ashish Singh wrote:
> Hello Ismael,
> 
> It is good to know that you are willing to review KAFKA-3600 again. As
> before, we at Cloudera are highly in support of client compatibility, and
> KAFKA-3600 has always been a building block for that. Now that client
> compatibility is at forefront again, thanks to Colin, I will be happy to
> rebase KAFKA-3600 on trunk. However, any comments/ feedback on
> KAFKA-3600's
> approach on maintaining api version info for each connected broker is
> highly welcome.
> 
> On Wed, Nov 30, 2016 at 5:47 AM, Ismael Juma  wrote:
> 
> > Thanks Colin, I think this is a good improvement.
> >
> > Ashish, some of the concerns with regards to KAFKA-3600 were related to the
> > cost versus benefit. Once one adds client compatibility, the benefit is
> > much higher. I would be happy to review and merge KAFKA-3600 if we think it
> > serves as a good first step towards client compatibility (if the vote for
> > this passes). Colin, maybe you can review the PR for KAFKA-3600 and see if
> > you can build on that? Ashish, it may be worth merging trunk into your
> > branch and fixing the conflicts.
> >
> > Ismael
> >
> > On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh  wrote:
> >
> > > Hello Colin,
> > >
> > > In the KIP you mentioned that currently the client uses supported api
> > > versions information to check if the server supports its desired
> > versions.
> > > Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> > > that, but it never went in. Also, I could not find how you plan to
> > perform
> > > version check on client side. In KAFKA-3600, I am maintaining api version
> > > for each live connection, and that made a few folks think it is too big
> > of
> > > a change.
> > >
> > > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe 
> > wrote:
> > >
> > > > Sorry, that link should be:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > > >
> > > >
> > > >
> > > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > > Hi all,
> > > > >
> > > > > I've been thinking about a KIP to improve the Kafka client's
> > > > > compatibility policy.  If you're interested, please check out:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > > >
> > > > > cheers,
> > > > > Colin
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashish
> > >
> >
> 
> 
> 
> -- 
> 
> Regards,
> Ashish

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Ashish Singh]

Hello Ismael,

It is good to know that you are willing to review KAFKA-3600 again. As
before, we at Cloudera are highly in support of client compatibility, and
KAFKA-3600 has always been a building block for that. Now that client
compatibility is at forefront again, thanks to Colin, I will be happy to
rebase KAFKA-3600 on trunk. However, any comments/ feedback on KAFKA-3600's
approach on maintaining api version info for each connected broker is
highly welcome.

On Wed, Nov 30, 2016 at 5:47 AM, Ismael Juma  wrote:

> Thanks Colin, I think this is a good improvement.
>
> Ashish, some of the concerns with regards to KAFKA-3600 were related to the
> cost versus benefit. Once one adds client compatibility, the benefit is
> much higher. I would be happy to review and merge KAFKA-3600 if we think it
> serves as a good first step towards client compatibility (if the vote for
> this passes). Colin, maybe you can review the PR for KAFKA-3600 and see if
> you can build on that? Ashish, it may be worth merging trunk into your
> branch and fixing the conflicts.
>
> Ismael
>
> On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh  wrote:
>
> > Hello Colin,
> >
> > In the KIP you mentioned that currently the client uses supported api
> > versions information to check if the server supports its desired
> versions.
> > Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> > that, but it never went in. Also, I could not find how you plan to
> perform
> > version check on client side. In KAFKA-3600, I am maintaining api version
> > for each live connection, and that made a few folks think it is too big
> of
> > a change.
> >
> > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe 
> wrote:
> >
> > > Sorry, that link should be:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > >
> > >
> > >
> > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > Hi all,
> > > >
> > > > I've been thinking about a KIP to improve the Kafka client's
> > > > compatibility policy.  If you're interested, please check out:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > >
> > > > cheers,
> > > > Colin
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Ismael Juma]

Thanks Colin, I think this is a good improvement.

Ashish, some of the concerns with regards to KAFKA-3600 were related to the
cost versus benefit. Once one adds client compatibility, the benefit is
much higher. I would be happy to review and merge KAFKA-3600 if we think it
serves as a good first step towards client compatibility (if the vote for
this passes). Colin, maybe you can review the PR for KAFKA-3600 and see if
you can build on that? Ashish, it may be worth merging trunk into your
branch and fixing the conflicts.

Ismael

On Tue, Nov 29, 2016 at 7:39 PM, Ashish Singh  wrote:

> Hello Colin,
>
> In the KIP you mentioned that currently the client uses supported api
> versions information to check if the server supports its desired versions.
> Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> that, but it never went in. Also, I could not find how you plan to perform
> version check on client side. In KAFKA-3600, I am maintaining api version
> for each live connection, and that made a few folks think it is too big of
> a change.
>
> On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe  wrote:
>
> > Sorry, that link should be:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> >
> >
> >
> > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > Hi all,
> > >
> > > I've been thinking about a KIP to improve the Kafka client's
> > > compatibility policy.  If you're interested, please check out:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 97%3A+Improved+Kafka+Compatibility+Policy
> > >
> > > cheers,
> > > Colin
> >
>
>
>
> --
>
> Regards,
> Ashish
>

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Ashish Singh]

On Tue, Nov 29, 2016 at 4:11 PM, Colin McCabe  wrote:

> On Tue, Nov 29, 2016, at 11:39, Ashish Singh wrote:
> > Hello Colin,
> >
> > In the KIP you mentioned that currently the client uses supported api
> > versions information to check if the server supports its desired
> > versions.
> > Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> > that, but it never went in.
>
> I was taking a look at the RPC implementation today myself and I think
> you're right.  I don't think the client actually invokes
> ApiVersionRequest-- or if it does, I can't find the invocation site.
> Still, the API versions are sent over the wire, and the server does use
> them to support older clients.  I suppose this means that currently,
> clients which are too new fail at the time they attempt to send an RPC
> which is not in the server's supported version range for that RPC.  I
> will edit the KIP to reflect this-- thanks.
>
> > Also, I could not find how you plan to perform
> > version check on client side. In KAFKA-3600, I am maintaining api version
> > for each live connection, and that made a few folks think it is too big
> > of a change.
>
> The version check would be performed by calling ApiVersionRequest, just
> like in your patch.
>
> I think that people are coming around to the idea that we need both
> forward and backwards compatibility for the client.  As the project
> grows, and there are bigger and more deployments, it is probably worth
> spending a little more time on compatibility...
>
> P.S. If you have any spare cycles, it would be great to collaborate on
> this!
>
Kafka-3600 was part of voted-in KIP-35, and a pre-requisite for client
compat. If people are coming around to the idea, maybe we can revisit
KAFKA-3600 and try to find why is it not meaningful to have it in? If we
get KAFKA-3600 in, follow up changes to enable feature selection in clients
should be pretty straight forward.

>
> cheers,
> Colin
>
> >
> > On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe 
> > wrote:
> >
> > > Sorry, that link should be:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > >
> > >
> > >
> > > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > > Hi all,
> > > >
> > > > I've been thinking about a KIP to improve the Kafka client's
> > > > compatibility policy.  If you're interested, please check out:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 97%3A+Improved+Kafka+Compatibility+Policy
> > > >
> > > > cheers,
> > > > Colin
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Colin McCabe]

On Tue, Nov 29, 2016, at 11:39, Ashish Singh wrote:
> Hello Colin,
> 
> In the KIP you mentioned that currently the client uses supported api
> versions information to check if the server supports its desired
> versions.
> Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
> that, but it never went in.

I was taking a look at the RPC implementation today myself and I think
you're right.  I don't think the client actually invokes
ApiVersionRequest-- or if it does, I can't find the invocation site. 
Still, the API versions are sent over the wire, and the server does use
them to support older clients.  I suppose this means that currently,
clients which are too new fail at the time they attempt to send an RPC
which is not in the server's supported version range for that RPC.  I
will edit the KIP to reflect this-- thanks.

> Also, I could not find how you plan to perform
> version check on client side. In KAFKA-3600, I am maintaining api version
> for each live connection, and that made a few folks think it is too big
> of a change.

The version check would be performed by calling ApiVersionRequest, just
like in your patch.

I think that people are coming around to the idea that we need both
forward and backwards compatibility for the client.  As the project
grows, and there are bigger and more deployments, it is probably worth
spending a little more time on compatibility...

P.S. If you have any spare cycles, it would be great to collaborate on
this!

cheers,
Colin

> 
> On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe 
> wrote:
> 
> > Sorry, that link should be:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> >
> >
> >
> > On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > > Hi all,
> > >
> > > I've been thinking about a KIP to improve the Kafka client's
> > > compatibility policy.  If you're interested, please check out:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 97%3A+Improved+Kafka+Compatibility+Policy
> > >
> > > cheers,
> > > Colin
> >
> 
> 
> 
> -- 
> 
> Regards,
> Ashish

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Ashish Singh]

Hello Colin,

In the KIP you mentioned that currently the client uses supported api
versions information to check if the server supports its desired versions.
Not sure, if that is true. I had put together a PR for KAFKA-3600, to do
that, but it never went in. Also, I could not find how you plan to perform
version check on client side. In KAFKA-3600, I am maintaining api version
for each live connection, and that made a few folks think it is too big of
a change.

On Tue, Nov 29, 2016 at 11:05 AM, Colin McCabe  wrote:

> Sorry, that link should be:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
>
>
>
> On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> > Hi all,
> >
> > I've been thinking about a KIP to improve the Kafka client's
> > compatibility policy.  If you're interested, please check out:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 97%3A+Improved+Kafka+Compatibility+Policy
> >
> > cheers,
> > Colin
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Colin McCabe]

Sorry, that link should be:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy



On Tue, Nov 29, 2016, at 11:04, Colin McCabe wrote:
> Hi all,
> 
> I've been thinking about a KIP to improve the Kafka client's
> compatibility policy.  If you're interested, please check out: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Compatibility+Policy
> 
> cheers,
> Colin

[DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

[Colin McCabe]

Hi all,

I've been thinking about a KIP to improve the Kafka client's
compatibility policy.  If you're interested, please check out: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Compatibility+Policy

cheers,
Colin