Thanks Ying for updating the KIP.
Hi Ismael,
Given min.api.version allows admin/users to specifiy min.version
for each request this should address your concerns right?
Thanks,
Harsha
On Wed, Apr 17, 2019, at 2:29 PM, Ying Zheng wrote:
> I have updated the config description in the KIP, made the example more
> clear
>
> The proposed change allows setting different min versions for different
> APIs, and the ApiVersionRequest change is already in the KIP.
>
> On Fri, Apr 12, 2019 at 8:22 PM Harsha <ka...@harsha.io> wrote:
>
> > Hi Ismael,
> > I meant to say blocking clients based on their API version
> > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L48
> > But If I understand what you are saying, since each client release can
> > support different versions for each of fetch, produce, offset commit etc..
> > and it's harder to block just based on single min.api.version setting
> > across different clients.
> > The idea I had in my mind was to do this via ApiVersionRequest, when a
> > client makes api request to broker in response we return min and max
> > version supported for each Api. When min.api.version enabled on broker, it
> > returns the maxVersion it supports for each of the requests in that release
> > as min versions to the clients.
> >
> > Example:
> > Kafka 1.1.1 broker and min.api.verson set to
> > https://github.com/apache/kafka/blob/1.1/core/src/main/scala/kafka/api/ApiVersion.scala#L79
> > (KAFKA_1_1_IV0) and client makes a ApiVersionsRequest and in response for
> > example produce request
> >
> > https://github.com/apache/kafka/blob/1.1/clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java#L112
> > Instead of returning all of the supported versions it will return
> > PRODUCE_REQUEST_V5 as the only supported version.
> >
> > Irrespective of the above approach I understand your point still stands
> > which is sarama might not choose to implement all the higher version
> > protocols for Kafka 1.1 release and they might introduce higher version of
> > produce request in a subsequent minor release and it will be harder for
> > users to figure out which release of sarama client they can use.
> >
> >
> > Ying, if you have a different apporach which might address this issue
> > please add.
> >
> >
> > Thanks,
> > Harsha
> >
> > On Fri, Apr 12, 2019, at 7:23 PM, Ismael Juma wrote:
> > > Hi Harsha,
> > >
> > > There is no such thing as 1.1 protocol. I encourage you to describe an
> > > example config that achieves what you are suggesting here. It's pretty
> > > complicated because the versions are per API and each client evolves
> > > independently.
> > >
> > > Ismael
> > >
> > > On Sat, Apr 13, 2019 at 4:09 AM Harsha <ka...@harsha.io> wrote:
> > >
> > > > Hi,
> > > >
> > > > "Relying on min.version seems like a pretty clunky way to achieve the
> > above
> > > > > list. The challenge is that it's pretty difficult to do it in a way
> > that
> > > > > works for clients across languages. They each add support for new
> > > > protocol
> > > > > versions independently (it could even happen in a bug fix release).
> > So,
> > > > if
> > > > > you tried to block Sarama in #2, you may block Java clients too."
> > > >
> > > > That's the intended effect, right? if you as the admin/operator
> > > > configures the broker to have min.api.version to be 1.1
> > > > it should block java , sarama clients etc.. which are below the 1.1
> > > > protocol. As mentioned this is not just related to log.format upgrade
> > > > problem but in general a forcing cause to get the users to upgrade
> > their
> > > > client version in a multi-tenant environment.
> > > >
> > > > "> For #3, it seems simplest to have a config that requires clients to
> > > > support
> > > > > a given message format version (or higher). For #2, it seems like
> > you'd
> > > > > want clients to advertise their versions. That would be useful for
> > > > multiple
> > > > > reasons."
> > > > This kip offers the ability to block clients based on the protocol they
> > > > support. This should be independent of the message format upgrade. Not
> > all
> > > > of the features or bugs are dependent on a message format and having a
> > > > message format dependency to block clients means we have to upgrade to
> > > > message.format and we cannot just say we've 1.1 brokers with 0.8.2
> > message
> > > > format and now we want to block all 0.8.x clients.
> > > >
> > > > min.api.version helps at the cluster level to say that all users
> > required
> > > > to upgrade clients to the at minimum need to speak the min.api.version
> > and
> > > > not tie to message.format because not all cases one wants to upgrade
> > the
> > > > message format and block the old clients.
> > > >
> > > >
> > > > To Gwen's point, I think we should also return in the error message
> > that
> > > > the broker only supports min.api.version and above. So that users can
> > see a
> > > > clear message and upgrade to a newer version.
> > > >
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > > On Fri, Apr 12, 2019, at 12:19 PM, Ismael Juma wrote:
> > > > > Hi Ying,
> > > > >
> > > > > The actual reasons are important so that people can evaluate the KIP
> > (and
> > > > > vote). :) Thanks for providing a few more:
> > > > >
> > > > > (1) force users to check pointing in Kafka instead of zookeeper
> > > > > (2) forbid an old go (sarama) client library which is known to have
> > some
> > > > > serious bugs
> > > > > (3) force kafka 1.x clients with the ability to roll back if there's
> > an
> > > > > issue (unlike a message format upgrade)
> > > > >
> > > > > Relying on min.version seems like a pretty clunky way to achieve the
> > > > above
> > > > > list. The challenge is that it's pretty difficult to do it in a way
> > that
> > > > > works for clients across languages. They each add support for new
> > > > protocol
> > > > > versions independently (it could even happen in a bug fix release).
> > So,
> > > > if
> > > > > you tried to block Sarama in #2, you may block Java clients too.
> > > > >
> > > > > For #3, it seems simplest to have a config that requires clients to
> > > > support
> > > > > a given message format version (or higher). For #2, it seems like
> > you'd
> > > > > want clients to advertise their versions. That would be useful for
> > > > multiple
> > > > > reasons.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Fri, Apr 12, 2019 at 8:42 PM Ying Zheng <yi...@uber.com.invalid>
> > > > wrote:
> > > > >
> > > > > > Hi Ismael,
> > > > > >
> > > > > > Those are just examples. I think the administrators should be able
> > to
> > > > block
> > > > > > certain client libraries for whatever reason. Some other possible
> > > > reasons
> > > > > > include, force users to check pointing in Kafka instead of
> > zookeeper,
> > > > > > forbid an old go (sarama) client library which is known to have
> > some
> > > > > > serious bugs.
> > > > > >
> > > > > > message.downconversion.enable does not solve our problems. We are
> > now
> > > > > > planning to upgrade to message format V3, and force users to
> > upgrade to
> > > > > > Kafka 1.x clients. With the proposed min.api.version setting, in
> > case
> > > > of
> > > > > > there is anything wrong, we can roll back the setting. If we
> > upgrade
> > > > the
> > > > > > file format, there is no way to rollback (Kafka doesn't support
> > > > downgrading
> > > > > > message format).
> > > > > >
> > > > > > On Thu, Apr 11, 2019 at 7:05 PM Ismael Juma <ism...@juma.me.uk>
> > wrote:
> > > > > >
> > > > > > > Hi Ying,
> > > > > > >
> > > > > > > It looks to me that all the examples given in the KIP can be
> > handled
> > > > with
> > > > > > > the existing "message.downconversion.enable" config and by
> > > > configuring
> > > > > > the
> > > > > > > message format to be the latest:
> > > > > > >
> > > > > > > 1. Kafka 8 / 9 / 10 consumer hangs when the message contains
> > message
> > > > > > header
> > > > > > > > ( KAFKA-6739 - Down-conversion fails for records with headers
> > > > > > RESOLVED )
> > > > > > > > 2. LZ4 is not correctly handled in Kafka 8 and Kafka 9 (
> > > > KAFKA-3160 -
> > > > > > > > Kafka LZ4 framing code miscalculates header checksum RESOLVED
> > )
> > > > > > > > 3. Performance penalty of converting message format from V3 to
> > V1
> > > > or V2
> > > > > > > > for the old consumers (KIP-31 - Move to relative offsets in
> > > > compressed
> > > > > > > > message sets)
> > > > > > >
> > > > > > >
> > > > > > > Am I missing something? Are there other examples that are not
> > > > related to
> > > > > > > message conversion?
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Thu, Apr 11, 2019 at 11:53 PM Ying Zheng
> > <yi...@uber.com.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi here,
> > > > > > > >
> > > > > > > > Please vote for
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-433%3A+Block+old+clients+on+brokers
> > > > > > > >
> > > > > > > > Thank you!
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
