[ANNOUNCE] New Committer: Manikumar Reddy

[Jason Gustafson]

Hi all,

The PMC for Apache Kafka has invited Manikumar Reddy as a committer and we
are
pleased to announce that he has accepted!

Manikumar has contributed 134 commits including significant work to add
support for delegation tokens in Kafka:

KIP-48:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
KIP-249:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-249%3A+Add+Delegation+Token+Operations+to+KafkaAdminClient

He has broad experience working with many of the core components in Kafka
and he has reviewed over 80 PRs. He has also made huge progress addressing
some of our technical debt.

We appreciate the contributions and we are looking forward to more.
Congrats Manikumar!

Jason, on behalf of the Apache Kafka PMC

Re: [DISCUSS] KIP-379: Multiple Consumer Group Management

[Jason Gustafson]

Hi Alex,

Thanks for the KIP. I think it makes sense, especially since most of the
group apis are intended for batching anyway.

The only questions I have are about compatibility. For example, the csv
format for resetting offsets is changed, so will we continue to support the
old format? Also, if only one `--group` option is passed, do you think it's
worth leaving it the group column out of the `--describe` output? That
would keep the describe output consistent with the current implementation
for the current usage. Finally, and this is just a nitpick, but do you
think it makes sense to put the group column first in the describe output?

Thanks,
Jason


On Wed, Oct 3, 2018 at 7:11 AM, Alex D  wrote:

> Hello, friends!
>
> Welcome to the Multiple Consumer Group Management feature for
> kafka-consumer-groups utility discussion thread.
>
> KIP is available here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-379%
> 3A+Multiple+Consumer+Group+Management
>
> Pull Request: https://github.com/apache/kafka/pull/5726
>
> JIRA ticket: https://issues.apache.org/jira/browse/KAFKA-7471
>
> What do you think?
>
> Thanks,
> Alexander Dunayevsky
>

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

gt; > Thanks for updating the KIP. It's shaping up well. Two things:
> > > >
> > > > 1) I am a little confused about the distinction for the leader. If
> the
> > > consumer node that was assigned leader does a bounce (goes down and
> > quickly
> > > comes up) to update application code, will a rebalance be triggered? I
> do
> > > not think a bounce of the leader should trigger a rebalance.
> > > >
> > > > 2) The timeout for shrink up makes a lot of sense and allows to
> > > gracefully increase the number of nodes in the cluster. I think we need
> > to
> > > support graceful shrink down as well. If I set the registration timeout
> > to
> > > 5 minutes to handle rolling restarts or intermittent failures without
> > > shuffling state, I don't want to wait 5 minutes in order for the group
> to
> > > rebalance if I am intentionally removing a node from the cluster. I am
> > not
> > > sure the best way to do this. One idea I had was adding the ability
> for a
> > > CLI or Admin API to force a rebalance of the group. This would allow
> for
> > an
> > > admin to trigger the rebalance manually without waiting the entire
> > > registration timeout on shrink down. What do you think?
> > > >
> > > > Mike
> > > >
> > > > On 10/30/18, 1:55 AM, "Boyang Chen"  wrote:
> > > >
> > > > Btw, I updated KIP 345 based on my understanding. Feel free to
> take
> > > another round of look:
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A
> +Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> > > > KIP-345: Introduce static membership protocol to reduce ...<
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A
> +Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> > > >
> > > > cwiki.apache.org
> > > > For stateful applications, one of the biggest performance bottleneck
> is
> > > the state shuffling. In Kafka consumer, there is a concept called
> > > "rebalance" which means that for given M partitions and N consumers in
> > one
> > > consumer group, Kafka will try to balance the load between consumers
> and
> > > ideally have ...
> > > >
> > > >
> > > >
> > > >
> > > > KIP-345: Introduce static membership protocol to reduce ...<
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A
> +Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> > > >
> > > > cwiki.apache.org
> > > > For stateful applications, one of the biggest performance
> > bottleneck
> > > is the state shuffling. In Kafka consumer, there is a concept called
> > > "rebalance" which means that for given M partitions and N consumers in
> > one
> > > consumer group, Kafka will try to balance the load between consumers
> and
> > > ideally have ...
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 
> > > > From: Boyang Chen 
> > > > Sent: Monday, October 29, 2018 12:34 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer
> rebalances
> > > by specifying member id
> > > >
> > > > Thanks everyone for the input on this thread! (Sorry it's been a
> > > while) I feel that we are very close to the final solution.
> > > >
> > > >
> > > > Hey Jason and Mike, I have two quick questions on the new
> features
> > > here:
> > > >
> > > >   1.  so our proposal is that until we add a new static member
> into
> > > the group (scale up), we will not trigger rebalance until the
> > "registration
> > > timeout"( the member has been offline for too long)? How about leader's
> > > rejoin request, I think we should still trigger rebalance when that
> > > happens, since the consumer group may have new topics to consume?
> > > >   2.  I'm not very clear on the scale up scenario in static
> > > membership here. Should we fallback to dynamic membership while
> > > adding/removing hosts (by setting member.name = null), or we still
> want
> > > to add instances w

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

>
> I feel this would make the current protocol harder to use. For example, on
> KStream we typically would expect (number of instances * number of threads
> per instance) consumers. Giving out all this many member names in a list
> may not be easy, compared with dynamic generation of member names, at least
> for KStream use case.


That's a fair point. What I had in mind is for the API to specify the
number of consumers in the group. Then the ids would be determined using
some convention. Most likely, we would just use sequential numbers 0, 1, 2,
etc. We do the same thing for partition ids. Streams folks can chime in and
say whether that would work or not.

So this suggests we will not rely on heartbeat and commit interval to
> trigger rebalance? Even in static membership, I feel tracking active
> members is still required to be handled by broker since not all users are
> fully equipped with monitoring tools, otherwise users will feel
> uncomfortable using static membership solely for reducing rebalance purpose.


Let me clarify. I think it is still useful to distinguish the liveness of
the consumer using the heartbeat mechanism and to propagate that
information to the leader during rebalances. This gives the group leader
the option (but not the requirement) to change the partition assignment for
inactive members. My suggestion was more about how long the static
registration would remain valid. In the current proposal we have a
registration timeout, and I think Mayuresh is fair to point out the
potential confusion with the session timeout. It's kind of the same, but
not exactly. So instead, I was thinking that the registration is specified
ahead of time and remains valid until changed. It would be more like a
replica assignment. We don't move partitions just because a broker is down.
The expectation is that it the broker will eventually return. Similarly for
stateful applications, we would have the option to do the same thing. The
registration is fixed until someone changes it.

Does that make sense?

Thanks,
Jason


On Wed, Nov 14, 2018 at 2:46 PM, Boyang Chen  wrote:

> Thanks Jason for the suggestions! I update the KIP with full schema
> changes.
>
> > we offer an admin API that lets a user define the expected members of
> the group.
>
> I feel this would make the current protocol harder to use. For example, on
> KStream we typically would expect (number of instances * number of threads
> per instance) consumers. Giving out all this many member names in a list
> may not be easy, compared with dynamic generation of member names, at least
> for KStream use case.
>
> > For the sake of discussion, I was wondering if we could just say that
> static members do not expire.
>
>
> So this suggests we will not rely on heartbeat and commit interval to
> trigger rebalance? Even in static membership, I feel tracking active
> members is still required to be handled by broker since not all users are
> fully equipped with monitoring tools, otherwise users will feel
> uncomfortable using static membership solely for reducing rebalance purpose.
>
>
> For the admin API design, I'm simplifying the join group request handling,
> while using admin tool to switch between static and dynamic membership and
> set the two corresponding timeouts. Do you think this approach makes sense?
> The version one implementation will be much more clean if we handle
> membership change through user intervention.
>
>
> Best,
>
> Boyang
>
> 
> From: Jason Gustafson 
> Sent: Wednesday, November 14, 2018 9:31 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hey Boyang,
>
> Thanks for the updates. From a high level, I think this actually
> complements Konstantine's writeup on incremental rebalancing. The gap we're
> addressing is providing a way to bind the the partition assignment of a
> group to a set of user-provided ids so that we are not so reliant on the
> group's immediate state. For example, these ids might identify the state
> store volume for particular streams instances. This is basically what you
> need to work well with k8s stateful sets (as far as I understand them).
>
> One key decision is how we would define and update the expected static
> members in a consumer group. The mechanics of the registration and
> expansion timeouts feel a little bit clunky. For the sake of discussion, I
> was wondering if we could just say that static members do not expire.
> Instead, we offer an admin API that lets a user define the expected members
> of the group. This API could be used to both grow and shrink a group. This
> would solve the rebalancing problems when applications are initially
> bootstrap

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

ncern was that if we forfeit the timeout on static membership to trigger
> rebalance, missing some pages during midnight could be negatively impacting
> the system performance since we may realize that some partitions stop
> working for a couple of hours already until next morning. So registration
> timeout serves as the "last line of defense" to guarantee liveness if no
> human intervention jumps in.
>
>
> I'm very interested in the replication protocol currently implemented on
> Kafka, but I'm not familiar with it. If we do have mechanism to handle
> issues like I mentioned above for replication (auto healing during
> mid-night if one broker is never back), we could continue discussing the
> new approaches to have basic guarantee of consumer group liveness.
>
>
> The discussion so far is to make sure that all the design approaches we
> have taken are pointing to real scenarios. Once we clarify the scenarios,
> we would definitely propose better solution on top of it. I hope these
> discussions make sense. Thanks again for helping make the design solid!
>
>
> Boyang
>
> 
> From: Jason Gustafson 
> Sent: Thursday, November 15, 2018 9:54 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> >
> > I feel this would make the current protocol harder to use. For example,
> on
> > KStream we typically would expect (number of instances * number of
> threads
> > per instance) consumers. Giving out all this many member names in a list
> > may not be easy, compared with dynamic generation of member names, at
> least
> > for KStream use case.
>
>
> That's a fair point. What I had in mind is for the API to specify the
> number of consumers in the group. Then the ids would be determined using
> some convention. Most likely, we would just use sequential numbers 0, 1, 2,
> etc. We do the same thing for partition ids. Streams folks can chime in and
> say whether that would work or not.
>
> So this suggests we will not rely on heartbeat and commit interval to
> > trigger rebalance? Even in static membership, I feel tracking active
> > members is still required to be handled by broker since not all users are
> > fully equipped with monitoring tools, otherwise users will feel
> > uncomfortable using static membership solely for reducing rebalance
> purpose.
>
>
> Let me clarify. I think it is still useful to distinguish the liveness of
> the consumer using the heartbeat mechanism and to propagate that
> information to the leader during rebalances. This gives the group leader
> the option (but not the requirement) to change the partition assignment for
> inactive members. My suggestion was more about how long the static
> registration would remain valid. In the current proposal we have a
> registration timeout, and I think Mayuresh is fair to point out the
> potential confusion with the session timeout. It's kind of the same, but
> not exactly. So instead, I was thinking that the registration is specified
> ahead of time and remains valid until changed. It would be more like a
> replica assignment. We don't move partitions just because a broker is down.
> The expectation is that it the broker will eventually return. Similarly for
> stateful applications, we would have the option to do the same thing. The
> registration is fixed until someone changes it.
>
> Does that make sense?
>
> Thanks,
> Jason
>
>
> On Wed, Nov 14, 2018 at 2:46 PM, Boyang Chen  wrote:
>
> > Thanks Jason for the suggestions! I update the KIP with full schema
> > changes.
> >
> > > we offer an admin API that lets a user define the expected members of
> > the group.
> >
> > I feel this would make the current protocol harder to use. For example,
> on
> > KStream we typically would expect (number of instances * number of
> threads
> > per instance) consumers. Giving out all this many member names in a list
> > may not be easy, compared with dynamic generation of member names, at
> least
> > for KStream use case.
> >
> > > For the sake of discussion, I was wondering if we could just say that
> > static members do not expire.
> >
> >
> > So this suggests we will not rely on heartbeat and commit interval to
> > trigger rebalance? Even in static membership, I feel tracking active
> > members is still required to be handled by broker since not all users are
> > fully equipped with monitoring tools, otherwise users will feel
> > uncomfortable using static membership solely for reducing rebalance
> purpose.
> >
> >
> > For the admin

Re: [VOTE] 2.1.0 RC1

[Jason Gustafson]

+1

I verified the release and upgrade notes. I also went through the basic
quickstart.

Great job running the release, Dong! Thanks for all the effort.

-Jason

On Mon, Nov 19, 2018 at 10:50 AM, Dong Lin  wrote:

> Hey Ismael,
>
> I checked out a clean copy of Kafka and reuploaded artifacts for 2.1.0-rc1
> without code change. There are still those new files in
> https://repository.apache.org/content/groups/staging/org/
> apache/kafka/kafka_2.12/2.1.0.
> I compared 2.0 and 2.1 branch but did not find any suspicious change in
> release.py and build.gradle.
>
> Since doing a new release could not address this right away and there is no
> known impact on user due to these redundant files, I am inclined to still
> release 2.1.0-rc1 so that user can start to use the new features soon. What
> do you think?
>
> Thanks,
> Dong
>
>
> On Mon, Nov 19, 2018 at 2:16 AM Dong Lin  wrote:
>
> > Hey Ismael,
> >
> > Thanks much for catching this! Sorry I didn't catch this issue before.
> >
> > These files were uploaded by release.py scrip in the repo.
> kafka_2.12-2.1.
> > 0.mapping contains the following string and the other files are the
> > signature and hash of the file kafka_2.12-2.1.0.mapping:
> >
> >
> > /home/dolin/research/kafka/.release_work_dir/kafka/core/
> build/libs/kafka_2.12-2.1.0.jar
> >
> > /home/dolin/research/kafka/.release_work_dir/kafka/core/build/tmp/scala/
> compilerAnalysis/compileScala.analysis
> >
> > It is weird to have these files.. Let me generate another release
> > candidate and try to fix this issue.
> >
> > Thanks,
> > Dong
> >
>

[DISCUSS] KIP-392: Allow consumers to fetch from the closest replica

[Jason Gustafson]

Hi All,

I've posted a KIP to add the often-requested support for fetching from
followers:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica.
Please take a look and let me know what you think.

Thanks,
Jason

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

[Jason Gustafson]

Hi Stanislav,

Thanks for the KIP. Can you clarify the compatibility impact here? What
will happen to groups that are already larger than the max size? Also, just
to be clear, the resource we are trying to conserve here is what? Memory?

-Jason

On Mon, Nov 26, 2018 at 2:44 AM Boyang Chen  wrote:

> Thanks Stanislav for the update! One suggestion I have is that it would be
> helpful to put your
>
> reasoning on deciding the current default value. For example, in certain
> use cases at Pinterest we are very likely
>
> to have more consumers than 250 when we configure 8 stream instances with
> 32 threads.
>
>
> For the effectiveness of this KIP, we should encourage people to discuss
> their opinions on the default setting and ideally reach a consensus.
>
>
> Best,
>
> Boyang
>
> 
> From: Stanislav Kozlovski 
> Sent: Monday, November 26, 2018 6:14 PM
> To: dev@kafka.apache.org
> Subject: Re: [Discuss] KIP-389: Enforce group.max.size to cap member
> metadata growth
>
> Hey everybody,
>
> It's been a week since this KIP and not much discussion has been made.
> I assume that this is a straight forward change and I will open a voting
> thread in the next couple of days if nobody has anything to suggest.
>
> Best,
> Stanislav
>
> On Thu, Nov 22, 2018 at 12:56 PM Stanislav Kozlovski <
> stanis...@confluent.io>
> wrote:
>
> > Greetings everybody,
> >
> > I have enriched the KIP a bit with a bigger Motivation section and also
> > renamed it.
> > KIP:
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-389%253A%2BIntroduce%2Ba%2Bconfigurable%2Bconsumer%2Bgroup%2Bsize%2Blimit&data=02%7C01%7C%7C085ed04564f2472e50f308d65387f4fd%7C84df9e7fe9f640afb435%7C1%7C0%7C636788240721218938&sdata=C6aXV4T6JWcNPtJhVSNxPrHSm2oTP%2BtGN4XvD4jSUOU%3D&reserved=0
> >
> > I'm looking forward to discussions around it.
> >
> > Best,
> > Stanislav
> >
> > On Tue, Nov 20, 2018 at 1:47 PM Stanislav Kozlovski <
> > stanis...@confluent.io> wrote:
> >
> >> Hey there everybody,
> >>
> >> Thanks for the introduction Boyang. I appreciate the effort you are
> >> putting into improving consumer behavior in Kafka.
> >>
> >> @Matt
> >> I also believe the default value is high. In my opinion, we should aim
> to
> >> a default cap around 250. This is because in the current model any
> consumer
> >> rebalance is disrupting to every consumer. The bigger the group, the
> longer
> >> this period of disruption.
> >>
> >> If you have such a large consumer group, chances are that your
> >> client-side logic could be structured better and that you are not using
> the
> >> high number of consumers to achieve high throughput.
> >> 250 can still be considered of a high upper bound, I believe in practice
> >> users should aim to not go over 100 consumers per consumer group.
> >>
> >> In regards to the cap being global/per-broker, I think that we should
> >> consider whether we want it to be global or *per-topic*. For the time
> >> being, I believe that having it per-topic with a global default might be
> >> the best situation. Having it global only seems a bit restricting to me
> and
> >> it never hurts to support more fine-grained configurability (given it's
> the
> >> same config, not a new one being introduced).
> >>
> >> On Tue, Nov 20, 2018 at 11:32 AM Boyang Chen 
> wrote:
> >>
> >>> Thanks Matt for the suggestion! I'm still open to any suggestion to
> >>> change the default value. Meanwhile I just want to point out that this
> >>> value is a just last line of defense, not a real scenario we would
> expect.
> >>>
> >>>
> >>> In the meanwhile, I discussed with Stanislav and he would be driving
> the
> >>> 389 effort from now on. Stanislav proposed the idea in the first place
> and
> >>> had already come up a draft design, while I will keep focusing on
> KIP-345
> >>> effort to ensure solving the edge case described in the JIRA<
> >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-7610&data=02%7C01%7C%7C085ed04564f2472e50f308d65387f4fd%7C84df9e7fe9f640afb435%7C1%7C0%7C636788240721218938&sdata=PyOSGb6FhjcIS0XL2vcv2YEUSaYk9lL593ioHS4rRHk%3D&reserved=0
> >.
> >>>
> >>>
> >>> Thank you Stanislav for making this happen!
> >>>
> >>>
> >>> Boyang
> >>>
> >>> 
> >>> From: Matt Farmer 
> >>> Sent: Tuesday, November 20, 2018 10:24 AM
> >>> To: dev@kafka.apache.org
> >>> Subject: Re: [Discuss] KIP-389: Enforce group.max.size to cap member
> >>> metadata growth
> >>>
> >>> Thanks for the KIP.
> >>>
> >>> Will this cap be a global cap across the entire cluster or per broker?
> >>>
> >>> Either way the default value seems a bit high to me, but that could
> just
> >>> be
> >>> from my own usage patterns. I’d have probably started with 500 or 1k
> but
> >>> could be easily convinced that’s wrong.
> >>>
> >>> Thanks,
> >>> Matt
> >>>
> >>> On Mon, Nov 19, 2018 a

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

Hi Boyang,

Thanks for the updates. Looks like we're headed in the right direction and
clearly the interest that this KIP is receiving shows how strong the
motivation is!

I have a few questions:

1. This may be the same thing that Mayuresh is asking about. I think the
suggestion in the KIP is that if a consumer sends JoinGroup with a member
name, but no member id, then we will return the current member id
associated with that name. It seems in this case that we wouldn't be able
to protect from having two consumers active with the same configured
member.name? For example, imagine that we had a consumer with member.name=A
which is assigned member.id=1. Suppose it becomes a zombie and a new
instance starts up with member.name=A. If it is also assigned member.id=1,
then how can we detect the zombie if it comes back to life? Both instances
will have the same member.id.

The goal is to avoid a rebalance on a rolling restart, but we still need to
fence previous members. I am wondering if we can generate a new member.id
every time we receive a request from a static member with an unknown member
id. If the old instance with the same member.name attempts any operation,
then it will be fenced with an UNKNOWN_MEMBER_ID error. As long as the
subscription of the new instance hasn't changed, then we can skip the
rebalance and return the current assignment without forcing a rebalance.

The trick to making this work is in the error handling of the zombie
consumer. If the zombie simply resets its member.id and rejoins to get a
new one upon receiving the UNKNOWN_MEMBER_ID error, then it would end up
fencing the new member. We want to avoid this. There needs to be an
expectation for static members that the member.id of a static member will
not be changed except when a new member with the same member.name joins the
group. Then we can treat UNKNOWN_MEMBER_ID as a fatal error for consumers
with static member names.

2. The mechanics of the ConsumerRebalance API seem unclear to me. As far as
I understand it, it is used for scaling down a consumer group and somehow
bypasses normal session timeout expiration. I am wondering how critical
this piece is and whether we can leave it for future work. If not, then it
would be helpful to elaborate on its implementation. How would the
coordinator know which members to kick out of the group?

3. I've been holding back on mentioning this, but I think we should
reconsider the name `member.name`. I think we want something that suggests
its expectation of uniqueness in the group. How about `group.instance.id`
to go along with `group.id`?

Thanks,
Jason



On Mon, Nov 26, 2018 at 10:18 AM Mayuresh Gharat 
wrote:

> Hi Boyang,
>
> Thanks a lot for replying to all the queries and discussions here, so
> patiently.
> Really appreciate it.
>
> Had a few questions and suggestions after rereading the current version of
> the KIP :
>
>
>1. Do you intend to have member.id is a static config like member.name
>after KIP-345 and KIP-394?
>2. Regarding "On client side, we add a new config called MEMBER_NAME in
>ConsumerConfig. On consumer service init, if the MEMBER_NAME config is
> set,
>we will put it in the initial join group request to identify itself as a
>static member (static membership); otherwise, we will still send
>UNKNOWN_MEMBER_ID to ask broker for allocating a new random ID (dynamic
>membership)."
>   - What is the value of member_id sent in the first JoinGroupRequest
>   when member_name is set (using static rebalance)? Is it
> UNKNOW_MEMBER_ID?
>3. Regarding "we are requiring member.id (if not unknown) to match the
>value stored in cache, otherwise reply MEMBER_ID_MISMATCH. The edge case
>that if we could have members with the same `member.name` (for example
>mis-configured instances with a valid member.id but added a used member
>name on runtime). When member name has duplicates, we could refuse join
>request from members with an outdated `member.id` (since we update the
>mapping upon each join group request). In an edge case where the client
>hits this exception in the response, it is suggesting that some other
>consumer takes its spot."
>   - The part of "some other consumer takes the spot" would be
>   intentional, right? Also when you say " The edge case that if we
>   could have members with the same `member.name` (for example
>   mis-configured instances *with a valid member.id 
> *but
>   added a used member name on runtime).", what do you mean by *valid
>   member id* here? Does it mean that there exist a mapping of
>   member.name to member.id like *MemberA -> id1* on the
>   GroupCoordinator and this consumer is trying to join with *
> member.name
>    = MemberB and member.id  =
> id1 *
>   ?
>4. Depending on your explanation for point 2 and the point 3 above
>regarding returning back MEMBER_ID_MISMATCH o

Re: [DISCUSS] KIP-391: Allow Producing with Offsets for Cluster Replication

[Jason Gustafson]

Another wrinkle to consider is KIP-320. If you are planning to replicate
__consumer_offsets directly, then you will have to account for leader epoch
information which is stored with the committed offsets. But I cannot think
how it would be possible to replicate the leader epoch information in
messages even if you can preserve offsets.

-Jason

On Mon, Nov 26, 2018 at 1:16 PM Mayuresh Gharat 
wrote:

> Hi Edoardo,
>
> Thanks a lot for the KIP.
>  I have a few questions/suggestions in addition to what Radai has mentioned
> above :
>
>1. Is this meant only for 1:1 replication, for example one Kafka cluster
>replicating to other, instead of having multiple Kafka clusters
> mirroring
>into one Kafka cluster?
>2. Are we relying on exactly once produce in the replicator? If not, how
>are retries handled in the replicator ?
>3. What is the recommended value for inflight requests, here. Is it
>suppose to be strictly 1, if yes, it would be great to mention that in
> the
>KIP.
>4. How is unclean Leader election between source cluster and destination
>cluster handled?
>5. How are offsets resets in case of the replicator's consumer handled?
>6. It would be good to explain the workflow in the KIP, with an
>example,  regarding how this KIP will change the replication scenario
> and
>how it will benefit the consumer apps.
>
> Thanks,
>
> Mayuresh
>
> On Mon, Nov 26, 2018 at 8:08 AM radai  wrote:
>
> > a few questions:
> >
> > 1. how do you handle possible duplications caused by the "special"
> > producer timing-out/retrying? are you explicitely relying on the
> > "exactly once" sequencing?
> > 2. what about the combination of log compacted topics + replicator
> > downtime? by the time the replicator comes back up there might be
> > "holes" in the source offsets (some msgs might have been compacted
> > out)? how is that recoverable?
> > 3. similarly, what if you try and fire up replication on a non-empty
> > source topic? does the kip allow for offsets starting at some
> > arbitrary X > 0 ? or would this have to be designed from the start.
> >
> > and lastly, since this KIP seems to be designed fro active-passive
> > failover (there can be no produce traffic except the replicator)
> > wouldnt a solution based on seeking to a time offset be more generic?
> > your producers could checkpoint the last (say log append) timestamp of
> > records theyve seen, and when restoring in the remote site seek to
> > those timestamps (which will be metadata in their committed offsets) -
> > assumming replication takes > 0 time you'd need to handle some dups,
> > but every kafka consumer setup needs to know how to handle those
> > anyway.
> > On Fri, Nov 23, 2018 at 2:27 AM Edoardo Comar  wrote:
> > >
> > > Hi Stanislav
> > >
> > > > > The flag is needed to distinguish a batch with a desired base
> offset
> > > of
> > > > 0,
> > > > from a regular batch for which offsets need to be generated.
> > > > If the producer can provide offsets, why not provide a base offset of
> > 0?
> > >
> > > a regular batch (for which offsets are generated by the broker on
> write)
> > > is sent with a base offset of 0.
> > > How could you distinguish it from a batch where you *want* the first
> > > record to be written at offset 0 (i.e. be the first in the partition
> and
> > > be rejected if there are records on the log already) ?
> > > We wanted to avoid a "deep" inspection (and potentially decompression)
> of
> > > the records.
> > >
> > > For the replicator use case, a single produce request where all the
> data
> > > is to be assumed with offset,
> > > or all without offsets, seems to suffice,
> > > So we added only a toplevel flag, not a per-topic-partition one.
> > >
> > > Thanks for your interest !
> > > cheers
> > > Edo
> > > --
> > >
> > > Edoardo Comar
> > >
> > > IBM Event Streams
> > > IBM UK Ltd, Hursley Park, SO21 2JN
> > >
> > >
> > > Stanislav Kozlovski  wrote on 22/11/2018
> > 22:32:42:
> > >
> > > > From: Stanislav Kozlovski 
> > > > To: dev@kafka.apache.org
> > > > Date: 22/11/2018 22:33
> > > > Subject: Re: [DISCUSS] KIP-391: Allow Producing with Offsets for
> > > > Cluster Replication
> > > >
> > > > Hey Edo & Mickael,
> > > >
> > > > > The flag is needed to distinguish a batch with a desired base
> offset
> > > of
> > > > 0,
> > > > from a regular batch for which offsets need to be generated.
> > > > If the producer can provide offsets, why not provide a base offset of
> > 0?
> > > >
> > > > > (I am reading your post thinking about
> > > > partitions rather than topics).
> > > > Yes, I meant partitions. Sorry about that.
> > > >
> > > > Thanks for answering my questions :)
> > > >
> > > > Best,
> > > > Stanislav
> > > >
> > > > On Thu, Nov 22, 2018 at 5:28 PM Edoardo Comar 
> > wrote:
> > > >
> > > > > Hi Stanislav,
> > > > >
> > > > > you're right we envision the replicator use case to have a single
> > > producer
> > > > > with offse

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

[Jason Gustafson]

Hey Guozhang,

Thanks for the comments. Responses below:

0. The new API is used between brokers, so we govern its usage using
`inter.broker.protocol.version`. If the other broker hasn't upgraded, we
will just fallback to the old logic, which is to accept the write. This is
similar to how we introduced the OffsetsForLeaderEpoch API. Does that seem
reasonable?

To tell the truth, after digging this KIP up and reading it over, I am
doubting how crucial this API is. It is attempting to protect a write from
a zombie which has just reset its sequence number after that producer had
had its state cleaned up. However, one of the other improvements in this
KIP is to maintain producer state beyond its retention in the log. I think
that makes this case sufficiently unlikely that we can leave it for future
work. I am not 100% sure this is the only scenario where transaction state
and log state can diverge anyway, so it would be better to consider this
problem more generally. What do you think?

1. Thanks, from memory, the term changed after the first iteration. I'll
make a pass and try to clarify usage.
2. I was attempting to handle some edge cases since this check would be
asynchronous. In any case, if we drop this validation as suggested above,
then we can ignore this.

-Jason



On Tue, Nov 13, 2018 at 6:23 PM Guozhang Wang  wrote:

> Hello Jason, thanks for the great write-up.
>
> 0. One question about the migration plan: "The new GetTransactionState API
> and the new version of the transaction state message will not be used until
> the inter-broker version supports it." I'm not so clear about the
> implementation details here: say a broker is on the newer version and the
> txn-coordinator is still on older version. Today the APIVersionsRequest can
> only help upgrade / downgrade the request version, but not forbidding
> sending any. Are you suggesting we add additional logic on the broker side
> to handle the case of "not sending the request"? If yes my concern is that
> this will be some tech-debt code that will live long before being removed.
>
> Some additional minor comments:
>
> 1. "last epoch" and "instance epoch" seem to be referring to the same thing
> in your wiki.
> 2. "The broker must verify after receiving the response that the producer
> state is still unknown.": not sure why we have to validate? If the metadata
> returned from the txn-coordinator can always be considered the
> source-of-truth, can't we just bindly use it to update the cache?
>
>
> Guozhang
>
>
>
> On Thu, Sep 6, 2018 at 9:10 PM Matthias J. Sax 
> wrote:
>
> > I am +1 on this :)
> >
> >
> > -Matthias
> >
> > On 9/4/18 9:55 AM, Jason Gustafson wrote:
> > > Bump. Thanks to Magnus for noticing that I forgot to link to the KIP:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-360%3A+Improve+handling+of+unknown+producer
> > > .
> > >
> > > -Jason
> > >
> > > On Tue, Aug 21, 2018 at 4:37 PM, Jason Gustafson 
> > wrote:
> > >
> > >> Hi All,
> > >>
> > >> I have a proposal to improve the transactional/idempotent producer's
> > >> handling of the UNKNOWN_PRODUCER error, which is the result of losing
> > >> producer state following segment removal. The current behavior is both
> > >> complex and limited. Please take a look and let me know what you
> think.
> > >>
> > >> Thanks in advance to Matthias Sax for feedback on the initial draft.
> > >>
> > >> -Jason
> > >>
> > >
> >
> >
>
> --
> -- Guozhang
>

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

[Jason Gustafson]

Hey Stanislav,

Clients will then find that coordinator
> and send `joinGroup` on it, effectively rebuilding the group, since the
> cache of active consumers is not stored outside the Coordinator's memory.
> (please do say if that is incorrect)


Groups do not typically rebalance after a coordinator change. You could
potentially force a rebalance if the group is too big and kick out the
slowest members or something. A more graceful solution is probably to just
accept the current size and prevent it from getting bigger. We could log a
warning potentially.

My thinking is that we should abstract away from conserving resources and
> focus on giving control to the broker. The issue that spawned this KIP was
> a memory problem but I feel this change is useful in a more general way.


So you probably already know why I'm asking about this. For consumer groups
anyway, resource usage would typically be proportional to the number of
partitions that a group is reading from and not the number of members. For
example, consider the memory use in the offsets cache. The benefit of this
KIP is probably limited to preventing "runaway" consumer groups due to
leaks or some other application bug. That still seems useful though.

I completely agree with this and I *ask everybody to chime in with opinions
> on a sensible default value*.


I think we would have to be very conservative. The group protocol is
generic in some sense, so there may be use cases we don't know of where
larger groups are reasonable. Probably we should make this an opt-in
feature so that we do not risk breaking anyone's application after an
upgrade. Either that, or use a very high default like 5,000.

Thanks,
Jason

On Tue, Nov 27, 2018 at 3:27 AM Stanislav Kozlovski 
wrote:

> Hey Jason and Boyang, those were important comments
>
> > One suggestion I have is that it would be helpful to put your reasoning
> on deciding the current default value. For example, in certain use cases at
> Pinterest we are very likely to have more consumers than 250 when we
> configure 8 stream instances with 32 threads.
> > For the effectiveness of this KIP, we should encourage people to discuss
> their opinions on the default setting and ideally reach a consensus.
>
> I completely agree with this and I *ask everybody to chime in with opinions
> on a sensible default value*.
> My thought process was that in the current model rebalances in large groups
> are more costly. I imagine most use cases in most Kafka users do not
> require more than 250 consumers.
> Boyang, you say that you are "likely to have... when we..." - do you have
> systems running with so many consumers in a group or are you planning to? I
> guess what I'm asking is whether this has been tested in production with
> the current rebalance model (ignoring KIP-345)
>
> >  Can you clarify the compatibility impact here? What
> > will happen to groups that are already larger than the max size?
> This is a very important question.
> From my current understanding, when a coordinator broker gets shut
> down during a cluster rolling upgrade, a replica will take leadership of
> the `__offset_commits` partition. Clients will then find that coordinator
> and send `joinGroup` on it, effectively rebuilding the group, since the
> cache of active consumers is not stored outside the Coordinator's memory.
> (please do say if that is incorrect)
> Then, I believe that working as if this is a new group is a reasonable
> approach. Namely, fail joinGroups when the max.size is exceeded.
> What do you guys think about this? (I'll update the KIP after we settle on
> a solution)
>
> >  Also, just to be clear, the resource we are trying to conserve here is
> what? Memory?
> My thinking is that we should abstract away from conserving resources and
> focus on giving control to the broker. The issue that spawned this KIP was
> a memory problem but I feel this change is useful in a more general way. It
> limits the control clients have on the cluster and helps Kafka become a
> more self-serving system. Admin/Ops teams can better control the impact
> application developers can have on a Kafka cluster with this change
>
> Best,
> Stanislav
>
>
> On Mon, Nov 26, 2018 at 8:00 PM Jason Gustafson 
> wrote:
>
> > Hi Stanislav,
> >
> > Thanks for the KIP. Can you clarify the compatibility impact here? What
> > will happen to groups that are already larger than the max size? Also,
> just
> > to be clear, the resource we are trying to conserve here is what? Memory?
> >
> > -Jason
> >
> > On Mon, Nov 26, 2018 at 2:44 AM Boyang Chen  wrote:
> >
> > > Thanks Stanislav for the update! One suggestion I have is that it would
> > be
> > > h

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

[Jason Gustafson]

Hi Boyang,

Thanks for the KIP. Looks good overall. I think we will need to bump the
version of the JoinGroup protocol in order to indicate compatibility with
the new behavior. The coordinator needs to know when it is safe to assume
the client will handle the error code.

Also, I was wondering if we could reuse the REBALANCE_IN_PROGRESS error
code. When the client sees this error code, it will take the memberId from
the response and rejoin. We'd still need the protocol bump since older
consumers do not have this logic.

Thanks,
Jason

On Mon, Nov 26, 2018 at 5:47 PM Boyang Chen  wrote:

> Hey friends,
>
>
> I would like to start a discussion thread for KIP-394 which is trying to
> mitigate broker cache bursting issue due to anonymous join group requests:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-394%3A+Require+member.id+for+initial+join+group+request
>
>
> Thanks!
>
> Boyang
>

Re: [DISCUSS] KIP-379: Multiple Consumer Group Management

[Jason Gustafson]

Hi Alex,

Sorry for the late reply. Your message didn't get attached to the main
thread and I missed it.

#1. Yeah, I think we should continue to accept the old csv format.
Compatibility is important for all public APIs.
#2. I think this is less important from a compatibility perspective. On the
one hand, it makes the output compatible with currently supported usage. On
the other, it makes it more annoying to write tools which invoke this
command because they need to treat the single group case separately. I'm
probably leaning toward not doing this one, but I don't have a strong
opinion.
#3. To clarify, my suggestion was to put the group id first. I think Vahid
was in agreement. From your comment, it sounds like you agree as well?
#4. I agree supporting regex can be left for future work.

Thanks,
Jason

On Mon, Nov 5, 2018 at 7:55 AM Alex D  wrote:

> Hello guys,
>
> Thank you for your suggestions!
> I've made a short resume of all suggestions proposed for further
> possible code corrections.
> Since not all opinions match, let's review once again and decide.
>
> #1. Support old csv format. Proposed by Jason.
> Yes: Jason, Vahid
>
> If backwards compatibility is important for this specific (and, I
> believe, infrequent) case, ready to make corrections. Final word?
>
> #2. Do not show group name for `--describe` output in case a single
> `--group` is specified. Proposed by Jason.
> Yes: Jason
>
> Alternatively, the user will always expect the output to be the
> same
> for any `--describe` query. Ready to make corrections if this is
> important. Final word?
>
> #3. GROUP column should not be the first in the row. Proposed by Jason.
> Yes: Jason
> No:  Vahid
>
> For the group offset configuration, the group entity appears to be
> the top priority and starting a table with a GROUP column makes more
> sense, I believe. Plus, it's quicker and easier to spot to which group
> the offsets belong to.
> Apply corrections or leave as is?
>
> #4. Single regex vs multiple `--group` flags. Proposed by eazama..
>
> There are a few reasons behind this. Firstly, there are no rules
> for
> defining group names unlike for topic names that have their own
> validation routine according to a simple regex. Group names may
> contain any symbols possible and simply splitting them by comma won't
> work, at least without using escape characters maybe. Secondly,
> repetition of the `--group` flag had already been implemented for the
> group deletion logic and we don't not want to break the backwards
> compatibility. Finally, visually, it's a bit easier to read and
> compose a long query with a large number of groups than throwing
> everything into one very long string.
>
> #5. Valid scenario where we would want to delete all consumer groups.
> Asked by Vahid.
>
> There should be one, I believe ;) Already received a few requests
> from colleagues.
>
> # KIP approvals:
> Suman: +1
>
> > Sat, 20 Oct 2018 17:10:16 GMT,  wrote:
> > Is there a reason for using multiple --group flags over having it accept
> a regex?
> >
> > The topics command currently accepts a regex for most operations and
> doesn't support using
> > multiple topics flags. It seems like it would be better to take a more
> standardized approach
> > to providing this type of information.
> >
> >
> >> On Oct 19, 2018, at 10:28 AM, Suman B N  wrote:
> >>
> >> This eases debugging metadata information of consumer groups and
> offsets in
> >> case of client hungs which we have been facing frequently.
> >> +1 from me. Well done Alex!
> >>
> >> -Suman
> >>
> >> On Fri, Oct 19, 2018 at 8:36 PM Vahid Hashemian <
> vahid.hashem...@gmail.com>
> >> wrote:
> >>
> >>> Thanks for proposing the KIP. Looks good to me overall.
> >>>
> >>> I agree with Jason's suggestion that it would be best to keep the
> current
> >>> output format when a single '--group' is present. Because otherwise,
> there
> >>> would be an impact to users who rely on the current output format.
> Also,
> >>> starting with a GROUP column makes more sense to me.
> >>>
> >>> Also, and for my own info, is there a valid scenario where we would
> want to
> >>> delete all consumer groups? It sounds to me like a potentially
> dangerous
> >>> feature. I would imagine that it would help more with dev/test
> >>> environments, where we normally have a few groups (for which the
> repeating
> >>

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

[Jason Gustafson]

Hey Guozhang,

To clarify, the broker does not actually use the ApiVersion API for
inter-broker communications. The use of an API and its corresponding
version is controlled by `inter.broker.protocol.version`.

Nevertheless, it sounds like we're on the same page about removing
DescribeTransactionState. The impact of a dangling transaction is a little
worse than what you describe though. Consumers with the read_committed
isolation level will be stuck. Still, I think we agree that this case
should be rare and we can reconsider for future work. Rather than
preventing dangling transactions, perhaps we should consider options which
allows us to detect them and recover. Anyway, this needs more thought. I
will update the KIP.

Best,
Jason

On Tue, Nov 27, 2018 at 6:51 PM Guozhang Wang  wrote:

> 0. My original question is about the implementation details primarily,
> since current the handling logic of the APIVersionResponse is simply "use
> the highest supported version of the corresponding request", but if the
> returned response from APIVersionRequest says "I don't even know about the
> DescribeTransactionStateRequest at all", then we need additional logic for
> the falling back logic. Currently this logic is embedded in NetworkClient
> which is shared by all clients, so I'd like to avoid making this logic more
> complicated.
>
> As for the general issue that a broker does not recognize a producer with
> sequence number 0, here's my thinking: as you mentioned in the wiki, this
> is only a concern for transactional producer since for idempotent producer
> it can just bump the epoch and go. For transactional producer, even if the
> producer request from a fenced producer gets accepted, its transaction will
> never be committed and hence messages not exposed to read-committed
> consumers as well. The drawback is though, 1) read-uncommitted consumers
> will still read those messages, 2) unnecessary storage for those fenced
> produce messages, but in practice should not accumulate to a large amount
> since producer should soon try to commit and be told it is fenced and then
> stop, 3) there will be no markers for those transactional messages ever.
> Looking at the list and thinking about the likelihood it may happen
> assuming we retain the producer up to transactional.id.timeout (default is
> 7 days), I feel comfortable leaving it as is.
>
> Guozhang
>
> On Mon, Nov 26, 2018 at 6:09 PM Jason Gustafson 
> wrote:
>
> > Hey Guozhang,
> >
> > Thanks for the comments. Responses below:
> >
> > 0. The new API is used between brokers, so we govern its usage using
> > `inter.broker.protocol.version`. If the other broker hasn't upgraded, we
> > will just fallback to the old logic, which is to accept the write. This
> is
> > similar to how we introduced the OffsetsForLeaderEpoch API. Does that
> seem
> > reasonable?
> >
> > To tell the truth, after digging this KIP up and reading it over, I am
> > doubting how crucial this API is. It is attempting to protect a write
> from
> > a zombie which has just reset its sequence number after that producer had
> > had its state cleaned up. However, one of the other improvements in this
> > KIP is to maintain producer state beyond its retention in the log. I
> think
> > that makes this case sufficiently unlikely that we can leave it for
> future
> > work. I am not 100% sure this is the only scenario where transaction
> state
> > and log state can diverge anyway, so it would be better to consider this
> > problem more generally. What do you think?
> >
> > 1. Thanks, from memory, the term changed after the first iteration. I'll
> > make a pass and try to clarify usage.
> > 2. I was attempting to handle some edge cases since this check would be
> > asynchronous. In any case, if we drop this validation as suggested above,
> > then we can ignore this.
> >
> > -Jason
> >
> >
> >
> > On Tue, Nov 13, 2018 at 6:23 PM Guozhang Wang 
> wrote:
> >
> > > Hello Jason, thanks for the great write-up.
> > >
> > > 0. One question about the migration plan: "The new GetTransactionState
> > API
> > > and the new version of the transaction state message will not be used
> > until
> > > the inter-broker version supports it." I'm not so clear about the
> > > implementation details here: say a broker is on the newer version and
> the
> > > txn-coordinator is still on older version. Today the APIVersionsRequest
> > can
> > > only help upgrade / downgrade the request version, but not forbidding
> > > sending any. Are you suggesting we add additiona

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

[Jason Gustafson]

Hey Stanislav,

What do you think about the use case I mentioned in my previous reply about
> a more resilient self-service Kafka? I believe the benefit there is bigger.


I see this config as analogous to the open file limit. Probably this limit
was intended to be prescriptive at some point about what was deemed a
reasonable number of open files for an application. But mostly people treat
it as an annoyance which they have to work around. If it happens to be hit,
usually you just increase it because it is not tied to an actual resource
constraint. However, occasionally hitting the limit does indicate an
application bug such as a leak, so I wouldn't say it is useless. Similarly,
the issue in KAFKA-7610 was a consumer leak and having this limit would
have allowed the problem to be detected before it impacted the cluster. To
me, that's the main benefit. It's possible that it could be used
prescriptively to prevent poor usage of groups, but like the open file
limit, I suspect administrators will just set it large enough that users
are unlikely to complain.

Anyway, just a couple additional questions:

1. I think it would be helpful to clarify the details on how the
coordinator will shrink the group. It will need to choose which members to
remove. Are we going to give current members an opportunity to commit
offsets before kicking them from the group?

2. Do you think we should make this a dynamic config?

Thanks,
Jason




On Wed, Nov 28, 2018 at 2:42 AM Stanislav Kozlovski 
wrote:

> Hi Jason,
>
> You raise some very valid points.
>
> > The benefit of this KIP is probably limited to preventing "runaway"
> consumer groups due to leaks or some other application bug
> What do you think about the use case I mentioned in my previous reply about
> a more resilient self-service Kafka? I believe the benefit there is bigger
>
> * Default value
> You're right, we probably do need to be conservative. Big consumer groups
> are considered an anti-pattern and my goal was to also hint at this through
> the config's default. Regardless, it is better to not have the potential to
> break applications with an upgrade.
> Choosing between the default of something big like 5000 or an opt-in
> option, I think we should go with the *disabled default option*  (-1).
> The only benefit we would get from a big default of 5000 is default
> protection against buggy/malicious applications that hit the KAFKA-7610
> issue.
> While this KIP was spawned from that issue, I believe its value is enabling
> the possibility of protection and helping move towards a more self-service
> Kafka. I also think that a default value of 5000 might be misleading to
> users and lead them to think that big consumer groups (> 250) are a good
> thing.
>
> The good news is that KAFKA-7610 should be fully resolved and the rebalance
> protocol should, in general, be more solid after the planned improvements
> in KIP-345 and KIP-394.
>
> * Handling bigger groups during upgrade
> I now see that we store the state of consumer groups in the log and why a
> rebalance isn't expected during a rolling upgrade.
> Since we're going with the default value of the max.size being disabled, I
> believe we can afford to be more strict here.
> During state reloading of a new Coordinator with a defined max.group.size
> config, I believe we should *force* rebalances for groups that exceed the
> configured size. Then, only some consumers will be able to join and the max
> size invariant will be satisfied.
>
> I updated the KIP with a migration plan, rejected alternatives and the new
> default value.
>
> Thanks,
> Stanislav
>
> On Tue, Nov 27, 2018 at 5:25 PM Jason Gustafson 
> wrote:
>
> > Hey Stanislav,
> >
> > Clients will then find that coordinator
> > > and send `joinGroup` on it, effectively rebuilding the group, since the
> > > cache of active consumers is not stored outside the Coordinator's
> memory.
> > > (please do say if that is incorrect)
> >
> >
> > Groups do not typically rebalance after a coordinator change. You could
> > potentially force a rebalance if the group is too big and kick out the
> > slowest members or something. A more graceful solution is probably to
> just
> > accept the current size and prevent it from getting bigger. We could log
> a
> > warning potentially.
> >
> > My thinking is that we should abstract away from conserving resources and
> > > focus on giving control to the broker. The issue that spawned this KIP
> > was
> > > a memory problem but I feel this change is useful in a more general
> way.
> >
> >
> > So you probably already know why I'm asking about this. For consumer
> groups

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

[Jason Gustafson]

he first to the
> >> second joinGroup request (or broker joinGroup response).
> >>
> >> But maybe I am missing something. Can you help me out?
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 11/27/18 6:00 PM, Boyang Chen wrote:
> >>> Thanks Stanislav and Jason for the suggestions!
> >>>
> >>>
> >>>> Thanks for the KIP. Looks good overall. I think we will need to bump
> the
> >>>> version of the JoinGroup protocol in order to indicate compatibility
> >> with
> >>>> the new behavior. The coordinator needs to know when it is safe to
> >> assume
> >>>> the client will handle the error code.
> >>>>
> >>>> Also, I was wondering if we could reuse the REBALANCE_IN_PROGRESS
> error
> >>>> code. When the client sees this error code, it will take the memberId
> >> from
> >>>> the response and rejoin. We'd still need the protocol bump since older
> >>>> consumers do not have this logic.
> >>>
> >>> I will add the join group protocol version change to the KIP. Meanwhile
> >> I feel for
> >>> understandability it's better to define a separate error code since
> >> REBALANCE_IN_PROGRESS
> >>> is not the actual cause of the returned error.
> >>>
> >>>> One small question I have is now that we have one and a half
> round-trips
> >>>> needed to join in a rebalance (1 full RT addition), is it worth it to
> >>>> consider increasing the default value of `
> >> group.initial.rebalance.delay.ms`?
> >>> I guess we could keep it for now. After KIP-345 and incremental
> >> cooperative rebalancing
> >>> work we should be safe to deprecate `group.initial.rebalance.delay.ms
> `.
> >> Also one round trip
> >>> shouldn't increase the latency too much IMO.
> >>>
> >>> Best,
> >>> Boyang
> >>> 
> >>> From: Stanislav Kozlovski 
> >>> Sent: Wednesday, November 28, 2018 2:32 AM
> >>> To: dev@kafka.apache.org
> >>> Subject: Re: [DISCUSS] KIP-394: Require member.id for initial join
> >> group request
> >>>
> >>> Hi Boyang,
> >>>
> >>> The KIP looks very good.
> >>> One small question I have is now that we have one and a half
> round-trips
> >>> needed to join in a rebalance (1 full RT addition), is it worth it to
> >>> consider increasing the default value of `
> >> group.initial.rebalance.delay.ms`?
> >>>
> >>> Best,
> >>> Stanislav
> >>>
> >>> On Tue, Nov 27, 2018 at 5:39 PM Jason Gustafson 
> >> wrote:
> >>>
> >>>> Hi Boyang,
> >>>>
> >>>> Thanks for the KIP. Looks good overall. I think we will need to bump
> the
> >>>> version of the JoinGroup protocol in order to indicate compatibility
> >> with
> >>>> the new behavior. The coordinator needs to know when it is safe to
> >> assume
> >>>> the client will handle the error code.
> >>>>
> >>>> Also, I was wondering if we could reuse the REBALANCE_IN_PROGRESS
> error
> >>>> code. When the client sees this error code, it will take the memberId
> >> from
> >>>> the response and rejoin. We'd still need the protocol bump since older
> >>>> consumers do not have this logic.
> >>>>
> >>>> Thanks,
> >>>> Jason
> >>>>
> >>>> On Mon, Nov 26, 2018 at 5:47 PM Boyang Chen 
> >> wrote:
> >>>>
> >>>>> Hey friends,
> >>>>>
> >>>>>
> >>>>> I would like to start a discussion thread for KIP-394 which is trying
> >> to
> >>>>> mitigate broker cache bursting issue due to anonymous join group
> >>>> requests:
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-394%253A%2BRequire%2Bmember.id%2Bfor%2Binitial%2Bjoin%2Bgroup%2Brequest&data=02%7C01%7C%7C046b858d134f4c1c576108d655277552%7C84df9e7fe9f640afb435%7C1%7C0%7C636790025286277394&sdata=QyBzqnislL%2B9fK1mXaRuJ0xpi9Y2JDvHrM881hzjq3A%3D&reserved=0
> >>>>>
> >>>>>
> >>>>> Thanks!
> >>>>>
> >>>>> Boyang
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Best,
> >>> Stanislav
> >>>
> >>
> >>
> >
> > --
> > Best,
> > Stanislav
> >
>
>

Re: [VOTE] KIP-351: Add --under-min-isr option to describe TopicCommand

[Jason Gustafson]

+1 Thanks for the KIP!

-Jason

On Wed, Nov 28, 2018 at 5:26 PM Kevin Lu  wrote:

> @Manikumar I have updated the Compatibility section to note that this
> option is not supported with the deprecated "--zookeeper" option.
>
> We still need 1 more binding vote for this to be accepted.
>
> Thanks~
>
> Regards,
> Kevin
>
> On Wed, Nov 28, 2018 at 9:23 AM Gwen Shapira  wrote:
>
> > +1 (binding)
> > On Thu, Nov 8, 2018 at 1:37 PM Kevin Lu  wrote:
> > >
> > > Hi All,
> > >
> > > I'm starting the vote thread for KIP-351: Add --under-min-isr option to
> > > describe topics command.
> > >
> > > KIP:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-351%3A+Add+--under-min-isr+option+to+describe+topics+command
> > >
> > > Discussion thread:
> > >
> >
> https://lists.apache.org/thread.html/90d1652ebc03a7be4100dd101b92a7dcefe63d144856c5f6c132381b@%3Cdev.kafka.apache.org%3E
> > >
> > > Thanks!
> > >
> > > Regards,
> > > Kevin
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>

Re: [VOTE] KIP-394: Require member.id for initial join group request

[Jason Gustafson]

+1 Thanks for the KIP, Boyang!

-Jason

On Mon, Dec 10, 2018 at 10:07 AM Boyang Chen  wrote:

> Thanks for voting my friends. Could someone give one more binding vote
> here?
>
> Best,
> Boyang
> 
> From: Bill Bejeck 
> Sent: Thursday, December 6, 2018 2:45 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-394: Require member.id for initial join group
> request
>
> +1
> Thanks for the KIP.
>
> -Bill
>
> On Wed, Dec 5, 2018 at 1:43 PM Matthias J. Sax 
> wrote:
>
> > Thanks for the KIP.
> >
> > +1 (binding)
> >
> > -Matthias
> >
> >
> > On 12/5/18 7:53 AM, Mayuresh Gharat wrote:
> > > +1 (non-binding)
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > >
> > > On Wed, Dec 5, 2018 at 3:59 AM Boyang Chen 
> wrote:
> > >
> > >> Hey friends,
> > >>
> > >> I would like to start a vote for KIP-394<
> > >>
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-394%253A%2BRequire%2Bmember.id%2Bfor%2Binitial%2Bjoin%2Bgroup%2Brequest&data=02%7C01%7C%7C78066780826b40ed5e7d08d65ae1eda1%7C84df9e7fe9f640afb435%7C1%7C0%7C636796323725452670&sdata=Gp%2FVlUuezVVck81fMXH7yaQ7zKd0WaJ9Kc7GhtJW2Qo%3D&reserved=0
> > >.
> > >> The goal of this KIP is to improve broker stability by fencing invalid
> > join
> > >> group requests.
> > >>
> > >> Best,
> > >> Boyang
> > >>
> > >>
> > >
> >
> >
>

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

[Jason Gustafson]

gt; > consumer group until
> > > it transits to PREPARE_REBALANCE. And we keep track of how many join
> > group
> > > requests
> > > we have seen so far during PREPARE_REBALANCE. After reaching the
> consumer
> > > cap,
> > > we start to inform over provisioned consumers that you should send
> > > LeaveGroupRequest and
> > > fail yourself. Or with what Mayuresh proposed in KIP-345, we could mark
> > > extra members
> > > as hot backup and rebalance without them.
> > >
> > > 2.Choose the `consumer.group.max.size`. I feel incremental rebalancing
> > > (you proposed) could be of help here.
> > > When a new cap is enforced, leader should be notified. If the current
> > > group size is already over limit, leader
> > > shall trigger a trivial rebalance to shuffle some topic partitions and
> > let
> > > a subset of consumers prepare the ownership
> > > transition. Until they are ready, we trigger a real rebalance to remove
> > > over-provisioned consumers. It is pretty much
> > > equivalent to `how do we scale down the consumer group without
> > > interrupting the current processing`.
> > >
> > > I personally feel inclined to 2 because we could kill two birds with
> one
> > > stone in a generic way. What do you think?
> > >
> > > Boyang
> > > 
> > > From: Stanislav Kozlovski 
> > > Sent: Monday, December 3, 2018 8:35 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [Discuss] KIP-389: Enforce group.max.size to cap member
> > > metadata growth
> > >
> > > Hi Jason,
> > >
> > > > 2. Do you think we should make this a dynamic config?
> > > I'm not sure. Looking at the config from the perspective of a
> > prescriptive
> > > config, we may get away with not updating it dynamically.
> > > But in my opinion, it always makes sense to have a config be
> dynamically
> > > configurable. As long as we limit it to being a cluster-wide config, we
> > > should be fine.
> > >
> > > > 1. I think it would be helpful to clarify the details on how the
> > > coordinator will shrink the group. It will need to choose which members
> > to
> > > remove. Are we going to give current members an opportunity to commit
> > > offsets before kicking them from the group?
> > >
> > > This turns out to be somewhat tricky. I think that we may not be able
> to
> > > guarantee that consumers don't process a message twice.
> > > My initial approach was to do as much as we could to let consumers
> commit
> > > offsets.
> > >
> > > I was thinking that we mark a group to be shrunk, we could keep a map
> of
> > > consumer_id->boolean indicating whether they have committed offsets. I
> > then
> > > thought we could delay the rebalance until every consumer commits (or
> > some
> > > time passes).
> > > In the meantime, we would block all incoming fetch calls (by either
> > > returning empty records or a retriable error) and we would continue to
> > > accept offset commits (even twice for a single consumer)
> > >
> > > I see two problems with this approach:
> > > * We have async offset commits, which implies that we can receive fetch
> > > requests before the offset commit req has been handled. i.e consmer
> sends
> > > fetchReq A, offsetCommit B, fetchReq C - we may receive A,C,B in the
> > > broker. Meaning we could have saved the offsets for B but rebalance
> > before
> > > the offsetCommit for the offsets processed in C come in.
> > > * KIP-392 Allow consumers to fetch from closest replica
> > > <
> > >
> >
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-392%253A%2BAllow%2Bconsumers%2Bto%2Bfetch%2Bfrom%2Bclosest%2Breplica&data=02%7C01%7C%7C9b6fddd2f2be41ce39c308d65c42c821%7C84df9e7fe9f640afb435%7C1%7C0%7C636797839221710310&sdata=a2G0cWk7Saia9OEz4UfxvKBQtzP25Zi5cCb5jWx9mZY%3D&reserved=0
> > > >
> > > would
> > > make it significantly harder to block poll() calls on consumers whose
> > > groups are being shrunk. Even if we implemented a solution, the same
> race
> > > condition noted above seems to apply and probably others
> > >
> > >
> > > Given those constraints, I think that we can simply mark the group as
> > > `PreparingRebalance` with a reb

Re: [DISCUSS] KIP-392: Allow consumers to fetch from the closest replica

[Jason Gustafson]

 from a consumer,
> > > > should it return a new type of error like ReplicaNotInSync?
> > > >
> > > > 3. Could ReplicaSelector be closable?
> > > >
> > > > 4. Currently, the ISR propagation from the leader to the controller
> > can be
> > > > delayed up to 60 secs through
> > ReplicaManager.IsrChangePropagationInterval.
> > > > In that window, the consumer could still be consuming from a non
> > in-sync
> > > > replica. The relatively large delay is mostly for reducing the ZK
> > writes
> > > > and the watcher overhead. Not sure what's the best way to address
> > this. We
> > > > could potentially make this configurable.
> > > >
> > > > 5. It may be worth mentioning that, to take advantage of affinity,
> one
> > may
> > > > also want to have a customized PartitionAssignor to have an affinity
> > aware
> > > > assignment in addition to a customized ReplicaSelector.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Nov 21, 2018 at 12:54 PM Jason Gustafson  >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I've posted a KIP to add the often-requested support for fetching
> > from
> > > > > followers:
> > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica
> > > > > .
> > > > > Please take a look and let me know what you think.
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Stanislav
> >
>

Re: [DISCUSS] KIP-392: Allow consumers to fetch from the closest replica

[Jason Gustafson]

Hey Mickael,

Thanks for the comments. Responses below:

- I'm guessing the selector will be invoke after each rebalance so
> every time the consumer is assigned a partition it will be able to
> select it. Is that true?


I'm not sure it is necessary to do it after every rebalance, but certainly
the selector would be invoked after the partition is first assigned. Was
there a specific concern you had in mind?

- From the selector API, I'm not sure how the consumer will be able to
> address some of the choices mentioned in "Finding the preferred
> follower". Especially the available bandwidth and the load balancing.
> By only having the list of Nodes, a consumer can pick the nereast
> replica (assuming the rack field means anything to users) or balance
> its own bandwidth but that might not necessarily mean improved
> performance or a balanced load on the brokers.


The intent is to provide a minimal extension point. Users would have to
rely on external sources for their own custom selection logic. It is
similar to other interfaces exposed in the clients, such as Partitioner and
PartitionAssignor. The interface exposes only metadata about the
replication state, but nothing stops users from leveraging other
information to make better decisions. Does that seem reasonable?

Thanks,
Jason



On Mon, Dec 10, 2018 at 11:41 AM Jason Gustafson  wrote:

> Hey Eno,
>
> Thanks for the comments. However, I'm a bit confused. I'm not suggesting
> we change Produce semantics in any way. All writes still go through the
> partition leader and nothing changes with respect to committing to the ISR.
> The main issue, as I've mentioned in the KIP, is the increased latency
> before a committed offset is exposed on followers.
>
> Perhaps I have misunderstood your question?
>
> Thanks,
> Jason
>
> On Mon, Dec 3, 2018 at 9:18 AM Eno Thereska 
> wrote:
>
>> Hi Jason,
>>
>> This is an interesting KIP. This will have massive implications for
>> consistency and serialization, since currently the leader for a partition
>> serializes requests. A few questions for now:
>>
>> - before we deal with the complexity, it'd be great to see a crisp example
>> in the motivation as to when this will have the most benefit for a
>> customer. In particular, although the customer might have a multi-DC
>> deployment, the DCs could still be close by in a region, so what is the
>> expected best-case scenario for a performance gain? E.g., if all DCs are
>> on
>> the east-cost, say. Right now it's not clear to me.
>> - perhaps performance is not the right metric. Is the metric you are
>> optimizing for latency, throughput or cross-DC cost? (I believe it is
>> cross-DC cost from the KIP). Just wanted to double-check since I'm not
>> sure
>> latency would improve. Throughput could really improve from parallelism
>> (especially in cases when there is mostly consuming going on). So it could
>> be throughput as well.
>> - the proposal would probably lead to choosing a more complex consistency.
>> I tend to like the description Doug Terry has in his paper "Replicated
>> Data
>> Consistency Explained Through Baseball"
>>
>> https://www.microsoft.com/en-us/research/wp-content/uploads/2011/10/ConsistencyAndBaseballReport.pdf
>> .
>> To start with, could we get in scenarios where a client that has both a
>> producer and a consumer (e.g., Kafka streams) produces a record, then
>> attempts to consume it back and the consume() comes back with "record does
>> not exist"? That's fine, but could complicate application handling of such
>> scenarios.
>>
>> Thanks,
>> Eno
>>
>> On Mon, Dec 3, 2018 at 12:24 PM Mickael Maison 
>> wrote:
>>
>> > Hi Jason,
>> >
>> > Very cool KIP!
>> > A couple of questions:
>> > - I'm guessing the selector will be invoke after each rebalance so
>> > every time the consumer is assigned a partition it will be able to
>> > select it. Is that true?
>> >
>> > - From the selector API, I'm not sure how the consumer will be able to
>> > address some of the choices mentioned in "Finding the preferred
>> > follower". Especially the available bandwidth and the load balancing.
>> > By only having the list of Nodes, a consumer can pick the nereast
>> > replica (assuming the rack field means anything to users) or balance
>> > its own bandwidth but that might not necessarily mean improved
>> > performance or a balanced load on the brokers.
>> >
>> > Thanks
>> > On Mon, Dec 3, 2018 at 11:35 AM Stanisl

Re: [DISCUSS] KIP-392: Allow consumers to fetch from the closest replica

[Jason Gustafson]

Hi Eno,

Thanks for the clarification. From a high level, the main thing to keep in
mind is that this is an opt-in feature. It is a bit like using acks=1 in
the sense that a user is accepting slightly weaker guarantees in order to
optimize for some metric (in this case, read locality). The default
behavior would read only from the leader and users will get the usual
semantics. That said, let me address the scenarios you raised:

- scenario 1: an application that both produces and consumes (e.g., like
> Kafka streams) produces synchronously a single record to a topic and then
> attempts to consume that record. Topic is 3-way replicated say. Could it be
> the case that the produce succeeds but the consume fails? The consume could
> go to a replica that has not yet fetched the produce record, right? Or is
> that not possible?


I think it depends on what you mean by "fails." From a replica in the ISR's
perspective, it has all of the committed data. The only question is what is
safe to expose since the high watermark is always one round trip behind.
The proposal is to return a retriable error in this case so that the
consumer can distinguish the case from an out of range error and retry. No
error will be returned to the user, but consumption will be delayed. One of
the main improvements in the KIP is ensuring that this delay is minimal in
the common case.

Note that even without follower fetching, this scenario is unavoidable.
When a replica becomes a leader, it doesn't know what the latest high
watermark is until it receives fetches from all followers in the ISR.
During this window, committed data is temporarily not visible. We handle
this similarly to what is proposed here. Basically we ask the consumer to
retry until we know the data is safe.

- scenario 2: an application C that only consumes. Again say there is only
> one record produced (by another application P) to a replicated topic and
> that record has not propagated to all replicas yet (it is only at the
> leader at time t0). Application C attempts to consume at time t1 and it
> does so successfully because the consume fetches from the leader. At time
> t2 the same application seeks to the beginning of the topic and attempts to
> consume again. Is there a scenario where this second attempt fails because
> the fetching happens from a replica that does not have the record yet? At
> time t3 all replicas have the record.


Yes, this is possible in the way that I described above. There is a
(typically short) window in which committed data may not be visible. It is
a bit like the partition itself being unavailable temporarily. The data has
not been lost and is guaranteed to be returned, but the consumer has to
wait until the follower knows it is safe to return.

One final note: I am iterating on the design a little bit in order to
address Jun's comments. Expect a few changes. I realized that there is some
inconsistency with the current fetch behavior and KIP-207. It is mainly in
regard to how we handle the transition from becoming a follower to becoming
a leader.

Thanks,
Jason



On Tue, Dec 11, 2018 at 3:46 AM Eno Thereska  wrote:

> Hi Jason,
>
> My question was on producer + consumer semantics, not just the producer
> semantics. I'll rephrase it slightly and split into two questions:
> - scenario 1: an application that both produces and consumes (e.g., like
> Kafka streams) produces synchronously a single record to a topic and then
> attempts to consume that record. Topic is 3-way replicated say. Could it be
> the case that the produce succeeds but the consume fails? The consume could
> go to a replica that has not yet fetched the produce record, right? Or is
> that not possible?
> - scenario 2: an application C that only consumes. Again say there is only
> one record produced (by another application P) to a replicated topic and
> that record has not propagated to all replicas yet (it is only at the
> leader at time t0). Application C attempts to consume at time t1 and it
> does so successfully because the consume fetches from the leader. At time
> t2 the same application seeks to the beginning of the topic and attempts to
> consume again. Is there a scenario where this second attempt fails because
> the fetching happens from a replica that does not have the record yet? At
> time t3 all replicas have the record.
>
> Thanks
> Eno
>
>
>
>
> On Mon, Dec 10, 2018 at 7:42 PM Jason Gustafson 
> wrote:
>
> > Hey Eno,
> >
> > Thanks for the comments. However, I'm a bit confused. I'm not suggesting
> we
> > change Produce semantics in any way. All writes still go through the
> > partition leader and nothing changes with respect to committing to the
> ISR.
> > The main issue, as I've mentioned in the KIP, is the increased latency
> > before a c

Re: [DISCUSS] KIP-404: Add Kafka Connect configuration parameter for disabling WADL output on OPTIONS request

[Jason Gustafson]

Hi Alex,

I think WADL support was likely unintentional, so this could be treated as
more of a bug. Unless we think it's a good idea to support it going
forward, I'd suggest going with the rejected alternative of just turning it
off. What do you think?

Thanks,
Jason

On Fri, Dec 14, 2018 at 3:06 PM Oleksandr Diachenko <
alex.diache...@confluent.io> wrote:

> Thanks, everyone for taking the time to review the KIP.
>
> It looks like there are no major objections on it, so I will start voting
> thread.
>
> Regards, Alex.
>
>
>
> On Thu, Dec 13, 2018 at 3:50 PM Randall Hauch  wrote:
>
> > Thanks, Alex. The KIP looks good to me.
> >
> > Randall
> >
> > On Wed, Dec 12, 2018 at 10:08 PM Guozhang Wang 
> wrote:
> >
> > > Alex,
> > >
> > > Thanks for putting up this KIP. The proposal lgtm.
> > >
> > > Guozhang
> > >
> > > On Wed, Dec 12, 2018 at 7:41 PM Oleksandr Diachenko <
> > odiache...@apache.org
> > > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I would like to start a discussing for the following KIP:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-404%3A+Add+Kafka+Connect+configuration+parameter+for+disabling+WADL+output+on+OPTIONS+request
> > > > .
> > > >
> > > > The KIP proposes to add a configuration parameter for Connect Worker,
> > > which
> > > > would allow to not expose WADL information in Connect REST api
> > responces.
> > > >
> > > > Feedback is appreciated, thanks in advance.
> > > >
> > > > Regards, Alex.
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Jason Gustafson]

Hi Boyang,

Thanks, the KIP looks good. Just one comment.

The new schema for the LeaveGroup request is slightly odd since it is
handling both the single consumer use case and the administrative use case.
I wonder we could make it consistent from a batching perspective.

In other words, instead of this:
LeaveGroupRequest => GroupId MemberId [GroupInstanceId]

Maybe we could do this:
LeaveGroupRequest => GroupId [GroupInstanceId MemberId]

For dynamic members, GroupInstanceId could be empty, which is consistent
with JoinGroup. What do you think?

Also, just for clarification, what is the expected behavior if the current
memberId of a static member is passed to LeaveGroup? Will the static member
be removed? I know the consumer will not do this, but we'll still have to
handle the case on the broker.

Best,
Jason


On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen  wrote:

> Thanks Stanislav！
>
> Get Outlook for iOS
>
> 
> From: Stanislav Kozlovski 
> Sent: Monday, December 10, 2018 11:28 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> This is great work, Boyang. Thank you very much.
>
> +1 (non-binding)
>
> On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen  wrote:
>
> > Hey there, could I get more votes on this thread?
> >
> > Thanks for the vote from Mayuresh and Mike :)
> >
> > Best,
> > Boyang
> > 
> > From: Mayuresh Gharat 
> > Sent: Thursday, December 6, 2018 10:53 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > reduce consumer rebalances
> >
> > +1 (non-binding)
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger <
> mike.freyber...@xandr.com>
> > wrote:
> >
> > > +1 (non binding)
> > >
> > > On 12/4/18, 9:43 AM, "Patrick Williams" <
> patrick.willi...@storageos.com
> > >
> > > wrote:
> > >
> > > Pls take me off this VOTE list
> > >
> > > Best,
> > >
> > > Patrick Williams
> > >
> > > Sales Manager, UK & Ireland, Nordics & Israel
> > > StorageOS
> > > +44 (0)7549 676279
> > > patrick.willi...@storageos.com
> > >
> > > 20 Midtown
> > > 20 Proctor Street
> > > Holborn
> > > London WC1V 6NX
> > >
> > > Twitter: @patch37
> > > LinkedIn: linkedin.com/in/patrickwilliams4 <
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fin%2Fpatrickwilliams4&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092&sdata=ipDTX%2FGARrFkwZfRuOY0M5m3iJ%2Bnkxovv6u9bBDaTyc%3D&reserved=0
> > >
> > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslack.storageos.com%2F&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092&sdata=hxuKU6aZdQU%2FpxpqaaThR6IjpEmwIP5%2F3NhYzMYijkw%3D&reserved=0
> > >
> > >
> > >
> > > On 03/12/2018, 17:34, "Guozhang Wang"  wrote:
> > >
> > > Hello Boyang,
> > >
> > > I've browsed through the new wiki and there are still a couple of
> > > minor
> > > things to notice:
> > >
> > > 1. RemoveMemberFromGroupOptions seems not defined anywhere.
> > >
> > > 2. LeaveGroupRequest added a list of group instance id, but still
> > > keep the
> > > member id as a singleton; is that intentional? I think to make
> > the
> > > protocol
> > > consistent both member id and instance ids could be plural.
> > >
> > > 3. About the *kafka-remove-member-from-group.sh *tool, I'm
> > > wondering if we
> > > can defer adding this while just add the corresponding calls of
> > the
> > > LeaveGroupRequest inside Streams until we have used it in
> > > production and
> > > hence have a better understanding on how flexible or extensible
> > if
> > > we want
> > > to add any cmd tools. The rationale is that if we do not
> > > necessarily need
> > > it now, we can always add it later with a more think-through API
> > > design,
> > > but if we add the tool in a rush, we may need to extend or modify
> > > it soon
> > > after we realize its limits in operations.
> > >
> > > Otherwise, I'm +1 on the proposal.
> > >
> > > Guozhang
> > >
> > >
> > > On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen 
> > > wrote:
> > >
> > > > Hey community friends,
> > > >
> > > > after another month of polishing, KIP-345<
> > > >
> > >
> >
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435%7C1%7C0%7C636801101252994092&sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&reserved=0
> > > >
> > > > design is ready for vote. Feel free to add your comment on the
> > > discussion
> > > > thread or here.
> > > >
> > > > Thanks for your time!
> > > >
> > > > Boyang
> > > > __

Re: [DISCUSS] KIP-392: Allow consumers to fetch from the closest replica

[Jason Gustafson]

ld potentially make this configurable.


This is related to the discussion above. We could make it configurable I
guess. I wonder if it would be reasonable to just reduce the default to
something like 10 seconds. Do you think we get much benefit from such a
long delay?

5. It may be worth mentioning that, to take advantage of affinity, one may
> also want to have a customized PartitionAssignor to have an affinity aware
> assignment in addition to a customized ReplicaSelector.


Yes, this is a good point. I was assuming a situation in which each
partition had its replicas in all the same datacenters, but you are right
that this need not be the case. I will mention this in the KIP and give it
some more thought. I think in the common case, these concerns can be
treated orthogonally, but it is a bit irritating if you need two separate
plugins to make the benefit more general.


Thanks,
Jason

On Tue, Dec 11, 2018 at 11:04 AM Jason Gustafson  wrote:

> Hi Eno,
>
> Thanks for the clarification. From a high level, the main thing to keep in
> mind is that this is an opt-in feature. It is a bit like using acks=1 in
> the sense that a user is accepting slightly weaker guarantees in order to
> optimize for some metric (in this case, read locality). The default
> behavior would read only from the leader and users will get the usual
> semantics. That said, let me address the scenarios you raised:
>
> - scenario 1: an application that both produces and consumes (e.g., like
>> Kafka streams) produces synchronously a single record to a topic and then
>> attempts to consume that record. Topic is 3-way replicated say. Could it
>> be
>> the case that the produce succeeds but the consume fails? The consume
>> could
>> go to a replica that has not yet fetched the produce record, right? Or is
>> that not possible?
>
>
> I think it depends on what you mean by "fails." From a replica in the
> ISR's perspective, it has all of the committed data. The only question is
> what is safe to expose since the high watermark is always one round trip
> behind. The proposal is to return a retriable error in this case so that
> the consumer can distinguish the case from an out of range error and retry.
> No error will be returned to the user, but consumption will be delayed. One
> of the main improvements in the KIP is ensuring that this delay is minimal
> in the common case.
>
> Note that even without follower fetching, this scenario is unavoidable.
> When a replica becomes a leader, it doesn't know what the latest high
> watermark is until it receives fetches from all followers in the ISR.
> During this window, committed data is temporarily not visible. We handle
> this similarly to what is proposed here. Basically we ask the consumer to
> retry until we know the data is safe.
>
> - scenario 2: an application C that only consumes. Again say there is only
>> one record produced (by another application P) to a replicated topic and
>> that record has not propagated to all replicas yet (it is only at the
>> leader at time t0). Application C attempts to consume at time t1 and it
>> does so successfully because the consume fetches from the leader. At time
>> t2 the same application seeks to the beginning of the topic and attempts
>> to
>> consume again. Is there a scenario where this second attempt fails because
>> the fetching happens from a replica that does not have the record yet? At
>> time t3 all replicas have the record.
>
>
> Yes, this is possible in the way that I described above. There is a
> (typically short) window in which committed data may not be visible. It is
> a bit like the partition itself being unavailable temporarily. The data has
> not been lost and is guaranteed to be returned, but the consumer has to
> wait until the follower knows it is safe to return.
>
> One final note: I am iterating on the design a little bit in order to
> address Jun's comments. Expect a few changes. I realized that there is some
> inconsistency with the current fetch behavior and KIP-207. It is mainly in
> regard to how we handle the transition from becoming a follower to becoming
> a leader.
>
> Thanks,
> Jason
>
>
>
> On Tue, Dec 11, 2018 at 3:46 AM Eno Thereska 
> wrote:
>
>> Hi Jason,
>>
>> My question was on producer + consumer semantics, not just the producer
>> semantics. I'll rephrase it slightly and split into two questions:
>> - scenario 1: an application that both produces and consumes (e.g., like
>> Kafka streams) produces synchronously a single record to a topic and then
>> attempts to consume that record. Topic is 3-way replicated say. Could it
>> be
>> the case that the produce succeeds but the consume fa

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

[Jason Gustafson]

Hey Boyang,

I had just a follow-up to my comment above. I wanted to suggest an
alternative schema for LeaveGroup:

LeaveGroupRequest => GroupId [GroupInstanceId MemberId]

So we have a single array instead of two arrays. Each element identifies a
single member. For dynamic members, we would use GroupInstanceId="" and
provide the dynamic MemberId, which is consistent with JoinGroup. For
static members, GroupInstanceId must be provided and Member could be
considered optional. I think this makes the schema more coherent, but I'll
leave it to you if there is a good reason to keep them separate.

In any case, my vote is +1. Thanks for the hard work on this KIP!

Best,
Jason

On Fri, Jan 4, 2019 at 1:09 PM Boyang Chen  wrote:

> Thanks Guozhang for the proposal! The update is done.
>
> 
> From: Guozhang Wang 
> Sent: Saturday, January 5, 2019 3:33 AM
> To: dev
> Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> reduce consumer rebalances
>
> Hello Boyang,
>
> I've made another pass on the wiki page again. One minor comment is on the
> "Server Behavior Changes" section, we should have a paragraph on the
> logical changes on handling new versions of LeaveGroupRequest (e.g. how to
> handle dynamic member v.s. static member etc).
>
> Other than that, I do not have further comments. I think we can continue
> the voting process after that.
>
> Guozhang
>
> On Wed, Jan 2, 2019 at 10:00 AM Boyang Chen  wrote:
>
> > Thanks Jason for the comment! I answered it on the discuss thread.
> >
> > Folks, could we continue the vote for this KIP? This is a very critical
> > improvement for our streaming system
> > stability and we need to get things rolling right at the start of 2019.
> >
> > Thank you for your time!
> > Boyang
> >
> > 
> > From: Jason Gustafson 
> > Sent: Tuesday, December 18, 2018 7:40 AM
> > To: dev
> > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > reduce consumer rebalances
> >
> > Hi Boyang,
> >
> > Thanks, the KIP looks good. Just one comment.
> >
> > The new schema for the LeaveGroup request is slightly odd since it is
> > handling both the single consumer use case and the administrative use
> case.
> > I wonder we could make it consistent from a batching perspective.
> >
> > In other words, instead of this:
> > LeaveGroupRequest => GroupId MemberId [GroupInstanceId]
> >
> > Maybe we could do this:
> > LeaveGroupRequest => GroupId [GroupInstanceId MemberId]
> >
> > For dynamic members, GroupInstanceId could be empty, which is consistent
> > with JoinGroup. What do you think?
> >
> > Also, just for clarification, what is the expected behavior if the
> current
> > memberId of a static member is passed to LeaveGroup? Will the static
> member
> > be removed? I know the consumer will not do this, but we'll still have to
> > handle the case on the broker.
> >
> > Best,
> > Jason
> >
> >
> > On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen 
> wrote:
> >
> > > Thanks Stanislav！
> > >
> > > Get Outlook for iOS<https://aka.ms/o0ukef>
> > >
> > > 
> > > From: Stanislav Kozlovski 
> > > Sent: Monday, December 10, 2018 11:28 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > > reduce consumer rebalances
> > >
> > > This is great work, Boyang. Thank you very much.
> > >
> > > +1 (non-binding)
> > >
> > > On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen 
> wrote:
> > >
> > > > Hey there, could I get more votes on this thread?
> > > >
> > > > Thanks for the vote from Mayuresh and Mike :)
> > > >
> > > > Best,
> > > > Boyang
> > > > 
> > > > From: Mayuresh Gharat 
> > > > Sent: Thursday, December 6, 2018 10:53 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to
> > > > reduce consumer rebalances
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks,
> > > >
> > > > Mayuresh
> > > >
> > > > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger <
> > > mike.freyber...@xandr.com>
> > > > wrote:
> > > >
> > > 

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

[Jason Gustafson]

Hey Guozhang,

Thanks for sharing the article. The INVALID_PRODUCER_ID_MAPPING error
occurs following expiration of the producerId. It's possible that another
producerId has been installed in its place following expiration (if another
producer instance has become active), or the mapping is empty. We can
safely retry the InitProducerId with the logic in this KIP in order to
detect which case it is. So I'd suggest something like this:

1. After receiving INVALID_PRODUCER_ID_MAPPING, the producer can send
InitProducerId using the current producerId and epoch.
2. If no mapping exists, the coordinator can generate a new producerId and
return it. If a transaction is in progress on the client, it will have to
be aborted, but the producer can continue afterwards.
3. Otherwise if a different producerId has been assigned, then we can
return INVALID_PRODUCER_ID_MAPPING. To simplify error handling, we can
probably raise this as ProducerFencedException since that is effectively
what has happened. Ideally this is the only fatal case that users have to
handle.

I'll give it a little more thought and update the KIP.

Thanks,
Jason

On Thu, Jan 3, 2019 at 1:38 PM Guozhang Wang  wrote:

> You're right about the dangling txn since it will actually block
> read-committed consumers from proceeding at all. I'd agree that since this
> is a very rare case, we can consider fixing it not via broker-side logic
> but via tooling in a future work.
>
> I've also discovered some related error handling logic inside producer that
> may be addressed together with this KIP (since it is mostly for internal
> implementations the wiki itself does not need to be modified):
>
>
> https://stackoverflow.com/questions/53976117/why-did-the-kafka-stream-fail-to-produce-data-after-a-long-time/54029181#54029181
>
> Guozhang
>
>
>
> On Thu, Nov 29, 2018 at 2:25 PM Jason Gustafson 
> wrote:
>
> > Hey Guozhang,
> >
> > To clarify, the broker does not actually use the ApiVersion API for
> > inter-broker communications. The use of an API and its corresponding
> > version is controlled by `inter.broker.protocol.version`.
> >
> > Nevertheless, it sounds like we're on the same page about removing
> > DescribeTransactionState. The impact of a dangling transaction is a
> little
> > worse than what you describe though. Consumers with the read_committed
> > isolation level will be stuck. Still, I think we agree that this case
> > should be rare and we can reconsider for future work. Rather than
> > preventing dangling transactions, perhaps we should consider options
> which
> > allows us to detect them and recover. Anyway, this needs more thought. I
> > will update the KIP.
> >
> > Best,
> > Jason
> >
> > On Tue, Nov 27, 2018 at 6:51 PM Guozhang Wang 
> wrote:
> >
> > > 0. My original question is about the implementation details primarily,
> > > since current the handling logic of the APIVersionResponse is simply
> "use
> > > the highest supported version of the corresponding request", but if the
> > > returned response from APIVersionRequest says "I don't even know about
> > the
> > > DescribeTransactionStateRequest at all", then we need additional logic
> > for
> > > the falling back logic. Currently this logic is embedded in
> NetworkClient
> > > which is shared by all clients, so I'd like to avoid making this logic
> > more
> > > complicated.
> > >
> > > As for the general issue that a broker does not recognize a producer
> with
> > > sequence number 0, here's my thinking: as you mentioned in the wiki,
> this
> > > is only a concern for transactional producer since for idempotent
> > producer
> > > it can just bump the epoch and go. For transactional producer, even if
> > the
> > > producer request from a fenced producer gets accepted, its transaction
> > will
> > > never be committed and hence messages not exposed to read-committed
> > > consumers as well. The drawback is though, 1) read-uncommitted
> consumers
> > > will still read those messages, 2) unnecessary storage for those fenced
> > > produce messages, but in practice should not accumulate to a large
> amount
> > > since producer should soon try to commit and be told it is fenced and
> > then
> > > stop, 3) there will be no markers for those transactional messages
> ever.
> > > Looking at the list and thinking about the likelihood it may happen
> > > assuming we retain the producer up to transactional.id.timeout (default
> > is
> > > 7 days), I feel comfortable leaving it as is

Re: [VOTE] KIP-379: Multiple Consumer Group Management

[Jason Gustafson]

+1 Thanks for the KIP!

-Jason

On Mon, Jan 14, 2019 at 5:15 PM Gwen Shapira  wrote:

> I was also wondering about that. I think its for compatibility with
> the existing output.
>
> We can have a separate KIP to add JSON output.
>
> On Mon, Jan 14, 2019 at 7:55 AM M. Manna  wrote:
> >
> > Hi,
> >
> > Thanks for this KIP. Could you kindly clarify why CSV format is useful,
> but
> > not anything else?
> >
> > CSV format is ancient and the only reason it keep existing is various
> > application is because legacy applications aren't moving away from using
> > them. Would you would consider JSON or YAML?
> >
> > Also, if you think about the kafka-reassign-partitions - it's also using
> > JSON, not CSV. That is my only point. However, if majority feels that
> it's
> > not
> > an issue I believe it's a team decision after all :).
> >
> > Thanks,
> >
> >
> > On Mon, 14 Jan 2019 at 15:06, Gwen Shapira  wrote:
> >
> > > +1. Thanks, that will be very helpful.
> > >
> > > On Mon, Jan 14, 2019, 4:43 AM Alex D  > >
> > > > Hello guys,
> > > >
> > > > We need your VOTES for the KIP-379: Multiple Consumer Group
> Management.
> > > >
> > > > KIP-379:
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-379%3A+Multiple+Consumer+Group+Management
> > > >
> > > > PR: https://github.com/apache/kafka/pull/5726/
> > > >
> > > > Let's start the voting session.
> > > >
> > > > Thank you,
> > > >
> > > > Alex Dunayevsky
> > > >
> > >
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: [VOTE] KIP-389: Introduce a configurable consumer group size limit

[Jason Gustafson]

+1 Thanks for the KIP!

-Jason

On Mon, Jan 14, 2019 at 12:54 AM Andrew Schofield 
wrote:

> +1 (non-binding)
>
> Looks like a good improvement.
>
> Andrew Schofield
> IBM Event Streams
>
> On 11/01/2019, 17:33, "Boyang Chen"  wrote:
>
> +1 (non-binding)
>
> Thanks for the great work!
>
> Get Outlook for iOS<
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=02%7C01%7C%7Cb3ec0b70e2174aa2f48808d677eade50%7C84df9e7fe9f640afb435%7C1%7C0%7C636828247959240611&sdata=2aqjyi7E6tAcrz5P0OG2ZFoGuD4W2GFc8Nh5kj6Rmic%3D&reserved=0
> >
>
> 
> From: Gwen Shapira 
> Sent: Friday, January 11, 2019 9:13 AM
> To: dev
> Subject: Re: [VOTE] KIP-389: Introduce a configurable consumer group
> size limit
>
> +1
> Thank you for driving this change!
>
> On Fri, Jan 11, 2019, 1:09 AM Stanislav Kozlovski <
> stanis...@confluent.io
> wrote:
>
> > Hey folks,
> >
> > I'd like to initiate a vote thread about KIP-389
> > <
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-389%253A%2BIntroduce%2Ba%2Bconfigurable%2Bconsumer%2Bgroup%2Bsize%2Blimit&data=02%7C01%7C%7Cb3ec0b70e2174aa2f48808d677eade50%7C84df9e7fe9f640afb435%7C1%7C0%7C636828247959240611&sdata=RdZyv1aMRkoFgDCvY7XcbFWV0fTegO7qKu%2BLAqgbpzs%3D&reserved=0
> > >
> > .
> >
> > --
> > Best,
> > Stanislav
> >
>
>
>

[ANNOUNCE] New Committer: Vahid Hashemian

[Jason Gustafson]

Hi All,

The PMC for Apache Kafka has invited Vahid Hashemian as a project committer and
we are
pleased to announce that he has accepted!

Vahid has made numerous contributions to the Kafka community over the past
few years. He has authored 13 KIPs with core improvements to the consumer
and the tooling around it. He has also contributed nearly 100 patches
affecting all parts of the codebase. Additionally, Vahid puts a lot of
effort into community engagement, helping others on the mail lists and
sharing his experience at conferences and meetups.

We appreciate the contributions and we are looking forward to more.
Congrats Vahid!

Jason, on behalf of the Apache Kafka PMC

Re: [VOTE] KIP-283: Efficient Memory Usage for Down-Conversion

[Jason Gustafson]

Hi Dhruvil,

Thanks for the KIP. +1 from me. Just a minor nitpick on the name of the new
config. I would suggest "record.downconversion.enable". The "record" prefix
emphasizes what is being down-converted and similar existing configs use
"enable" rather than "enabled."

-Jason

On Wed, May 2, 2018 at 9:35 AM, Ted Yu  wrote:

> +1
>
> On Wed, May 2, 2018 at 9:27 AM, Dhruvil Shah  wrote:
>
> > Hi all,
> >
> > I would like to start the vote on KIP-238: Efficient Memory Usage for
> > Down-Conversion.
> >
> > For reference, the link to the KIP is here:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 283%3A+Efficient+Memory+Usage+for+Down-Conversion
> >
> > and the discussion thread is here:
> > https://www.mail-archive.com/dev@kafka.apache.org/msg86799.html
> >
> > Thanks,
> > Dhruvil
> >
>

Re: [DISCUSS] KIP-278: Add version option to Kafka's commands

[Jason Gustafson]

Hi Sasaki,

Thanks for the update. The KIP looks good to me. I'd suggest moving to a
vote.

Thanks,
Jason

On Mon, May 7, 2018 at 7:08 AM, Sasaki Toru 
wrote:

> Hi Manikumar, Colin,
>
> Thank you for your comment.
>
> As Colin said, I proposed adding an option to show version information of
> "local" tool,
> because many software have this option and I think Kafka also needs this
> one.
>
> As you said, the function to show remote Kafka version is useful,
> but I think it is better to create new KIP because this function has some
> points which should be considered.
>
> If you have any better ideas, could you please tell us?
>
>
> Many thanks,
> Sasaki
>
> From: Manikumar 
>> Date: 2018-05-03 4:11 GMT+09:00
>>
>> Subject: Re: [DISCUSS] KIP-278: Add version option to Kafka's commands
>> To: dev 
>>
>>
>> Hi Colin,
>>
>> Thanks for explanation. It's definitely useful to have  --version flag.
>>
>> kafka-broker-api-versions.sh gives the API versions, not Kafka release
>> version.
>> Is not easy to figure out release version from API versions. Currently
>> release version is available via metric/JMX.
>> If required, we can implement this in future.
>>
>>
>> Thanks,
>>
>> On Wed, May 2, 2018 at 10:58 PM, Colin McCabe  wrote:
>>
>> Hi Manikumar,
>>>
>>> We already have a tool for getting the Kafka broker API versions,
>>> "./bin/kafka-broker-api-versions.sh".  It was added as part of KIP-97.
>>>
>>> What Saski is proposing here is having a way of getting the version of
>>> locally installed Kafka software, which may be different from the server
>>> version.  Many pieces of software offer a --version flag, and it's always
>>> understood to refer to the local version of the software, not a version
>>> running somewhere else.  The user has no way to get this information now,
>>> other than perhaps trying to look at he names of jar files.
>>>
>>> cheers,
>>> Colin
>>>
>>> On Tue, May 1, 2018, at 08:20, Manikumar wrote:
>>>
>>>> I assume the intent of the KIP to find out the Kafka broker version.  In
>>>> this case, maybe we should expose
>>>> version using a Kafka request. This will help the remote scripts/tools
>>>>
>>> to
>>
>>> query the Kafka version.
>>>> scripts (kafka-topics.sh, kafka-configs.sh, etc..)  may run from remote
>>>> machines  and may use
>>>> older Kafka versions. In this case, current proposal prints on the older
>>>> version.
>>>>
>>>> On Tue, May 1, 2018 at 7:47 PM, Colin McCabe 
>>>> wrote:
>>>>
>>>> Thanks, Sasaki.
>>>>>
>>>>> Colin
>>>>>
>>>>> On Sat, Apr 28, 2018, at 00:55, Sasaki Toru wrote:
>>>>>
>>>>>> Hi Colin, Jason,
>>>>>>
>>>>>> Thank you for your beneficial comment.
>>>>>> I have updated my Pull Request to show git commit hash in version
>>>>>> information.> In my current Pull Request, we cat get the result such
>>>>>>
>>>>> below:
>>>>>
>>>>>> $ bin/kafka-topics.sh --version
>>>>>> (snip)
>>>>>> 2.0.0-SNAPSHOT (Commit:f3876cd9617faf7e)
>>>>>>
>>>>>>
>>>>>> I have also updated to accept double-dash for this option (--
>>>>>> version) only.>
>>>>>>
>>>>>> Many thanks,
>>>>>> Sasaki
>>>>>>
>>>>>> From: Jason Gustafson 
>>>>>>> Date: 2018-04-25 9:42 GMT+09:00
>>>>>>> Subject: Re: [DISCUSS] KIP-278: Add version option to Kafka's
>>>>>>> commands> > To: dev 
>>>>>>>
>>>>>>>
>>>>>>> +1 on adding the git commit id to the output. We often encounter
>>>>>>> environments which are testing off of trunk or have modifications
>>>>>>>
>>>>>> on
>>>
>>>> top of> > an existing release.
>>>>>>>
>>>>>>> -Jason
>>>>>>>
>>>>>>> On Tue, Apr 24, 2018 at 10:06 AM, Colin McCabe >>>>>> wrote:> >
>>>>>>>
>>>>>>>> On Tue, Apr 24, 2018, at 05:36, Sas

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Jason Gustafson]

Thanks for the KIP, +1 (binding).

One small correction: the KIP mentions that close() will be deprecated, but
we do not want to do this because it is needed by the Closeable interface.
We only want to deprecate close(long, TimeUnit) in favor of close(Duration).

-Jason

On Tue, May 8, 2018 at 12:43 AM, khaireddine Rezgui <
khaireddine...@gmail.com> wrote:

> +1
>
> 2018-05-07 20:35 GMT+01:00 Bill Bejeck :
>
> > +1
> >
> > Thanks,
> > Bill
> >
> > On Fri, May 4, 2018 at 7:21 PM, Richard Yu 
> > wrote:
> >
> > > Hi all, I would like to bump this thread since discussion in the KIP
> > > appears to be reaching its conclusion.
> > >
> > >
> > >
> > > On Thu, Mar 15, 2018 at 3:30 PM, Richard Yu <
> yohan.richard...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Since there does not seem to be too much discussion in KIP-266, I
> will
> > be
> > > > starting a voting thread.
> > > > Here is the link to KIP-266 for reference:
> > > >
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=75974886
> > > >
> > > > Recently, I have made some updates to the KIP. To reiterate, I have
> > > > included KafkaConsumer's commitSync,
> > > > poll, and committed in the KIP. (we will be adding to a
> > TimeoutException
> > > > to them as well, in a similar manner
> > > > to what we will be doing for position())
> > > >
> > > > Thanks,
> > > > Richard Yu
> > > >
> > > >
> > >
> >
>
>
>
> --
> Ingénieur en informatique
>

Re: [VOTE] KIP-222 - Add "describe consumer group" to KafkaAdminClient

[Jason Gustafson]

Hi All,

We found one more gap in this KIP. It is useful for the
describeConsumerGroups API to expose the group state so that users can tell
when a group is rebalancing. Colin submitted a PR with the proposed changes
here: https://github.com/apache/kafka/pull/4980. Please review and let us
know if you have any concerns.

Thanks,
Jason

On Sun, Apr 15, 2018 at 10:22 AM, Guozhang Wang  wrote:

> Hello folks,
>
> I have update the KIP-222 wiki slightly upon the discussions we had during
> the implementations of the KIP. Please find more information on the
> following PRs:
>
> https://github.com/apache/kafka/pull/4454
>
> https://github.com/apache/kafka/pull/4856
>
>
> Guozhang
>
> On Tue, Feb 6, 2018 at 2:33 PM, Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Thanks Matthias. I have updated the version on KIP main page also.
> >
> > There are some changes that arise on the Pull Request. I will comment
> them
> > on the discussion thread.
> >
> > Cheers,
> > Jorge.
> >
> > El vie., 2 feb. 2018 a las 20:50, Matthias J. Sax (<
> matth...@confluent.io
> > >)
> > escribió:
> >
> > > Feature freeze for 1.1 passed already, thus, KIP-222 will not be part
> of
> > > 1.1 release.
> > >
> > > I updated the JIRA with target version 1.2.
> > >
> > > -Matthias
> > >
> > > On 2/1/18 3:57 PM, Jeff Widman wrote:
> > > > Don't forget to update the wiki page now that the vote has passed--it
> > > > currently says this KIP is "under discussion":
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 222+-+Add+Consumer+Group+operations+to+Admin+API
> > > >
> > > > Also, should the JIRA ticket be tagged with 1.1.0 (provided this is
> > > merged
> > > > by then)?
> > > >
> > > > On Mon, Jan 22, 2018 at 3:26 AM, Jorge Esteban Quilcate Otoya <
> > > > quilcate.jo...@gmail.com> wrote:
> > > >
> > > >> My bad, KIP is updated:
> > > >>
> > > >> ```
> > > >> public class MemberDescription {
> > > >> private final String consumerId;
> > > >> private final String clientId;
> > > >> private final String host;
> > > >> private final MemberAssignment assignment;
> > > >> }
> > > >> public class MemberAssignment {
> > > >> private final List assignment;
> > > >> }
> > > >> ```
> > > >>
> > > >> Cheers,
> > > >> Jorge.
> > > >>
> > > >> El lun., 22 ene. 2018 a las 6:46, Jun Rao ()
> > > escribió:
> > > >>
> > > >>> Hi, Jorge,
> > > >>>
> > > >>> For #3, I wasn't suggesting using the internal Assignment. We can
> > just
> > > >>> introduce a new public type that wraps List. We can
> > > call
> > > >> it
> > > >>> sth like MemberAssignment to distinguish it from the internal one.
> > This
> > > >>> makes extending the type in the future easier.
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Jun
> > > >>>
> > > >>> On Sun, Jan 21, 2018 at 3:19 PM, Jorge Esteban Quilcate Otoya <
> > > >>> quilcate.jo...@gmail.com> wrote:
> > > >>>
> > >  Hi all,
> > > 
> > >  Thanks all for your votes and approving this KIP :)
> > > 
> > >  @Jun Rao:
> > > 
> > >  1. Yes, KIP is updated with MemberDescription.
> > >  2. Changed:
> > >  ```
> > >  public class ListGroupOffsetsResult {
> > >  final KafkaFuture>
> > future;
> > >  ```
> > >  3. Not sure about this one as Assignment type is part of
> > >  o.a.k.clients.consumer.internals. Will we be breaking
> encapsulation
> > > >> if we
> > >  expose it as part of AdminClient?
> > >  Currently is defined as:
> > >  ```
> > >  public class MemberDescription {
> > >  private final String consumerId;
> > >  private final String clientId;
> > >  private final String host;
> > >  private final List assignment;
> > >  }
> > >  ```
> > > 
> > >  BTW: I've created a PR with the work in progress:
> > >  https://github.com/apache/kafka/pull/4454
> > > 
> > >  Cheers,
> > >  Jorge.
> > > 
> > >  El vie., 19 ene. 2018 a las 23:52, Jun Rao ()
> > > >>> escribió:
> > > 
> > > > Hi, Jorge,
> > > >
> > > > Thanks for the KIP. Looks good to me overall. A few comments
> below.
> > > >
> > > > 1. It seems that ConsumerDescription should be MemberDescription?
> > > >
> > > > 2. Each offset can have an optional metadata. So, in
> > > > ListGroupOffsetsResult, perhaps it's better to have
> > > > KafkaFuture>, where
> > > > OffsetAndMetadata contains an offset and a metadata of String.
> > > >
> > > > 3. As Jason mentioned in the discussion, it would be nice to
> extend
> > > >>> this
> > > > api to support general group management, instead of just the
> > consumer
> > >  group
> > > > in the future. For that, it might be better for MemberDescription
> > to
> > > >>> have
> > > > assignment of type Assignment, which consists of a list of
> > > >> partitions.
> > > > Then, in the future, we can add other fields to Assignment.
> > > >
> > > > Jun
> > > >>>

Re: [VOTE] KIP-278: Add version option to Kafka's commands

[Jason Gustafson]

Thanks for the KIP! +1 (binding)

On Fri, May 11, 2018 at 12:35 AM, Manikumar 
wrote:

> +1 (non-binding)
>
> Thanks for the KIP.
>
> On Fri, May 11, 2018 at 12:56 PM, zhenya Sun  wrote:
>
> > +1 building
> > > 在 2018年5月11日，上午9:51，Ted Yu  写道：
> > >
> > > +1
> > >
> > > On Thu, May 10, 2018 at 6:42 PM, Sasaki Toru <
> sasaki...@oss.nttdata.com>
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> I would like to start the vote on KIP-278: Add version option to
> Kafka's
> > >> commands.
> > >>
> > >> The link to this KIP is here:
> > >>  > >> +Add+version+option+to+Kafka%27s+commands>
> > >>
> > >> The discussion thread is here:
> > >> 
> > >>
> > >>
> > >> Many thanks,
> > >> Sasaki
> > >>
> > >> --
> > >> Sasaki Toru(sasaki...@oss.nttdata.com) NTT DATA CORPORATION
> > >>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

[Jason Gustafson]

Hi Magesh,

Thanks for the KIP. It's definitely useful to have a pluggable auth layer,
as we have for the kafka brokers. I was a bit unclear why we needed to
expose all this health information in the context though. Since it is the
bigger part of the API, I was hoping to see a little more motivation for
why a plugin would need this. Do the current status endpoints not integrate
well with common alerting systems? Is that something that can be fixed?

Thanks,
Jason

On Thu, May 17, 2018 at 10:05 PM, Ewen Cheslack-Postava 
wrote:

> Yup, thanks for the changes. The 'health' package in particular feels like
> a nice fit given the way we expect it to be used.
>
> -Ewen
>
> On Wed, May 16, 2018 at 7:02 PM Randall Hauch  wrote:
>
> > Looks good to me. Thanks for quickly making the changes! Great work!
> >
> > Best regards,
> >
> > Randall
> >
> > > On May 16, 2018, at 8:07 PM, Magesh Nandakumar 
> > wrote:
> > >
> > > Randall,
> > >
> > > I have adjusted the package names per Ewen's suggestions and also made
> > some
> > > minor edits per your suggestions. Since there are no major outstanding
> > > issues, i'm moving this to vote.
> > >
> > > Thanks
> > > Magesh
> > >
> > >> On Wed, May 16, 2018 at 4:38 PM, Randall Hauch 
> > wrote:
> > >>
> > >> A few very minor suggestions:
> > >>
> > >>
> > >>   1. There are a few formatting issues with paragraphs that use a
> > >>   monospace font. Minor, but it would be nice to fix.
> > >>   2. Would be nice to link to the PR
> > >>   3. Do we need the org.apache.kafka.connect.rest.extension.entities
> > >>   package? Could we just move the two classes into the parent
> > >>   org.apache.kafka.connect.rest.extension package?
> > >>   4. This sentence "The above approach helps alleviate any issues that
> > >>   could arise if Extension accidentally reregister the" is cut off.
> > >>   5. The "ConnectRestExtensionContext.configure(...)" method's
> JavaDoc
> > >>   should describe the behaviors that are mentioned in the "Rest
> > Extension
> > >>   Integration with Connect" section; e.g., behavior when an extension
> > >> adds a
> > >>   resource that is already registered, whether unregistering works,
> etc.
> > >>   Also, ideally the "close()" method would have JavaDoc that explained
> > >> when
> > >>   it is called (e.g., no other methods will be called on the extension
> > >> after
> > >>   this, etc.).
> > >>   6. Packaging requirements are different for this component vs
> > >>   connectors, transformations, and converters, since this now mandates
> > the
> > >>   Service Loader manifest file. This should be called out more
> > explicitly.
> > >>   7. It'd be nice if the example included how extension-specific
> config
> > >>   properties are to be defined in the worker configuration file.
> > >>
> > >> As I said, these are all minor suggestions that only affect the KIP
> > >> document. Once these are fixed, I think this is ready to move to
> voting.
> > >>
> > >> Best regards,
> > >>
> > >> Randall
> > >>
> > >> On Tue, May 15, 2018 at 11:30 AM, Magesh Nandakumar <
> > mage...@confluent.io>
> > >> wrote:
> > >>
> > >>> Randall- I think I have addressed all the comments. Let me know if we
> > can
> > >>> take this to Vote.
> > >>>
> > >>> Thanks
> > >>> Magesh
> > >>>
> > >>> On Tue, May 8, 2018 at 10:12 PM, Magesh Nandakumar <
> > mage...@confluent.io
> > >>>
> > >>> wrote:
> > >>>
> >  Hi All,
> > 
> >  I have updated the KIP to reflect changes based on the PR
> >  https://github.com/apache/kafka/pull/4931. Its mostly has minor
> > >> changes
> >  to the interfaces and includes details on packages for the
> interfaces
> > >> and
> >  the classes. Let me know your thoughts.
> > 
> >  Thanks
> >  Magesh
> > 
> >  On Fri, Apr 27, 2018 at 12:03 PM, Randall Hauch 
> > >>> wrote:
> > 
> > > Great work, Magesh. I like the overall approach a lot, so I left
> some
> > > pretty nuanced comments about specific details.
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> > > On Wed, Apr 25, 2018 at 3:03 PM, Magesh Nandakumar <
> > >>> mage...@confluent.io>
> > > wrote:
> > >
> > >> Thanks Randall for your thoughts. I have created a replica of the
> > > required
> > >> entities in the draft implementation. If you can take a look at
> the
> > >> PR
> > > and
> > >> let me know your thoughts, I will update the KIP to reflect the
> same
> > >>
> > >> https://github.com/apache/kafka/pull/4931
> > >>
> > >> On Tue, Apr 24, 2018 at 11:44 AM, Randall Hauch  >
> > > wrote:
> > >>
> > >>> Magesh, I think our last emails cross in mid-stream.
> > >>>
> > >>> We definitely want to put the new public interfaces/classes in
> the
> > >>> API
> > >>> module, and implementation in the runtime module. Yes, this will
> > > affect
> > >> the
> > >>> design, since for example we don't want to expose runtime types
> to
> > >>> the
> > >> AP

Re: [DISCUSS] KIP-298: Error Handling in Connect

[Jason Gustafson]

Hi Arjun,

Thanks for the KIP. Just a few comments/questions:

1. The proposal allows users to configure the number of retries. I usually
find it easier as a user to work with timeouts since it's difficult to know
how long a retry might take. Have you considered adding a timeout option
which would retry until the timeout expires?
2. The configs are named very generically (e.g. errors.retries.limit). Do
you think it will be clear to users what operations these configs apply to?
3. I wasn't entirely clear what messages are stored in the dead letter
queue. It sounds like it includes both configs and messages since we have
errors.dlq.include.configs? Is there a specific schema you have in mind?
4. I didn't see it mentioned explicitly in the KIP, but I assume the
tolerance metrics are reset after every task rebalance?
5. I wonder if we can do without errors.tolerance.limit. You can get a
similar effect using errors.tolerance.rate.limit if you allow longer
durations. I'm not sure how useful an absolute counter is in practice.

Thanks,
Jason


On Fri, May 18, 2018 at 2:55 PM, Arjun Satish 
wrote:

> Super! Thanks, Magesh!
>
> On Fri, May 18, 2018 at 2:53 PM, Magesh Nandakumar 
> wrote:
>
> > Arjun,
> >
> > Thanks for all the updates. I think it looks great and I don't have any
> > other concerns.
> >
> > Thanks
> > Magesh
> >
> > On Fri, May 18, 2018 at 2:36 PM, Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > > The updated version of the KIP that uses the dead-letter-queue only for
> > > sink records and only to store the raw record data looks better and
> > easier
> > > to understand.
> > > I think it's moving to the right direction.
> > >
> > > No further comments from my side.
> > >
> > > Thanks Arjun!
> > >
> > > - Konstantine
> > >
> > > On Fri, May 18, 2018 at 1:07 AM, Arjun Satish 
> > > wrote:
> > >
> > > > Ewen,
> > > >
> > > > Thanks a lot for your comments!
> > > >
> > > > 1. For errors.retry.delay.max.ms, yes we plan to use exponential
> > > backoffs
> > > > with an fixed initial value. Updated the KIP to say this.
> > > >
> > > > 2. A failed operation will be retried (potentially multiple times).
> If
> > > all
> > > > the retries fail, we declare this to be an error. This is where
> > tolerance
> > > > kicks in. Hence, you can have 0 retries, but infinite tolerance (
> > > > errors.tolerance.limit = -1), where we will never retry any failure,
> > but
> > > > all of bad records will be skipped. Updated the KIP. Hopefully this
> is
> > > > clear now.
> > > >
> > > > 3. Yes, for error messages we have some base information (what
> > operation
> > > > failed and with what exception and stacktrace, for example). Hence,
> the
> > > > three configs. The main reason for having properties for disabling
> > > messages
> > > > and configs is to avoid logging sensitive information to unsecured
> > > > locations (for example, the file logs). Updated the KIP to describe
> > this.
> > > >
> > > > I think topic name should be mandatory: if we have a default topic,
> > then
> > > > all the connectors in a cluster will produce messages into it, making
> > it
> > > > confusing to read from. We could have a default pattern for
> > constructing
> > > > topic names, for example: a format like ${connector-name}-errors.
> > > >
> > > > 4. The reason for multiple clusters is to allow users with sensitive
> > data
> > > > to log errors into secure clusters. There are defaults for these
> > > > properties, but if you think this is making the config too complex,
> we
> > > can
> > > > drop the errors.deadletterqueue.producer.* properties from this
> > > > implementation.
> > > >
> > > > 5. I had mentioned that the format is in JSON in the proposed changes
> > > > section. Updated the public interface section to say this again. We
> > could
> > > > provide overrides for the Converter used here, and use an
> AvroConverter
> > > > instead, which should preserve the structure and schema of the data.
> > The
> > > > avro binary would be base64 encoded in the logged records. But yes,
> > this
> > > > brings in configurable converters and their configurations which
> > > introduces
> > > > a new level of complexity (AvroConverters and their dependency on
> > Schema
> > > > Registry, for instance). Hence, they were not included in this
> > proposal.
> > > >
> > > > Another option is to add a StructSerializer and StructDeserializer,
> > which
> > > > can retain the schema and structure of the Structs in the schema. If
> we
> > > do
> > > > this, non-Java clients which need to read these error records would
> > need
> > > to
> > > > port the deserialization logic. Ultimately, we need to indicate what
> > the
> > > > record looks like, and
> > > >
> > > > Could you point out what is unclear w.r.t reprocessing?
> > > >
> > > > Let me know what you think.
> > > >
> > > >
> > > > On Thu, May 17, 2018 at 11:02 PM, Ewen Cheslack-Postava <
> > > e...@confluent.io
> > > > >
> > > > wrote:
> > > >
> > > > > A few more thoug

Re: [DISCUSS] KIP-298: Error Handling in Connect

[Jason Gustafson]

Thanks Arjun. I had one additional question. High level, I'm wondering if
it makes sense to treat processing errors such as serialization failures
the same as IO errors. In the former case, retrying typically doesn't help
because the processing is deterministic. In the latter case, the failure
may be downstream (e.g. a kafka partition may be temporarily unavailable).
As a user, I would probably want the option to skip over processing
failures, but retry indefinitely if the downstream system is unavailable.
Is that use case supported?

Thanks,
Jason



On Mon, May 21, 2018 at 12:39 PM, Arjun Satish 
wrote:

> Thanks a lot, Ewen! I'll make sure the documentation is clear on the
> differences between retries an tolerance.
>
> Do you think percentage would have the same problem as the one you brought
> up? Also, if we say 10% tolerance, do we have to wait for the duration to
> finish before failing the task, or should we fail as soon as we hit 10%
> error.
>
> Alternatively, do you think making tolerance an Enum would be simpler?
> Where it's values are NONE (any errors kill), ALL (tolerate all errors and
> skip records) and FIRST (tolerate the first error, but fail after that)?
>
> Best,
>
>
> On Mon, May 21, 2018 at 11:28 AM, Ewen Cheslack-Postava  >
> wrote:
>
> > Arjun,
> >
> > Understood on retries vs tolerance -- though I suspect this will end up
> > being a bit confusing to users as well. It's two levels of error handling
> > which is what tripped me up.
> >
> > One last comment on KIP (which otherwise looks good): for the tolerance
> > setting, do we want it to be an absolute value or something like a
> > percentage? Given the current way of setting things, I'm not sure I'd
> ever
> > set it to anything but -1 or 0, with maybe 1 as an easy option for
> > restarting a connector to get it past one bad message, then reverting
> back
> > to -1 or 0.
> >
> > -Ewen
> >
> > On Mon, May 21, 2018 at 11:01 AM Arjun Satish 
> > wrote:
> >
> > > Hey Jason,
> > >
> > > Thanks for your comments. Please find answers inline:
> > >
> > > On Mon, May 21, 2018 at 9:52 AM, Jason Gustafson 
> > > wrote:
> > >
> > > > Hi Arjun,
> > > >
> > > > Thanks for the KIP. Just a few comments/questions:
> > > >
> > > > 1. The proposal allows users to configure the number of retries. I
> > > usually
> > > > find it easier as a user to work with timeouts since it's difficult
> to
> > > know
> > > > how long a retry might take. Have you considered adding a timeout
> > option
> > > > which would retry until the timeout expires?
> > > >
> > >
> > > Good point. Updated the KIP.
> > >
> > > 2. The configs are named very generically (e.g. errors.retries.limit).
> Do
> > > > you think it will be clear to users what operations these configs
> apply
> > > to?
> > > >
> > >
> > > As of now, these configs are applicable to all operations in the
> > connector
> > > pipeline (as mentioned in the proposed changes section). We decided not
> > to
> > > have per operation limit because of the additional config complexity.
> > >
> > >
> > > > 3. I wasn't entirely clear what messages are stored in the dead
> letter
> > > > queue. It sounds like it includes both configs and messages since we
> > have
> > > > errors.dlq.include.configs? Is there a specific schema you have in
> > mind?
> > > >
> > >
> > > This has been addressed in the KIP. The DLQ will now contain only raw
> > > messages (no additional context). We are also supporting DLQs only for
> > sink
> > > connectors now.
> > >
> > >
> > > > 4. I didn't see it mentioned explicitly in the KIP, but I assume the
> > > > tolerance metrics are reset after every task rebalance?
> > > >
> > >
> > > Great question. Yes, we will reset the tolerance metrics on every
> > > rebalance.
> > >
> > >
> > > > 5. I wonder if we can do without errors.tolerance.limit. You can get
> a
> > > > similar effect using errors.tolerance.rate.limit if you allow longer
> > > > durations. I'm not sure how useful an absolute counter is in
> practice.
> > > >
> > >
> > > Yeah, the rate limit does subsume the features offered by the absolute
> > > counter. Removed it.
> > >
> > >
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-306: Configuration for Delaying Response to Failed Client Authentication

[Jason Gustafson]

+1. Just one nit: could we use an INT type for the config? I can't imagine
that not being enough.

-Jason

On Mon, May 21, 2018 at 3:59 PM, Ismael Juma  wrote:

> Thanks for the KIP, +1 (binding).
>
> Ismael
>
> On Mon, May 21, 2018 at 7:52 AM Dhruvil Shah  wrote:
>
> > Hi,
> >
> > I would like to start a vote on KIP-306 which proposes to add a
> > configuration to delay responses to failed authentication.
> >
> > Link to the KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 306%3A+Configuration+for+Delaying+Response+to+Failed+Client+Authentication
> >
> > Thanks,
> > Dhruvil
> >
>

Re: [VOTE] KIP-298: Error Handling in Connect kafka

[Jason Gustafson]

+1. Thanks for the KIP!

On Mon, May 21, 2018 at 1:34 PM, Arjun Satish 
wrote:

> All,
>
> Thanks so much for your feedback on this KIP. I've made some small
> modifications today. I'll wait till midnight today (PDT) to close this
> vote. Please let me know if there are any further comments.
>
> Best,
>
> On Mon, May 21, 2018 at 11:29 AM, Ewen Cheslack-Postava  >
> wrote:
>
> > +1 binding. I had one last comment in the DISCUSS thread, but not really
> a
> > blocker.
> >
> > -Ewen
> >
> > On Mon, May 21, 2018 at 9:48 AM Matthias J. Sax 
> > wrote:
> >
> > > +1 (binding)
> > >
> > >
> > >
> > > On 5/21/18 9:30 AM, Randall Hauch wrote:
> > > > Thanks, Arjun. +1 (non-binding)
> > > >
> > > > Regards,
> > > > Randall
> > > >
> > > > On Mon, May 21, 2018 at 11:14 AM, Guozhang Wang 
> > > wrote:
> > > >
> > > >> Thanks for the KIP. +1 (binding)
> > > >>
> > > >>
> > > >> Guozhang
> > > >>
> > > >> On Fri, May 18, 2018 at 3:36 PM, Gwen Shapira 
> > > wrote:
> > > >>
> > > >>> +1
> > > >>>
> > > >>> Thank you! Error handling in Connect will be a huge improvement.
> > > >>>
> > > >>> On Thu, May 17, 2018, 1:58 AM Arjun Satish  >
> > > >> wrote:
> > > >>>
> > >  All,
> > > 
> > >  Many thanks for all the feedback on KIP-298. Highly appreciate the
> > > time
> > > >>> and
> > >  effort you all put into it.
> > > 
> > >  I've updated the KIP accordingly, and would like to start to
> start a
> > > >> vote
> > >  on it.
> > > 
> > >  KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >  298%3A+Error+Handling+in+Connect
> > >  JIRA: https://issues.apache.org/jira/browse/KAFKA-6738
> > >  Discussion Thread: https://www.mail-archive.com/
> > >  dev@kafka.apache.org/msg87660.html
> > > 
> > >  Thanks very much!
> > > 
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> -- Guozhang
> > > >>
> > > >
> > >
> > >
> >
>

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Jason Gustafson]

Hi Everyone,

There remains some inconsistency in the timeout behavior of the consumer
APIs which do not accept a timeout. Some of them block forever (e.g.
position()) and some of them use request.timeout.ms (e.g. parititonsFor()).
I think we'd probably all agree that blocking forever is not useful
behavior and using request.timeout.ms has always been a hack since it
controls a separate concern. I think there are basically two options to
address this:

1. We can add max.block.ms to match the producer and use it as the default
timeout when a timeout is not explicitly provided. This will fix the
indefinite blocking behavior and avoid conflating request.timeout.ms.
2. We can deprecate the methods which don't accept a timeout.

I'm leaning toward the first solution because I think we want to push users
to specifying timeouts through configuration rather than in code (Jay's
original argument). I think the overloads are still useful for advanced
usage (e.g. in kafka streams), but we should give users an easy option with
reasonable default behavior.

If that sounds ok, I'd propose we add it to this KIP and fix it now. This
gives users an easy way to get the benefit of the improvements from this
KIP without changing any code.

Thanks,
Jason




On Sun, May 13, 2018 at 7:58 PM, Richard Yu 
wrote:

> Hi,
>
> With 3 binding votes and 6 non-binding, this KIP would be accepted.
>
> Thanks for participating.
>
> On Thu, May 10, 2018 at 2:35 AM, Edoardo Comar  wrote:
>
> > +1 (non-binding)
> >
> > On 10 May 2018 at 10:29, zhenya Sun  wrote:
> >
> > > +1 non-binding
> > >
> > > > 在 2018年5月10日，下午5:19，Manikumar  写道：
> > > >
> > > > +1 (non-binding).
> > > > Thanks.
> > > >
> > > > On Thu, May 10, 2018 at 2:33 PM, Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > >> +1 (non binding)
> > > >> Thanks
> > > >>
> > > >> On Thu, May 10, 2018 at 9:39 AM, Rajini Sivaram <
> > > rajinisiva...@gmail.com>
> > > >> wrote:
> > > >>> Hi Richard, Thanks for the KIP.
> > > >>>
> > > >>> +1 (binding)
> > > >>>
> > > >>> Regards,
> > > >>>
> > > >>> Rajini
> > > >>>
> > > >>> On Wed, May 9, 2018 at 10:54 PM, Guozhang Wang  >
> > > >> wrote:
> > > >>>
> > > >>>> +1 from me, thanks!
> > > >>>>
> > > >>>>
> > > >>>> Guozhang
> > > >>>>
> > > >>>> On Wed, May 9, 2018 at 10:46 AM, Jason Gustafson <
> > ja...@confluent.io>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Thanks for the KIP, +1 (binding).
> > > >>>>>
> > > >>>>> One small correction: the KIP mentions that close() will be
> > > >> deprecated,
> > > >>>> but
> > > >>>>> we do not want to do this because it is needed by the Closeable
> > > >>>> interface.
> > > >>>>> We only want to deprecate close(long, TimeUnit) in favor of
> > > >>>>> close(Duration).
> > > >>>>>
> > > >>>>> -Jason
> > > >>>>>
> > > >>>>> On Tue, May 8, 2018 at 12:43 AM, khaireddine Rezgui <
> > > >>>>> khaireddine...@gmail.com> wrote:
> > > >>>>>
> > > >>>>>> +1
> > > >>>>>>
> > > >>>>>> 2018-05-07 20:35 GMT+01:00 Bill Bejeck :
> > > >>>>>>
> > > >>>>>>> +1
> > > >>>>>>>
> > > >>>>>>> Thanks,
> > > >>>>>>> Bill
> > > >>>>>>>
> > > >>>>>>> On Fri, May 4, 2018 at 7:21 PM, Richard Yu <
> > > >>>> yohan.richard...@gmail.com
> > > >>>>>>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Hi all, I would like to bump this thread since discussion in
> the
> > > >>>> KIP
> > > >>>>>>>> appears to be reaching its conclusion.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Mar 15, 2018 at 3:30 PM, Richard Yu <
> > > >>>>>> yohan.richard...@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi all,
> > > >>>>>>>>>
> > > >>>>>>>>> Since there does not seem to be too much discussion in
> > > >> KIP-266, I
> > > >>>>>> will
> > > >>>>>>> be
> > > >>>>>>>>> starting a voting thread.
> > > >>>>>>>>> Here is the link to KIP-266 for reference:
> > > >>>>>>>>>
> > > >>>>>>>>> https://cwiki.apache.org/confluence/pages/viewpage.
> > > >>>>>>>> action?pageId=75974886
> > > >>>>>>>>>
> > > >>>>>>>>> Recently, I have made some updates to the KIP. To reiterate,
> I
> > > >>>> have
> > > >>>>>>>>> included KafkaConsumer's commitSync,
> > > >>>>>>>>> poll, and committed in the KIP. (we will be adding to a
> > > >>>>>>> TimeoutException
> > > >>>>>>>>> to them as well, in a similar manner
> > > >>>>>>>>> to what we will be doing for position())
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>> Richard Yu
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> Ingénieur en informatique
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> -- Guozhang
> > > >>>>
> > > >>
> > >
> > >
> >
> >
> > --
> > "When the people fear their government, there is tyranny; when the
> > government fears the people, there is liberty." [Thomas Jefferson]
> >
>

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Jason Gustafson]

Perhaps one minute? That is the default used by the producer.

-Jason

On Wed, May 30, 2018 at 9:50 AM, Ismael Juma  wrote:

> Option 1 sounds good to me provided that we can come up with a good
> default. What would you suggest?
>
> Ismael
>
> On Wed, May 30, 2018 at 9:41 AM Jason Gustafson 
> wrote:
>
> > Hi Everyone,
> >
> > There remains some inconsistency in the timeout behavior of the consumer
> > APIs which do not accept a timeout. Some of them block forever (e.g.
> > position()) and some of them use request.timeout.ms (e.g.
> > parititonsFor()).
> > I think we'd probably all agree that blocking forever is not useful
> > behavior and using request.timeout.ms has always been a hack since it
> > controls a separate concern. I think there are basically two options to
> > address this:
> >
> > 1. We can add max.block.ms to match the producer and use it as the
> default
> > timeout when a timeout is not explicitly provided. This will fix the
> > indefinite blocking behavior and avoid conflating request.timeout.ms.
> > 2. We can deprecate the methods which don't accept a timeout.
> >
> > I'm leaning toward the first solution because I think we want to push
> users
> > to specifying timeouts through configuration rather than in code (Jay's
> > original argument). I think the overloads are still useful for advanced
> > usage (e.g. in kafka streams), but we should give users an easy option
> with
> > reasonable default behavior.
> >
> > If that sounds ok, I'd propose we add it to this KIP and fix it now. This
> > gives users an easy way to get the benefit of the improvements from this
> > KIP without changing any code.
> >
> > Thanks,
> > Jason
> >
> >
> >
> >
> > On Sun, May 13, 2018 at 7:58 PM, Richard Yu 
> > wrote:
> >
> > > Hi,
> > >
> > > With 3 binding votes and 6 non-binding, this KIP would be accepted.
> > >
> > > Thanks for participating.
> > >
> > > On Thu, May 10, 2018 at 2:35 AM, Edoardo Comar 
> > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > On 10 May 2018 at 10:29, zhenya Sun  wrote:
> > > >
> > > > > +1 non-binding
> > > > >
> > > > > > 在 2018年5月10日，下午5:19，Manikumar  写道：
> > > > > >
> > > > > > +1 (non-binding).
> > > > > > Thanks.
> > > > > >
> > > > > > On Thu, May 10, 2018 at 2:33 PM, Mickael Maison <
> > > > > mickael.mai...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> +1 (non binding)
> > > > > >> Thanks
> > > > > >>
> > > > > >> On Thu, May 10, 2018 at 9:39 AM, Rajini Sivaram <
> > > > > rajinisiva...@gmail.com>
> > > > > >> wrote:
> > > > > >>> Hi Richard, Thanks for the KIP.
> > > > > >>>
> > > > > >>> +1 (binding)
> > > > > >>>
> > > > > >>> Regards,
> > > > > >>>
> > > > > >>> Rajini
> > > > > >>>
> > > > > >>> On Wed, May 9, 2018 at 10:54 PM, Guozhang Wang <
> > wangg...@gmail.com
> > > >
> > > > > >> wrote:
> > > > > >>>
> > > > > >>>> +1 from me, thanks!
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> Guozhang
> > > > > >>>>
> > > > > >>>> On Wed, May 9, 2018 at 10:46 AM, Jason Gustafson <
> > > > ja...@confluent.io>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>> Thanks for the KIP, +1 (binding).
> > > > > >>>>>
> > > > > >>>>> One small correction: the KIP mentions that close() will be
> > > > > >> deprecated,
> > > > > >>>> but
> > > > > >>>>> we do not want to do this because it is needed by the
> Closeable
> > > > > >>>> interface.
> > > > > >>>>> We only want to deprecate close(long, TimeUnit) in favor of
> > > > > >>>>> close(Duration).
> > > > > >&

Re: [VOTE] KIP-281: ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User

[Jason Gustafson]

I'm going to call this vote since it has enough binding votes and it has
been open for a long time. The final tally:

Binding +4: me, Ismael, Rajini, and Guozhang
Non-binding +4: Ted, Moshe, Alex, and Dhruvil

There were no -1 votes.

Thanks to Alexander Dunayevsky for the KIP!

Thanks,
Jason

On Fri, May 11, 2018 at 8:55 AM, Guozhang Wang  wrote:

> +1 (binding), thanks!
>
>
> Guozhang
>
>
> On Thu, May 10, 2018 at 4:41 PM, Ismael Juma  wrote:
>
> > Thanks for the KIP, +1 (binding). A few suggestions:
> >
> > 1. We normally include the time unit in configs. Not sure if we do it for
> > command line parameters though, so can we please verify and make it
> > consistent?
> > 2. The KIP mentions --polling-loop-timeout and --timeout. Which is it?
> > 3. Can we include the description of the new parameter in the KIP? In the
> > PR it says "Consumer polling loop timeout", but I think this is a bit
> > unclear. What are we actually measuring here?
> >
> > Ismael
> >
> > On Mon, Apr 16, 2018 at 2:25 PM Alex Dunayevsky 
> > wrote:
> >
> > > Hello friends,
> > >
> > > Let's start the vote for KIP-281: ConsumerPerformance: Increase Polling
> > > Loop Timeout and Make It Reachable by the End User:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 281%3A+ConsumerPerformance%3A+Increase+Polling+Loop+Timeout+
> > and+Make+It+Reachable+by+the+End+User
> > >
> > > Thank you,
> > > Alexander Dunayevsky
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Jason Gustafson]

Thanks everyone for the feedback. I've updated the KIP and added KAFKA-6979.

-Jason

On Wed, May 30, 2018 at 3:50 PM, Guozhang Wang  wrote:

> Thanks Jason. I'm in favor of option 1 as well.
>
> On Wed, May 30, 2018 at 1:37 PM, Bill Bejeck  wrote:
>
> > For what it's worth I'm +1 on Option 1 and the default value for the
> > timeout.
> >
> > In addition to reasons outlined above by Jason, I think it will help to
> > reason about consumer behavior (with respect to blocking) having the
> > configuration and default value aligned with the producer.
> >
> > -Bill
> >
> > On Wed, May 30, 2018 at 3:43 PM, Ismael Juma  wrote:
> >
> > > Sounds good to me,
> > >
> > > On Wed, May 30, 2018 at 12:40 PM Jason Gustafson 
> > > wrote:
> > >
> > > > Perhaps one minute? That is the default used by the producer.
> > > >
> > > > -Jason
> > > >
> > > > On Wed, May 30, 2018 at 9:50 AM, Ismael Juma 
> > wrote:
> > > >
> > > > > Option 1 sounds good to me provided that we can come up with a good
> > > > > default. What would you suggest?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Wed, May 30, 2018 at 9:41 AM Jason Gustafson <
> ja...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Hi Everyone,
> > > > > >
> > > > > > There remains some inconsistency in the timeout behavior of the
> > > > consumer
> > > > > > APIs which do not accept a timeout. Some of them block forever
> > (e.g.
> > > > > > position()) and some of them use request.timeout.ms (e.g.
> > > > > > parititonsFor()).
> > > > > > I think we'd probably all agree that blocking forever is not
> useful
> > > > > > behavior and using request.timeout.ms has always been a hack
> since
> > > it
> > > > > > controls a separate concern. I think there are basically two
> > options
> > > to
> > > > > > address this:
> > > > > >
> > > > > > 1. We can add max.block.ms to match the producer and use it as
> the
> > > > > default
> > > > > > timeout when a timeout is not explicitly provided. This will fix
> > the
> > > > > > indefinite blocking behavior and avoid conflating
> > request.timeout.ms
> > > .
> > > > > > 2. We can deprecate the methods which don't accept a timeout.
> > > > > >
> > > > > > I'm leaning toward the first solution because I think we want to
> > push
> > > > > users
> > > > > > to specifying timeouts through configuration rather than in code
> > > (Jay's
> > > > > > original argument). I think the overloads are still useful for
> > > advanced
> > > > > > usage (e.g. in kafka streams), but we should give users an easy
> > > option
> > > > > with
> > > > > > reasonable default behavior.
> > > > > >
> > > > > > If that sounds ok, I'd propose we add it to this KIP and fix it
> > now.
> > > > This
> > > > > > gives users an easy way to get the benefit of the improvements
> from
> > > > this
> > > > > > KIP without changing any code.
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, May 13, 2018 at 7:58 PM, Richard Yu <
> > > > yohan.richard...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > With 3 binding votes and 6 non-binding, this KIP would be
> > accepted.
> > > > > > >
> > > > > > > Thanks for participating.
> > > > > > >
> > > > > > > On Thu, May 10, 2018 at 2:35 AM, Edoardo Comar <
> > edoco...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > On 10 May 2018 at 10:29, zhenya Sun  wrote:
> > > > &g

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Jason Gustafson]

Hey All,

One more minor follow-up. As I was reviewing the change mentioned above, I
felt the name `max.block.ms` was a little bit misleading since it only
applies to methods which do not have an explicit timeout. A clearer name
given its usage might be `default.timeout.ms`. It is the default timeout
for any blocking API which does not have a timeout. I'm leaning toward
using this name since the current one seems likely to cause confusion. Any
thoughts?

Thanks,
Jason


On Thu, May 31, 2018 at 6:09 PM, Dong Lin  wrote:

> Thanks for the KIP! I am in favor of the option 1.
>
> +1 as well.
>
> On Thu, May 31, 2018 at 6:00 PM, Jason Gustafson 
> wrote:
>
> > Thanks everyone for the feedback. I've updated the KIP and added
> > KAFKA-6979.
> >
> > -Jason
> >
> > On Wed, May 30, 2018 at 3:50 PM, Guozhang Wang 
> wrote:
> >
> > > Thanks Jason. I'm in favor of option 1 as well.
> > >
> > > On Wed, May 30, 2018 at 1:37 PM, Bill Bejeck 
> wrote:
> > >
> > > > For what it's worth I'm +1 on Option 1 and the default value for the
> > > > timeout.
> > > >
> > > > In addition to reasons outlined above by Jason, I think it will help
> to
> > > > reason about consumer behavior (with respect to blocking) having the
> > > > configuration and default value aligned with the producer.
> > > >
> > > > -Bill
> > > >
> > > > On Wed, May 30, 2018 at 3:43 PM, Ismael Juma 
> > wrote:
> > > >
> > > > > Sounds good to me,
> > > > >
> > > > > On Wed, May 30, 2018 at 12:40 PM Jason Gustafson <
> ja...@confluent.io
> > >
> > > > > wrote:
> > > > >
> > > > > > Perhaps one minute? That is the default used by the producer.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, May 30, 2018 at 9:50 AM, Ismael Juma 
> > > > wrote:
> > > > > >
> > > > > > > Option 1 sounds good to me provided that we can come up with a
> > good
> > > > > > > default. What would you suggest?
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Wed, May 30, 2018 at 9:41 AM Jason Gustafson <
> > > ja...@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Everyone,
> > > > > > > >
> > > > > > > > There remains some inconsistency in the timeout behavior of
> the
> > > > > > consumer
> > > > > > > > APIs which do not accept a timeout. Some of them block
> forever
> > > > (e.g.
> > > > > > > > position()) and some of them use request.timeout.ms (e.g.
> > > > > > > > parititonsFor()).
> > > > > > > > I think we'd probably all agree that blocking forever is not
> > > useful
> > > > > > > > behavior and using request.timeout.ms has always been a hack
> > > since
> > > > > it
> > > > > > > > controls a separate concern. I think there are basically two
> > > > options
> > > > > to
> > > > > > > > address this:
> > > > > > > >
> > > > > > > > 1. We can add max.block.ms to match the producer and use it
> as
> > > the
> > > > > > > default
> > > > > > > > timeout when a timeout is not explicitly provided. This will
> > fix
> > > > the
> > > > > > > > indefinite blocking behavior and avoid conflating
> > > > request.timeout.ms
> > > > > .
> > > > > > > > 2. We can deprecate the methods which don't accept a timeout.
> > > > > > > >
> > > > > > > > I'm leaning toward the first solution because I think we want
> > to
> > > > push
> > > > > > > users
> > > > > > > > to specifying timeouts through configuration rather than in
> > code
> > > > > (Jay's
> > > > > > > > original argument). I think the overloads are still useful
> for
> > > > > advanced
> > > > > > > > usage (e.g. in kafka streams), but we should 

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Jason Gustafson]

Thanks for the comments. I'm not sure I understand why we want to avoid the
term "timeout." Semantically, that's what it is. If we don't want another
timeout config, we could avoid it and hard-code a reasonable default or try
to wrap the behavior into one of the other timeouts (which is sort of what
we were already doing with request.timeout.ms). But I'm not too sure
calling it something else addresses the problem.

-Jason

On Tue, Jun 5, 2018 at 2:59 PM, Dhruvil Shah  wrote:

> I agree that using `default.timeout.ms` could cause confusion since we
> already have other timeout configurations in the consumer.
>
> +1 for using `default.block.ms`.
>
> Thanks,
> Dhruvil
>
> On Tue, Jun 5, 2018 at 11:48 AM, Bill Bejeck  wrote:
>
> > Hi Jason,
> >
> > At first, I thought the same name between the producer and the consumer
> was
> > ideal.
> >
> > But your comment makes me realize consistent names with different
> semantics
> > is even more confusing.
> >
> > I'm +1 for not using `max.block.ms`.  I like Guozhang's suggestion of `
> > default.block.ms` for the name.
> >
> > Thanks,
> > Bill
> >
> > On Tue, Jun 5, 2018 at 1:33 PM, Guozhang Wang 
> wrote:
> >
> > > Hi Jason,
> > >
> > > Yeah I agree that "max.block.ms" makes people thinking of the
> producer's
> > > config with the same name, but their semantics are different.
> > >
> > > On the other hand, I'm a bit concerned with the reusing of the term
> > > `timeout` as we already have `session.timeout.ms` and `
> > request.timeout.ms`
> > > in ConsumerConfig.. How about using the name `default.api.block.ms` or
> > > simply `default.block.ms`?
> > >
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Jun 5, 2018 at 8:57 AM, Jason Gustafson 
> > > wrote:
> > >
> > > > Hey All,
> > > >
> > > > One more minor follow-up. As I was reviewing the change mentioned
> > above,
> > > I
> > > > felt the name `max.block.ms` was a little bit misleading since it
> only
> > > > applies to methods which do not have an explicit timeout. A clearer
> > name
> > > > given its usage might be `default.timeout.ms`. It is the default
> > timeout
> > > > for any blocking API which does not have a timeout. I'm leaning
> toward
> > > > using this name since the current one seems likely to cause
> confusion.
> > > Any
> > > > thoughts?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > > On Thu, May 31, 2018 at 6:09 PM, Dong Lin 
> wrote:
> > > >
> > > > > Thanks for the KIP! I am in favor of the option 1.
> > > > >
> > > > > +1 as well.
> > > > >
> > > > > On Thu, May 31, 2018 at 6:00 PM, Jason Gustafson <
> ja...@confluent.io
> > >
> > > > > wrote:
> > > > >
> > > > > > Thanks everyone for the feedback. I've updated the KIP and added
> > > > > > KAFKA-6979.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, May 30, 2018 at 3:50 PM, Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Thanks Jason. I'm in favor of option 1 as well.
> > > > > > >
> > > > > > > On Wed, May 30, 2018 at 1:37 PM, Bill Bejeck <
> bbej...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > For what it's worth I'm +1 on Option 1 and the default value
> > for
> > > > the
> > > > > > > > timeout.
> > > > > > > >
> > > > > > > > In addition to reasons outlined above by Jason, I think it
> will
> > > > help
> > > > > to
> > > > > > > > reason about consumer behavior (with respect to blocking)
> > having
> > > > the
> > > > > > > > configuration and default value aligned with the producer.
> > > > > > > >
> > > > > > > > -Bill
> > > > > > > >
> > > > > > > > On Wed, May 30, 2018 at 3:43 PM, Ismael Juma <
> > ism...@juma.me.uk>
> >

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Jason Gustafson]

t; > > > >
> > > > > Looking at some existing samples:
> > > > > In tests/kafkatest/tests/connect/templates/connect-distributed.
> > > properties
> > > > ,
> > > > > we have:
> > > > >   request.timeout.ms=3
> > > > >
> > > > > Isn't the above formula putting an upper bound 15000 for the RPC
> > > timeout
> > > > ?
> > > >
> > > > Correct.  It would put a 15 second default on the RPC timeout in this
> > > > case.  If that's too short, the user has the option to change it.
> > > >
> > > > If we feel that 15 seconds is too short, we could put a floor of 30
> > > > seconds or whatever on the RPC timeout, instead of 1 second.
> > > >
> > > > >
> > > > > By fixed duration, I meant something like
> > > > >   request.timeout.ms + 15000
> > > >
> > > > The RPC timeout should be shorter than the request timeout, so that
> we
> > > get
> > > > multiple tries if the RPC hangs due to network issues.  It should
> not be
> > > > longer.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Tue, Jun 5, 2018 at 4:27 PM, Colin McCabe 
> > > wrote:
> > > > >
> > > > > > I don't think it can be fixed.  The RPC duration is something
> that
> > > you
> > > > > > might reasonably want to tune.  For example, if it's too low, you
> > > > might not
> > > > > > be able to make progress at all on a heavily loaded server.
> > > > > >
> > > > > > We could probably come up with a good default, however.
> > > > rpc.timeout.ms
> > > > > > could be set to something like
> > > > > > max(1000, 0.5 * request.timeout.ms)
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 5, 2018, at 16:21, Ted Yu wrote:
> > > > > > > bq. we must make the API timeout longer than the RPC timeout
> > > > > > >
> > > > > > > I agree with the above.
> > > > > > >
> > > > > > > How about adding a fixed duration on top of request.timeout.ms
> ?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Tue, Jun 5, 2018 at 4:18 PM, Colin McCabe <
> cmcc...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > As Jason noted earlier, request.timeout.ms controls
> something
> > > > > > different:
> > > > > > > > the amount of time we're willing to wait for an RPC to
> complete.
> > > > > > > >
> > > > > > > > Empirically, RPCs sometimes hang for a long time.  If the API
> > > > timeout
> > > > > > is
> > > > > > > > the same as the RPC timeout, we are not robust against this
> > > failure
> > > > > > > > condition.  The whole call fails rather than trying another
> > > server
> > > > or
> > > > > > a new
> > > > > > > > socket.
> > > > > > > >
> > > > > > > > In order to fix this, we must make the API timeout longer
> than
> > > the
> > > > RPC
> > > > > > > > timeout.
> > > > > > > >
> > > > > > > > However, we have a lot of users who have been trained to use
> > > > > > > > request.timeout.ms to make their clients time out earlier.
> If
> > > we
> > > > > > > > introduce a new config to do this instead, it's kind of a
> > > breaking
> > > > > > change
> > > > > > > > for them.  Perhaps we should go the other direction and
> create a
> > > > new
> > > > > > > > configuration like rpc.timeout.ms to do what
> request.timeout.ms
> > > is
> > > > > > doing
> > > > > > > > now.  Then request.timeout.ms can become what users already
> > > think
> > > > it
> &

[DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jason Gustafson]

Hey All,

I wrote up a KIP to handle one more edge case in the replication protocol
and to support better handling of truncation in the consumer when unclean
leader election is enabled. Let me know what you think.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-320%3A+Allow+fetchers+to+detect+and+handle+log+truncation

Thanks to Anna Povzner and Dong Lin for initial feedback.

Thanks,
Jason

Re: Request permission to assign JIRA

[Jason Gustafson]

Added. Thanks for contributing!

-Jason

On Mon, Jun 25, 2018 at 9:27 AM, lambdaliu(刘少波) 
wrote:

> Hi Team,
>
> I am trying to claim a bug in Jira, Could you please help me gain the
> required permissions.
> my JIRA usernane is lambdaliu.
>
> thanks,
> lambdaliu.
>

Re: [DISCUSS] KIP-308: Support dynamic update of max.connections.per.ip/max.connections.per.ip.overrides configs

[Jason Gustafson]

Hey Manikumar,

Thanks for the KIP. This seems useful.

-Jason

On Thu, Jun 21, 2018 at 11:38 PM, Manikumar 
wrote:

> Hi all,
>
> I have created a KIP to add support for dynamic update of
> max.connections.per.ip/max.connections.per.ip.overrides configs
>
> *https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=85474993
>  >*
>
> Any feedback is appreciated.
>
> Thanks
>

Re: Request access to create KIP

[Jason Gustafson]

Done. Thanks for contributing!

-Jason

On Mon, Jun 25, 2018 at 12:49 PM, Yishun Guan  wrote:

> Hi, could someone give me access to create KIP? Thanks! - Yishun
>
> On Mon, Jun 25, 2018, 10:44 AM Yishun Guan  wrote:
>
> > Hi, my wiki id is gyishun. Thanks! - Yishun
> >
>

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jason Gustafson]

Hey Dong,

Thanks for the comments.

- The KIP says that, with auto.offset.reset="closest", timestamp is used to
> find offset if truncation offset is unknown. It seems that if consumer
> knows the timestamp of the last message, then the consumer should also know
> the (offset, leaderEpoch) of the last message which can then be used for
> find the truncation offset. Can you explain why truncation offset is
> unknown in this case?


The intent of the new reset policy is to automatically locate the closest
offset within the limits of Kafka log semantics. Unlike replicas, consumers
do not know the full history of leader epochs that have been previously
read. In some scenarios, they may not be able to precisely find the offset
where the log diverged after a sequence of unclean leader elections (see
KIP-279 for more detail). It seemed unfortunate in these cases to have to
resort to the coarse-grained resetting using either the earliest or latest
offset. Using the timestamp, we can find a more accurate reset point and
minimize the amount of loss or duplication.

- How does consumer differentiates between "Offset out of rnage (too low)"
> and "Offset out of range (unknown truncation offset)", i.e. the two columns
> in table provided in the KIP?


We know when an offset is too low because we have the start offset of the
log from the fetch response. Following this KIP, that should really be the
only time we get an OutOfRange error (other than buggy application code).
The other two cases are distinguished based on whether we are able to find
the right offset of divergence.

- It is probably a typo. Maybe fix "This is not the last The" in the
> Proposed Section.


Thanks. Magnus noticed this too and I fixed it earlier this morning. Good
to know who's actually reading the proposal!

-Jason



On Tue, Jun 26, 2018 at 11:09 AM, Dong Lin  wrote:

> Hey Jason,
>
> Thanks for the KIP! It is pretty useful.
>
> At high level the new set of reset policies may be a bit complicated and
> confusing to users. I am wondering whether we can simplify it. A few
> questions below:
>
> - The KIP says that, with auto.offset.reset="closest", timestamp is used to
> find offset if truncation offset is unknown. It seems that if consumer
> knows the timestamp of the last message, then the consumer should also know
> the (offset, leaderEpoch) of the last message which can then be used for
> find the truncation offset. Can you explain why truncation offset is
> unknown in this case?
>
> - How does consumer differentiates between "Offset out of rnage (too low)"
> and "Offset out of range (unknown truncation offset)", i.e. the two columns
> in table provided in the KIP?
>
> - It is probably a typo. Maybe fix "This is not the last The" in the
> Proposed Section.
>
>
> Thanks,
> Dong
>
> On Mon, Jun 25, 2018 at 9:17 AM, Jason Gustafson 
> wrote:
>
> > Hey All,
> >
> > I wrote up a KIP to handle one more edge case in the replication protocol
> > and to support better handling of truncation in the consumer when unclean
> > leader election is enabled. Let me know what you think.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-320%3A
> > +Allow+fetchers+to+detect+and+handle+log+truncation
> >
> > Thanks to Anna Povzner and Dong Lin for initial feedback.
> >
> > Thanks,
> > Jason
> >
>

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jason Gustafson]

The other thing I forgot to mention is that resetting the offset using the
leader epoch is only available with the latest message format. By
supporting reset by timestamp, users on the v1 format can still get some
benefit from this KIP.

-Jason

On Tue, Jun 26, 2018 at 11:47 AM, Jason Gustafson 
wrote:

> Hey Dong,
>
> Thanks for the comments.
>
> - The KIP says that, with auto.offset.reset="closest", timestamp is used to
>> find offset if truncation offset is unknown. It seems that if consumer
>> knows the timestamp of the last message, then the consumer should also
>> know
>> the (offset, leaderEpoch) of the last message which can then be used for
>> find the truncation offset. Can you explain why truncation offset is
>> unknown in this case?
>
>
> The intent of the new reset policy is to automatically locate the closest
> offset within the limits of Kafka log semantics. Unlike replicas,
> consumers do not know the full history of leader epochs that have been
> previously read. In some scenarios, they may not be able to precisely find
> the offset where the log diverged after a sequence of unclean leader
> elections (see KIP-279 for more detail). It seemed unfortunate in these
> cases to have to resort to the coarse-grained resetting using either the
> earliest or latest offset. Using the timestamp, we can find a more accurate
> reset point and minimize the amount of loss or duplication.
>
> - How does consumer differentiates between "Offset out of rnage (too low)"
>> and "Offset out of range (unknown truncation offset)", i.e. the two
>> columns
>> in table provided in the KIP?
>
>
> We know when an offset is too low because we have the start offset of the
> log from the fetch response. Following this KIP, that should really be the
> only time we get an OutOfRange error (other than buggy application code).
> The other two cases are distinguished based on whether we are able to find
> the right offset of divergence.
>
> - It is probably a typo. Maybe fix "This is not the last The" in the
>> Proposed Section.
>
>
> Thanks. Magnus noticed this too and I fixed it earlier this morning. Good
> to know who's actually reading the proposal!
>
> -Jason
>
>
>
> On Tue, Jun 26, 2018 at 11:09 AM, Dong Lin  wrote:
>
>> Hey Jason,
>>
>> Thanks for the KIP! It is pretty useful.
>>
>> At high level the new set of reset policies may be a bit complicated and
>> confusing to users. I am wondering whether we can simplify it. A few
>> questions below:
>>
>> - The KIP says that, with auto.offset.reset="closest", timestamp is used
>> to
>> find offset if truncation offset is unknown. It seems that if consumer
>> knows the timestamp of the last message, then the consumer should also
>> know
>> the (offset, leaderEpoch) of the last message which can then be used for
>> find the truncation offset. Can you explain why truncation offset is
>> unknown in this case?
>>
>> - How does consumer differentiates between "Offset out of rnage (too low)"
>> and "Offset out of range (unknown truncation offset)", i.e. the two
>> columns
>> in table provided in the KIP?
>>
>> - It is probably a typo. Maybe fix "This is not the last The" in the
>> Proposed Section.
>>
>>
>> Thanks,
>> Dong
>>
>> On Mon, Jun 25, 2018 at 9:17 AM, Jason Gustafson 
>> wrote:
>>
>> > Hey All,
>> >
>> > I wrote up a KIP to handle one more edge case in the replication
>> protocol
>> > and to support better handling of truncation in the consumer when
>> unclean
>> > leader election is enabled. Let me know what you think.
>> >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-320%3A
>> > +Allow+fetchers+to+detect+and+handle+log+truncation
>> >
>> > Thanks to Anna Povzner and Dong Lin for initial feedback.
>> >
>> > Thanks,
>> > Jason
>> >
>>
>
>

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

[Jason Gustafson]

Hey Dong,

Sorry for being slow to catch up to this.

I think the benefit of the sanity check seems a little dubious in the first
place. We detect garbage at the end of the index file, but that's about it.
Is there any reason to think that corruption is more likely to occur there
or any other reason to think this check is still beneficial for flushed
data? I assume we did the check because we presumed it was cheap, but
perhaps the cost is adding up as the number of partitions grows. How much
does startup time improve if we skip the sanity check for data earlier than
the recovery point? Does the lazy loading itself give some additional
benefit beyond skipping the sanity check? As Jay mentions above, the sanity
checks seem strictly speaking optional. We don't bother checking the
segments themselves for example.

Thanks,
Jason




It probably still makes sense for segments beyond the recovery point

On Wed, Mar 21, 2018 at 9:59 PM, Dong Lin  wrote:

> Hey Jay,
>
> Yeah our existing sanity check only read the last entry in the index files.
> I must have miscommunicated if I previously said it was reading the full
> index. Broker appears to be spending a lot of time just to read the last
> entry of index files for every log segment. This is probably because OS
> will load a chunk of data that is much larger than the entry itself from
> disk to page cache. This KIP tries to make this part of operation lazy. I
> guess you are suggesting that we should just make the lazy loading the
> default behavior?
>
> Yes we currently require manual intervention if the log file is corrupted,
> i.e. if two messages with the same offset are appended to the disk
> (KAFKA-6488). The sanity check on broker startup is a bit different since
> it deals with the corruption of index files (e.g. offset index, time index
> and snapshot files) instead of the log data. In this case if index files
> are corrupted broker will automatically recover it by rebuilding the index
> files using data in the log files, without requiring manual intervention.
> Thus the design question is whether this should be done before broker can
> become leader for any partitions -- there is tradeoff between broker
> startup time and risk of delaying user requests if broker need to rebuild
> index files when it is already leader. I prefer lazy loading to reduce
> broker startup time. Not sure what are the feedback from the community on
> this issue.
>
> Thanks,
> Dong
>
>
> On Wed, Mar 21, 2018 at 7:36 AM, Jay Kreps  wrote:
>
> > Hey Dong,
> >
> > Makes total sense. What I'm saying is I don't think that the sanity check
> > is part of any formal guarantee we provide. It is true that corruption of
> > data flushed to disk will be a potential problem, but I don't think the
> > sanity check solves that it just has a couple heuristics to help detect
> > certain possible instances of it, right? In general I think our
> assumption
> > has been that flushed data doesn't disappear or get corrupted and if it
> > does you need to manually intervene. I don't think people want to
> configure
> > things at this level so what I was suggesting was understanding why the
> > sanity check is slow and trying to avoid that rather than making it
> > configurable. I think you mentioned it was reading the full index into
> > memory. Based on the performance you describe this could be true, but it
> > definitely should not be reading anything but the last entry in the index
> > so that would be a bug. That read also happens in sanityCheck() only in
> the
> > time-based index right? In the offset index we do the same read but it
> > happens in initialization. If that read is the slow thing it might make
> > sense to try to remove it or make it lazy in both cases. If it is some
> > other part of the code then (e.g. the size check) then that may be able
> to
> > be avoided entirely (I think by the time we sanity check we already know
> > the file size from the mapping...). That was what I meant by doing some
> > data driven analysis. Maybe a quick run with hprof would help determine
> the
> > root cause of why sanityCheck is slow?
> >
> > -Jay
> >
> > On Tue, Mar 20, 2018 at 12:13 AM Dong Lin  wrote:
> >
> > > Hey Jay,
> > >
> > > Thanks for your comments!
> > >
> > > Yeah recovery is different from the sanity check. They are correlated
> in
> > > the sense that there may still be corrupted index files even after
> clean
> > > broker shutdown. And in that case if we delay the sanity check then we
> > may
> > > delay the log recovery. The main goal of this KIP is to optimize the
> > sanity
> > > check related work so that it does not delay the broker startup much.
> > >
> > > The KIP mentioned that the sanity check is done using log recovery
> > > background thread. The name "recovery" is mentioned mainly because the
> > > background thread number is determined using the existing
> > > config num.recovery.threads.per.data.dir. I have updated the KIP to
> make
> > > this less confusing.
> > >

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

[Jason Gustafson]

Hey Dong,


So the main concern with the above approach is that, if for any reason the
> index files of inactive segment is deleted or corrupted, the broker will
> halt if there is only one log directory. This is different from the
> existing behavior where the broker will rebuild the index for this inactive
> segment before it can accept any request from consumer. Though we don't
> have provide guarantee for segments already flushed to disk, this still
> seems like a change in behavior for user. Maybe we don't have to worry
> about this if we decide it is very rare, e.g. it happens only when there is
> disk error or when there is human error.


I think we should probably still handle the case when an index file is
missing during startup? But considering how weak the sanity check is, it
seems fine to skip it.  Also, could we just make this change without a KIP?
Adding a config to enable a wimpy sanity check seems unnecessary.

One scenario that does come up with users is actual segment corruption,
which is only detected by consumers that are validating CRCs. To fix it, we
have to manually delete the segments and force re-replication. It would be
helpful to have a config to enable deep checking on startup for particular
topics or partitions. This could also just be a separate tool though
("kafka-fsck" or something).

Thinking longer term, I think we need a more systematic approach to dealing
with corruption, not just in index files, but in the segments as well. It
might be nice, for example, if the consumer had a way to hint the broker
that a particular offset is corrupt. The leader might then demote itself,
for example, and try to repair. Lots to think through though.

-Jason




On Wed, Jun 27, 2018 at 12:29 AM, Dong Lin  wrote:

> So the main concern with the above approach is that, if for any reason the
> index files of inactive segment is deleted or corrupted, the broker will
> halt if there is only one log directory. This is different from the
> existing behavior where the broker will rebuild the index for this inactive
> segment before it can accept any request from consumer. Though we don't
> have provide guarantee for segments already flushed to disk, this still
> seems like a change in behavior for user. Maybe we don't have to worry
> about this if we decide it is very rare, e.g. it happens only when there is
> disk error or when there is human error.
>
>
>
> On Wed, Jun 27, 2018 at 12:04 AM, Dong Lin  wrote:
>
> > Hey Jason,
> >
> > Thanks for the comment!
> >
> > Your comment reminded me to read through Jay's comments and my reply
> > again. It seems that I probably have not captured idea of Jay's comment
> > that says sanity check is not part of any formal guarantee we provide. I
> > probably should have thought about this comment more. Let me reply to
> both
> > yours and Jay's comment and see if I can understand you better.
> >
> > Here are some clarifications:
> > - KIP does not intend to optimize recovery. It aims to optimize the the
> > sanity check when there is clean shutdown.
> > - Sanity check only read the last entry of the index rather than the full
> > index
> > - We have already done data driven investigation though it is not done
> > using hprof or strace. The resulting rolling bounce time is acceptable
> now.
> > If it appears to be an issue e.g. after more data then we may need to
> > revisit this with more data driven investigation
> >
> > I agree with the following comments:
> > - We should optimize the default behavior instead of adding a new config.
> > - sanity check of the segments before recovery offset is not part of any
> > formal guarantee and thus we probably can just skip it.
> >
> > So we are all leaning towards skipping the sanity check of all segments
> > before the recovery offset. This solution would be pretty straightforward
> > to understand and implement. And I am sure it will give us all the
> benefits
> > that this KIP intends to achieve. Here is only one question to double
> check:
> >
> > If consumer fetches from an inactive segment, broker will just use the
> > index of that inactive segment. If anything goes wrong, e.g. the index
> file
> > is corrupted or the index file does not exist, then the broker will just
> > consider it as IOException, mark the disk and the partitions on the disk
> > offline and respond KafkaStorageException to consumer. Does this sound
> OK?
> > One alternative solution is to let broker rebuild index. But this
> > alternative solution is inconsistent with the idea that "sanity check is
> not
> > part of any formal guarantee" and it may tie up all request

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jason Gustafson]

@Dong

Those are fair points. Both approaches require some fuzziness to reset the
offset in these pathological scenarios and we cannot guarantee
at-least-once delivery either way unless we have the full history of leader
epochs that were consumed. The KIP-101 logic may actually be more accurate
than using timestamps because it does not depend on the messages which are
written after the unclean leader election. The case we're talking about
should be extremely rare in practice anyway. I also agree that we may not
want to add new machinery if it only helps the old message format. Ok,
let's go ahead and drop the timestamp.

@Guozhang

* My current understanding is that, with unclean leader election turned on,
> exactly-once is out of the window since we cannot guarantee that all
> committed message markers will not be lost. And hence there is no need to
> have special handling logic for LOG_TRUNCATED or OOR error codes with
> read.committed turned on. Is that right?


Yes, that's right. EoS and unclean leader election don't mix well. It may
be worth considering separately whether we should try to reconcile the
transaction log following an unclean leader election. At least we may be
able to prevent dangling transactions from blocking consumers. This KIP
does not address this problem.

* MINOR: "if the epoch is greater than the minimum expected epoch, that the
> new epoch does not begin at an earlier offset than the fetch offset.  In
> the latter case, the leader can respond with a new LOG_TRUNCATION error
> code" should it be "does not begin at a later offset than the fetch
> offset"?


I think the comment is correct, though the phrasing may be confusing. We
know truncation has occurred if there exists a larger epoch with a starting
offset that is lower than the fetch offset. Let me try to rephrase this.

Thanks,
Jason

On Wed, Jun 27, 2018 at 9:17 AM, Guozhang Wang  wrote:

> Jason, thanks for the KIP. A few comments:
>
> * I think Dong's question about whether to use timestamp-based approach
> v.s. start-offset-of-first-larger-epoch is valid; more specifically, with
> timestamp-based approach we may still be reseting to an offset falling into
> the truncated interval, and hence we may still miss some data, i.e. not
> guaranteeing at-least-once still. With the
> start-offset-of-first-larger-epoch, I'm not sure if it will guarantee no
> valid data is missed when we have consecutive log truncations (maybe we
> need to look back into details of KIP-101 to figure it out). If the latter
> can indeed guarantee at least once, we could consider using that approach.
>
> * My current understanding is that, with unclean leader election turned on,
> exactly-once is out of the window since we cannot guarantee that all
> committed message markers will not be lost. And hence there is no need to
> have special handling logic for LOG_TRUNCATED or OOR error codes with
> read.committed turned on. Is that right?
>
> * MINOR: "if the epoch is greater than the minimum expected epoch, that the
> new epoch does not begin at an earlier offset than the fetch offset.  In
> the latter case, the leader can respond with a new LOG_TRUNCATION error
> code" should it be "does not begin at a later offset than the fetch
> offset"?
>
>
>
> Guozhang
>
>
> On Tue, Jun 26, 2018 at 6:51 PM, Dong Lin  wrote:
>
> > Hey Jason,
> >
> > Thanks for the explanation.
> >
> > Please correct me if this is wrong. The "unknown truncation offset"
> > scenario happens when consumer does not have the full leaderEpoch ->
> offset
> > mapping. In this case we can still use the KIP-101-based approach to
> > truncate offset to "start offset of the first Leader Epoch larger than
> last
> > epoch of the consumer" but it may be inaccurate. So the KIP chooses to
> use
> > the timestamp-based approach which is also best-effort.
> >
> > If this understanding is correct, for "closest" offset reset policy and
> > "unknown truncation offset" scenario, I am wondering whether it maybe
> > better to replace timestamp-based approach with KIP-101 based approach.
> In
> > comparison to timestamp-based approach, the KIP-101-based approach seems
> to
> > simplify the API a bit since user does not need to understand timestamp.
> > Similar to the timestamp-based approach, both approaches are best-effort
> > and do not guarantee that consumer can consume all messages. It is not
> like
> > KIP-279 which guarantees that follower broker can consume all messages
> from
> > the leader.
> >
> > Then it seems that the remaining difference is mostly about accuracy,
> i.e.
> > how much message will be duplicated or missed in the "unknown truncation
> > offset" scenario. Not sure either one is clearly better than the other.
> > Note that there are two scenarios mentioned in KIP-279 which are not
> > addressed by KIP-101. Both scenarios require quick leadership change
> > between brokers, which seems to suggest that the offset based obtained
> > by "start
> > offset of the first Leader Epoch larger

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jason Gustafson]

Thanks for the feedback. I've updated the KIP. Specifically I removed the
"closest" reset option and the proposal to reset by timestamp when the
precise truncation point cannot be determined. Instead, I proposed that we
always reset using the nearest epoch when a reset policy is defined (either
"earliest" or "latest"). Does that sound reasonable?

One thing I am still debating is whether it would be better to have a
separate API to find the closest offset using the leader epoch. In the
current KIP, I suggested to piggyback this information on an exception, but
I'm beginning to think it would be better not to hide the lookup. It is
awkward to implement since it means delaying the exception and the API may
actually be useful when customizing reset logic if no auto reset policy is
defined. I was thinking we can add an API like the following:

Map
offsetsForLeaderEpochs(Map epochsToSearch)

Thoughts?

-Jason




On Wed, Jun 27, 2018 at 11:12 AM, Jason Gustafson 
wrote:

> @Dong
>
> Those are fair points. Both approaches require some fuzziness to reset the
> offset in these pathological scenarios and we cannot guarantee
> at-least-once delivery either way unless we have the full history of leader
> epochs that were consumed. The KIP-101 logic may actually be more accurate
> than using timestamps because it does not depend on the messages which are
> written after the unclean leader election. The case we're talking about
> should be extremely rare in practice anyway. I also agree that we may not
> want to add new machinery if it only helps the old message format. Ok,
> let's go ahead and drop the timestamp.
>
> @Guozhang
>
> * My current understanding is that, with unclean leader election turned on,
>> exactly-once is out of the window since we cannot guarantee that all
>> committed message markers will not be lost. And hence there is no need to
>> have special handling logic for LOG_TRUNCATED or OOR error codes with
>> read.committed turned on. Is that right?
>
>
> Yes, that's right. EoS and unclean leader election don't mix well. It may
> be worth considering separately whether we should try to reconcile the
> transaction log following an unclean leader election. At least we may be
> able to prevent dangling transactions from blocking consumers. This KIP
> does not address this problem.
>
> * MINOR: "if the epoch is greater than the minimum expected epoch, that the
>> new epoch does not begin at an earlier offset than the fetch offset.  In
>> the latter case, the leader can respond with a new LOG_TRUNCATION error
>> code" should it be "does not begin at a later offset than the fetch
>> offset"?
>
>
> I think the comment is correct, though the phrasing may be confusing. We
> know truncation has occurred if there exists a larger epoch with a starting
> offset that is lower than the fetch offset. Let me try to rephrase this.
>
> Thanks,
> Jason
>
> On Wed, Jun 27, 2018 at 9:17 AM, Guozhang Wang  wrote:
>
>> Jason, thanks for the KIP. A few comments:
>>
>> * I think Dong's question about whether to use timestamp-based approach
>> v.s. start-offset-of-first-larger-epoch is valid; more specifically, with
>> timestamp-based approach we may still be reseting to an offset falling
>> into
>> the truncated interval, and hence we may still miss some data, i.e. not
>> guaranteeing at-least-once still. With the
>> start-offset-of-first-larger-epoch, I'm not sure if it will guarantee no
>> valid data is missed when we have consecutive log truncations (maybe we
>> need to look back into details of KIP-101 to figure it out). If the latter
>> can indeed guarantee at least once, we could consider using that approach.
>>
>> * My current understanding is that, with unclean leader election turned
>> on,
>> exactly-once is out of the window since we cannot guarantee that all
>> committed message markers will not be lost. And hence there is no need to
>> have special handling logic for LOG_TRUNCATED or OOR error codes with
>> read.committed turned on. Is that right?
>>
>> * MINOR: "if the epoch is greater than the minimum expected epoch, that
>> the
>> new epoch does not begin at an earlier offset than the fetch offset.  In
>> the latter case, the leader can respond with a new LOG_TRUNCATION error
>> code" should it be "does not begin at a later offset than the fetch
>> offset"?
>>
>>
>>
>> Guozhang
>>
>>
>> On Tue, Jun 26, 2018 at 6:51 PM, Dong Lin  wrote:
>>
>> > Hey Jason,
>> >
>> > Thanks for the explanation.
>> >
>> > Please

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jason Gustafson]

Hey Guozhang,

That's fair. In fact, perhaps we do not need this API at all. We already
have the new seek() in this KIP which can do the lookup based on epoch for
this use case. I guess we should probably call it seekToNearest() though to
make it clear that the final position may be different from what was
requested.

Thanks,
Jason

On Wed, Jun 27, 2018 at 3:20 PM, Guozhang Wang  wrote:

> Hi Jason,
>
> I think it is less worthwhile to add KafkaConsumer#offsetsForLeaderEpochs,
> since probably only very advanced users are aware of the leaderEpoch, and
> hence ever care to use it anyways. It is more like an admin client
> operation than a consumer client operation: if the motivation is to
> facility customized reset policy, maybe adding it as
> AdminClient#offsetsForLeaderEpochs
> is better as it is not an aggressive assumption that for such advanced
> users they are willing to use some admin client to get further information?
>
>
> Guozhang
>
>
> On Wed, Jun 27, 2018 at 2:07 PM, Jason Gustafson 
> wrote:
>
> > Thanks for the feedback. I've updated the KIP. Specifically I removed the
> > "closest" reset option and the proposal to reset by timestamp when the
> > precise truncation point cannot be determined. Instead, I proposed that
> we
> > always reset using the nearest epoch when a reset policy is defined
> (either
> > "earliest" or "latest"). Does that sound reasonable?
> >
> > One thing I am still debating is whether it would be better to have a
> > separate API to find the closest offset using the leader epoch. In the
> > current KIP, I suggested to piggyback this information on an exception,
> but
> > I'm beginning to think it would be better not to hide the lookup. It is
> > awkward to implement since it means delaying the exception and the API
> may
> > actually be useful when customizing reset logic if no auto reset policy
> is
> > defined. I was thinking we can add an API like the following:
> >
> > Map
> > offsetsForLeaderEpochs(Map epochsToSearch)
> >
> > Thoughts?
> >
> > -Jason
> >
> >
> >
> >
> > On Wed, Jun 27, 2018 at 11:12 AM, Jason Gustafson 
> > wrote:
> >
> > > @Dong
> > >
> > > Those are fair points. Both approaches require some fuzziness to reset
> > the
> > > offset in these pathological scenarios and we cannot guarantee
> > > at-least-once delivery either way unless we have the full history of
> > leader
> > > epochs that were consumed. The KIP-101 logic may actually be more
> > accurate
> > > than using timestamps because it does not depend on the messages which
> > are
> > > written after the unclean leader election. The case we're talking about
> > > should be extremely rare in practice anyway. I also agree that we may
> not
> > > want to add new machinery if it only helps the old message format. Ok,
> > > let's go ahead and drop the timestamp.
> > >
> > > @Guozhang
> > >
> > > * My current understanding is that, with unclean leader election turned
> > on,
> > >> exactly-once is out of the window since we cannot guarantee that all
> > >> committed message markers will not be lost. And hence there is no need
> > to
> > >> have special handling logic for LOG_TRUNCATED or OOR error codes with
> > >> read.committed turned on. Is that right?
> > >
> > >
> > > Yes, that's right. EoS and unclean leader election don't mix well. It
> may
> > > be worth considering separately whether we should try to reconcile the
> > > transaction log following an unclean leader election. At least we may
> be
> > > able to prevent dangling transactions from blocking consumers. This KIP
> > > does not address this problem.
> > >
> > > * MINOR: "if the epoch is greater than the minimum expected epoch, that
> > the
> > >> new epoch does not begin at an earlier offset than the fetch offset.
> In
> > >> the latter case, the leader can respond with a new LOG_TRUNCATION
> error
> > >> code" should it be "does not begin at a later offset than the fetch
> > >> offset"?
> > >
> > >
> > > I think the comment is correct, though the phrasing may be confusing.
> We
> > > know truncation has occurred if there exists a larger epoch with a
> > starting
> > > offset that is lower than the fetch offset. Let me try to rephrase
> this.
> > >
> > > Thanks,
> > > Jason
&g

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

[Jason Gustafson]

Hey Gwen,

Why do users want partition size? It seems like a weird thing to be
concerned about. Perhaps they are trying to get a sense of the lag as a
percentage of the total size of the partition or something like that?

-Jason

On Tue, Jun 26, 2018 at 9:12 PM, Gwen Shapira  wrote:

> Thank you!
>
> On Tue, Jun 26, 2018 at 8:47 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Thanks for the feedback. The KIP is updated to also include a "partition
> > size" column.
> >
> > --Vahid
> >
> >
> >
> >
> > From:   Ted Yu 
> > To: dev@kafka.apache.org
> > Date:   06/26/2018 06:21 PM
> > Subject:Re: [DISCUSS] KIP-325: Extend Consumer Group Command to
> > Show Beginning Offsets
> >
> >
> >
> > nit:
> >
> > bq. leaving this empty for compacted topics
> >
> > Some user(s) may be confused by empty partition size. How about emitting
> > 'compacted' for compacted topics ?
> >
> > Cheers
> >
> > On Tue, Jun 26, 2018 at 4:42 PM, Gwen Shapira  wrote:
> >
> > > It will be. In my experience most topics aren't compacted, so it will
> > still
> > > be valuable. If not difficult, leaving this empty for compacted topics
> > to
> > > avoid confusion will also be nice.
> > >
> > > On Tue, Jun 26, 2018 at 4:29 PM, Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com> wrote:
> > >
> > > > Hi Gwen,
> > > >
> > > > Thanks for the feedback.
> > > > Regarding the partition size, couldn't "end offset - start offset" be
> > > > misleading for compacted topics?
> > > >
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From:   Gwen Shapira 
> > > > To: dev 
> > > > Date:   06/26/2018 02:36 PM
> > > > Subject:Re: [DISCUSS] KIP-325: Extend Consumer Group Command
> > to
> > > > Show Beginning Offsets
> > > >
> > > >
> > > >
> > > > Small suggestion: you can also add a "partition size" column -
> > difference
> > > > between log-end and log-start. We've had users ask for this.
> > > >
> > > > On Tue, Jun 26, 2018 at 2:34 PM, Gwen Shapira 
> > wrote:
> > > >
> > > > > This will be useful! Thank you :)
> > > > >
> > > > > On Tue, Jun 26, 2018 at 11:23 AM, Vahid S Hashemian <
> > > > > vahidhashem...@us.ibm.com> wrote:
> > > > >
> > > > >> Hi everyone,
> > > > >>
> > > > >> I have created a trivial KIP to improve the offset reporting of
> the
> > > > >> consumer group command:
> > > > >>
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-325%
> >
> > > >
> > > > >> 3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets
> > > > >> Looking forward to your feedback!
> > > > >>
> > > > >> Thanks.
> > > > >> --Vahid
> > > > >>
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > *Gwen Shapira*
> > > > > Product Manager | Confluent
> > > > > 650.450.2760 | @gwenshap
> > > > > Follow us: Twitter <
> > > >
> > https://twitter.com/ConfluentInc
> >
> > > > > | blog
> > > > > <
> > > >
> > http://www.confluent.io/blog
> >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > *Gwen Shapira*
> > > > Product Manager | Confluent
> > > > 650.450.2760 | @gwenshap
> > > > Follow us: Twitter <
> > > >
> > https://twitter.com/ConfluentInc
> >
> > > > > | blog
> > > > <
> > > >
> > http://www.confluent.io/blog
> >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > *Gwen Shapira*
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter <
> > https://twitter.com/ConfluentInc
> > > | blog
> > > <
> > http://www.confluent.io/blog
> > >
> > >
> >
> >
> >
> >
> >
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter  | blog
> 
>

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

[Jason Gustafson]

Hey Gwen/Vahid,

I think that use case makes sense, but isn't it a little odd to go through
the consumer group tool? I would expect to find something like that from
the kafka-topics tool instead. I don't feel too strongly about it, but I
hate to complicate the tool by adding the need to query topic configs. If
we don't have a meaningful number to report for compacted topics anyway,
then it feels like only a half solution. I'd probably suggest leaving this
out or just reporting the absolute difference even if a topic is compacted.

-Jason



On Thu, Jun 28, 2018 at 1:05 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi James,
>
>
>
> Thanks for the feedback. I updated the KIP and added some of the benefits
>
> of this improvement (including some that you mentioned).
>
>
>
> Regards.
>
> --Vahid
>
>
>
>
>
>
>
> From:   James Cheng 
>
> To: dev@kafka.apache.org
>
> Date:   06/27/2018 09:13 PM
>
> Subject:Re: [DISCUSS] KIP-325: Extend Consumer Group Command to
>
> Show Beginning Offsets
>
>
>
>
>
>
>
> The “Motivation” section of the KIP says that the starting offset will be
>
> useful but doesn’t say why. Can you add a use-case or two to describe how
>
> it will be useful?
>
>
>
> In our case (see
>
> https://github.com/wushujames/kafka-utilities/blob/master/Co
> nsumerGroupLag/README.md
>
> ), we found the starting offset useful so that we could calculate
>
> partition size so that we could identify empty partitions (partitions
>
> where all the data had expired). In particular, we needed that info so
>
> that we could calculate “lag”. Consider that case where we are asked to
>
> mirror an abandoned topic where startOffset==endOffset==100. We would
>
> have no committed offset on it, and the topic has no data in it, so we
>
> won’t soon get any committed offset, and so “lag” is kind of undefined. We
>
> used the additional startOffset to allow us to detect this case.
>
>
>
> -James
>
>
>
> Sent from my iPhone
>
>
>
> > On Jun 26, 2018, at 11:23 AM, Vahid S Hashemian
>
>  wrote:
>
> >
>
> > Hi everyone,
>
> >
>
> > I have created a trivial KIP to improve the offset reporting of the
>
> > consumer group command:
>
> >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-325%
> 3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets
>
>
>
> > Looking forward to your feedback!
>
> >
>
> > Thanks.
>
> > --Vahid
>
> >
>
> >
>
>
>
>
>
>
>
>
>
>

Re: [VOTE] KIP-308: Support dynamic update of max.connections.per.ip/max.connections.per.ip.overrides configs

[Jason Gustafson]

+1

Thanks Manikumar!

On Fri, Jun 29, 2018 at 8:37 AM, Rajini Sivaram 
wrote:

> +1 (binding)
>
> Thanks for the KIP, Manikumar!
>
> On Fri, Jun 29, 2018 at 4:23 PM, Dong Lin  wrote:
>
> > +1
> >
> > Thanks!
> >
> > On Fri, 29 Jun 2018 at 7:36 AM Ted Yu  wrote:
> >
> > > +1
> > >
> > > On Fri, Jun 29, 2018 at 7:29 AM, Manikumar 
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I would like to start voting on KIP-308 which would add support for
> > > dynamic
> > > > update of max.connections.per.ip/max.connections.per.ip.overrides
> > configs
> > > >
> > > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=85474993
> > > >
> > > > Thanks,
> > > >
> > >
> >
>

Re: [VOTE] 0.10.2.2 RC1

[Jason Gustafson]

+1 (binding). I checked release notes, documentation, and went through the
quickstart.

Thanks Matthias!

On Fri, Jun 22, 2018 at 6:43 PM, Matthias J. Sax 
wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 0.10.2.2.
>
> Note, that RC0 was created before the upgrade to Gradle 4.8.1 and thus,
> we discarded it in favor of RC1 (without sending out a email for RC0).
>
> This is a bug fix release closing 29 tickets:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.2.2
>
> Release notes for the 0.10.2.2 release:
> http://home.apache.org/~mjsax/kafka-0.10.2.2-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Tuesday, 6/26/18 end-of-day, so we
> can close the vote on Wednesday.
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~mjsax/kafka-0.10.2.2-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~mjsax/kafka-0.10.2.2-rc1/javadoc/
>
> * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.2 tag:
> https://github.com/apache/kafka/releases/tag/0.10.2.2-rc1
>
> * Documentation:
> http://kafka.apache.org/0102/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0102/protocol.html
>
> * Successful Jenkins builds for the 0.10.2 branch:
> Unit/integration tests: https://builds.apache.org/job/
> kafka-0.10.2-jdk7/220/
>
> /**
>
> Thanks,
>   -Matthias
>
>

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jason Gustafson]

Hey Dong,

Thanks for the feedback. The first three points are easy:

1. Yes, we should be consistent.
2. Yes, I will add this.
3. Yes, I think we should document the changes to the committed offset
schema. I meant to do this, but it slipped my mind.

The latter questions are tougher. One option I was considering is to have
only `offsetsForLeaderEpochs` exposed from the consumer and to drop the new
seek() API. That seems more consistent with the current use of
`offsetsForTimes` (we don't have a separate `seekToTimestamp` API). An
alternative might be to take a page from the AdminClient API and add a new
method to generalize offset lookup. For example, we could have
`lookupOffsets(LookupOptions)`. We could then deprecate `offsetsForTimes`
and this would open the door for future extensions without needing new APIs.

The case of position() is a little more annoying. It would have been better
had we let this return an object so that it is easier to extend. This is
the only reason I didn't add the API to the KIP. Maybe we should bite the
bullet and fix this now? Unfortunately we'll have to come up with a new
name. Maybe `currentPosition`?

Thoughts?

-Jason


On Fri, Jun 29, 2018 at 10:40 AM, Dong Lin  wrote:

> Regarding points 4) and 5) above, motivation for the alternative APIs is
> that, if we decide that leaderEpoch is equally important as offset in
> identifying a message, then it may be reasonable to always specify it
> wherever offset is currently required in the consumer API to identify a
> message, e.g. position(), seek(). For example, since we allow user to
> retrieve offset using position() instead of asking user to keep track of
> the offset of the latest ConsumerRecord, may be it will be more consistent
> for user to also retrieve  leaderEpoch using position()?
>
>
>
> On Fri, Jun 29, 2018 at 10:30 AM, Dong Lin  wrote:
>
> > Hey Jason,
> >
> > Thanks for the update. It looks pretty good. Just some minor comments
> > below:
> >
> > 1) The KIP adds new error code "LOG_TRUNCATION" and new exception
> TruncatedPartitionException.
> > Can we make the name more consistent, e.g. LogTruncationException?
> >
> > 2) Do we need to add UnknownLeaderEpochException as part of API change?
> >
> > 3) Not sure if the offset topic schema is also public API. If so, maybe
> we
> > should also include the schema change in the API?
> >
> > 4) For users who store offset externally, currently they get offset using
> > position(..), store the offset externally, and use seek(..) to initialize
> > the consumer next time. After this KIP they will need to store and use
> the
> > leaderEpoch together with the offset. Should we also update the API so
> that
> > user can also get leaderEpoch from position(...)? Not sure if it is OK to
> > ask user to track the latest leaderEpoch of ConsumerRecord by themselves.
> >
> > 5) Also for users who store offset externally, they need to call seek(..)
> > with leaderEpoch to initialize consumer. With current KIP users need to
> > call seekToNearest(), whose name suggests that the final position may be
> > different from what was requested. However, if users may want to avoid
> auto
> > offset reset and be notified explicitly when there is log truncation,
> then seekToNearest()
> > probably does not help here. Would it make sense to replace
> seekToNearest()
> > with seek(offset, leaderEpoch) + AminClient.offsetsForLeaderEpochs(...)?
> >
> >
> > Thanks,
> > Dong
> >
> >
> > On Wed, Jun 27, 2018 at 3:57 PM, Jason Gustafson 
> > wrote:
> >
> >> Hey Guozhang,
> >>
> >> That's fair. In fact, perhaps we do not need this API at all. We already
> >> have the new seek() in this KIP which can do the lookup based on epoch
> for
> >> this use case. I guess we should probably call it seekToNearest() though
> >> to
> >> make it clear that the final position may be different from what was
> >> requested.
> >>
> >> Thanks,
> >> Jason
> >>
> >> On Wed, Jun 27, 2018 at 3:20 PM, Guozhang Wang 
> >> wrote:
> >>
> >> > Hi Jason,
> >> >
> >> > I think it is less worthwhile to add KafkaConsumer#offsetsForLeader
> >> Epochs,
> >> > since probably only very advanced users are aware of the leaderEpoch,
> >> and
> >> > hence ever care to use it anyways. It is more like an admin client
> >> > operation than a consumer client operation: if the motivation is to
> >> > facility customized reset policy, maybe adding it as
> >> > AdminClient#offsetsForLeaderEpoc

Re: [VOTE] 1.0.2 RC0

[Jason Gustafson]

Hey Matthias,

I don't see the artifacts for scala 2.12. Did they not get uploaded?

Thanks,
Jason

On Fri, Jun 29, 2018 at 12:41 PM, Guozhang Wang  wrote:

> +1 (binding), checked release note, java doc, and ran unit tests on source
> tgz.
>
>
> Guozhang
>
> On Mon, Jun 25, 2018 at 8:05 PM Manikumar 
> wrote:
>
> > +1 (non-binding) Verified tests, quick start, producer/consumer perf
> tests.
> >
> > On Sat, Jun 23, 2018 at 2:25 AM Ted Yu  wrote:
> >
> > > +1
> > >
> > > Ran test suite.
> > >
> > > Checked signatures.
> > >
> > > On Fri, Jun 22, 2018 at 11:42 AM, Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Built from source and ran quickstart successfully on Ubuntu (with
> Java
> > > 8).
> > > >
> > > > Thanks for running the release Matthias!
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From:   "Matthias J. Sax" 
> > > > To: dev@kafka.apache.org, us...@kafka.apache.org,
> > > > kafka-clie...@googlegroups.com
> > > > Date:   06/22/2018 10:42 AM
> > > > Subject:[VOTE] 1.0.2 RC0
> > > >
> > > >
> > > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the first candidate for release of Apache Kafka 1.0.2.
> > > >
> > > > This is a bug fix release closing 26 tickets:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.0.2
> > > >
> > > > Release notes for the 1.0.2 release:
> > > > http://home.apache.org/~mjsax/kafka-1.0.2-rc0/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Tuesday, 6/26/18 end-of-day, so
> > we
> > > > can close the vote on Wednesday.
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > http://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > http://home.apache.org/~mjsax/kafka-1.0.2-rc0/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > > https://repository.apache.org/content/groups/staging/
> > > >
> > > > * Javadoc:
> > > > http://home.apache.org/~mjsax/kafka-1.0.2-rc0/javadoc/
> > > >
> > > > * Tag to be voted upon (off 1.0 branch) is the 1.0.2 tag:
> > > > https://github.com/apache/kafka/releases/tag/1.0.2-rc0
> > > >
> > > > * Documentation:
> > > > http://kafka.apache.org/10/documentation.html
> > > >
> > > > * Protocol:
> > > > http://kafka.apache.org/10/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 1.0 branch:
> > > > Unit/integration tests:
> > > https://builds.apache.org/job/kafka-1.0-jdk7/211/
> > > > System tests:
> > > > https://jenkins.confluent.io/job/system-test-kafka/job/1.0/217/
> > > >
> > > > /**
> > > >
> > > > Thanks,
> > > >   -Matthias
> > > >
> > > > [attachment "signature.asc" deleted by Vahid S Hashemian/Silicon
> > > > Valley/IBM]
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jason Gustafson]

Hi Dong,

Thanks, I've been thinking about your suggestions a bit. It is challenging
to make this work given the current APIs. One of the difficulties is that
we don't have an API to find the leader epoch for a given offset at the
moment. So if the user does a seek to offset 5, then we'll need a new API
to find the corresponding epoch in order to fulfill the new position() API.
Potentially we could modify ListOffsets to enable finding the leader epoch,
but I am not sure it is worthwhile. Perhaps it is reasonable for advanced
usage to expect that the epoch information, if needed, will be extracted
from the records directly? It might make sense to expose a helper in
`ConsumerRecords` to make this a little easier though.

Alternatively, if we think it is important to have this information exposed
directly, we could create batch APIs to solve the naming problem. For
example:

Map positions();
void seek(Map positions);

However, I'm actually leaning toward leaving the seek() and position() APIs
unchanged. Instead, we can add a new API to search for offset by timestamp
or by offset/leader epoch. Let's say we call it `findOffsets`. If the user
hits a log truncation error, they can use this API to find the closest
offset and then do a seek(). At the same time, we deprecate the
`offsetsForTimes` APIs. We now have two use cases which require finding
offsets, so I think we should make this API general and leave the door open
for future extensions.

By the way, I'm unclear about the desire to move part of this functionality
to AdminClient. Guozhang suggested this previously, but I think it only
makes sense for cross-cutting capabilities such as topic creation. If we
have an API which is primarily useful by consumers, then I think that's
where it should be exposed. The AdminClient also has its own API integrity
and should not become a dumping ground for advanced use cases. I'll update
the KIP with the  `findOffsets` API suggested above and we can see if it
does a good enough job of keeping the API simple for common cases.

Thanks,
Jason


On Sat, Jun 30, 2018 at 4:39 AM, Dong Lin  wrote:

> Hey Jason,
>
> Regarding seek(...), it seems that we want an API for user to initialize
> consumer with (offset, leaderEpoch) and that API should allow throwing
> PartitionTruncationException. Suppose we agree on this, then
> seekToNearest() is not sufficient because it will always swallow
> PartitionTruncationException. Here we have two options. The first option is
> to add API offsetsForLeaderEpochs() to translate (leaderEpoch, offset) to
> offset. The second option is to have add seek(offset, leaderEpoch). It
> seems that second option may be more simpler because it makes it clear that
> (offset, leaderEpoch) will be used to identify consumer's position in a
> partition. And user only needs to handle PartitionTruncationException from
> the poll(). In comparison the first option seems a bit harder to use
> because user have to also handle the PartitionTruncationException if
> offsetsForLeaderEpochs() returns different offset from user-provided
> offset. What do you think?
>
> If we decide to add API seek(offset, leaderEpoch), then we can decide
> whether and how to add API to translate (offset, leaderEpoch) to offset. It
> seems that this API will be needed by advanced user to don't want auto
> offset reset (so that it can be notified) but still wants to reset offset
> to closest. For those users if probably makes sense to only have the API in
> AdminClient. offsetsForTimes() seems like a common API that will be needed
> by user's of consumer in general, so it may be more reasonable to stay in
> the consumer API. I don't have a strong opinion on whether
> offsetsForTimes() should be replaced by API in AdminClient.
>
> Though (offset, leaderEpoch) is needed to uniquely identify a message in
> general, it is only needed for advanced users who has turned on unclean
> leader election, need to use seek(..), and don't want auto offset reset.
> Most other users probably just want to enable auto offset reset and store
> offset in Kafka. Thus we might want to keep the existing offset-only APIs
> (e.g. seek() and position()) for most users while adding new APIs for
> advanced users. And yes, it seems that we need new name for position().
>
> Though I think we need new APIs to carry the new information (e.g.
> leaderEpoch), I am not very sure how that should look like. One possible
> option is those APIs in KIP-232. Another option is something like this:
>
> `
> class OffsetEpochs {
>   long offset;
>   int leaderEpoch;
>   int partitionEpoch;   // This may be needed later as discussed in KIP-232
>   ... // Hopefully these are all we need to identify message in Kafka. But
> if we need more then we can add new fields in this class.
>

Re: [ANNOUNCE] Apache Kafka 0.11.0.3 Released

[Jason Gustafson]

Awesome. Thanks Matthias!

On Tue, Jul 3, 2018 at 12:44 PM, Yishun Guan  wrote:

> Nice! Thanks~
>
> On Tue, Jul 3, 2018, 12:16 PM Ismael Juma  wrote:
>
> > Thanks Matthias!
> >
> > On Tue, 3 Jul 2018, 11:31 Matthias J. Sax,  wrote:
> >
> > > -BEGIN PGP SIGNED MESSAGE-
> > > Hash: SHA512
> > >
> > > The Apache Kafka community is pleased to announce the release for
> > > Apache Kafka 0.11.0.3.
> > >
> > >
> > > This is a bug fix release and it includes fixes and improvements from
> > > 27 JIRAs, including a few critical bugs.
> > >
> > >
> > > All of the changes in this release can be found in the release notes:
> > >
> > >
> > > https://dist.apache.org/repos/dist/release/kafka/0.11.0.3/
> RELEASE_NOTES.
> > > html
> > > <
> > https://dist.apache.org/repos/dist/release/kafka/0.11.0.3/
> RELEASE_NOTES.html
> > >
> > >
> > >
> > >
> > > You can download the source release from:
> > >
> > >
> > > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.
> 3/kafka-0.11.0.
> > > 3-src.tgz
> > > <
> > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.
> 3/kafka-0.11.0.3-src.tgz
> > >
> > >
> > >
> > > and binary releases from:
> > >
> > >
> > > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.
> 3/kafka_2.11-0.
> > > 11.0.3.tgz
> > > (Scala 2.11)
> > >
> > > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.
> 3/kafka_2.12-0.
> > > 11.0.3.tgz
> > > (Scala 2.12)
> > >
> > >
> > > -
> > 
> > > - ---
> > >
> > >
> > > Apache Kafka is a distributed streaming platform with four core APIs:
> > >
> > >
> > > ** The Producer API allows an application to publish a stream records
> > > to one or more Kafka topics.
> > >
> > >
> > > ** The Consumer API allows an application to subscribe to one or more
> > > topics and process the stream of records produced to them.
> > >
> > >
> > > ** The Streams API allows an application to act as a stream processor,
> > > consuming an input stream from one or more topics and producing an
> > > output stream to one or more output topics, effectively transforming
> > > the input streams to output streams.
> > >
> > >
> > > ** The Connector API allows building and running reusable producers or
> > > consumers that connect Kafka topics to existing applications or data
> > > systems. For example, a connector to a relational database might
> > > capture every change to a table.three key capabilities:
> > >
> > >
> > >
> > > With these APIs, Kafka can be used for two broad classes of
> application:
> > >
> > >
> > > ** Building real-time streaming data pipelines that reliably get data
> > > between systems or applications.
> > >
> > >
> > > ** Building real-time streaming applications that transform or react
> > > to the streams of data.
> > >
> > >
> > >
> > > Apache Kafka is in use at large and small companies worldwide,
> > > including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> > > Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> > > Zalando, among others.
> > >
> > >
> > >
> > > A big thank you for the following 26 contributors to this release!
> > >
> > >
> > > Matthias J. Sax, Ewen Cheslack-Postava, Konstantine Karantasis,
> > > Guozhang Wang, Rajini Sivaram, Randall Hauch, tedyu, Jagadesh
> > > Adireddi, Jarek Rudzinski, Jason Gustafson, Jeremy Custenborder, Anna
> > > Povzner, Joel Hamill, John Roesler, Max Zheng, Mickael Maison, Robert
> > > Yokota, Yaswanth Kumar, parafiend, Jiangjie (Becket) Qin, Arjun
> > > Satish, Bill Bejeck, Damian Guy, Gitomain, Gunnar Morling, Ismael Juma
> > >
> > >
> > > We welcome your help and feedback. For more information on how to
> > > report problems, and to get involved, visit the project website at
> > > http://kafka.apache.org/
> > >
> > >
> > > Thank you!
> > >
> > >
> > > Regards,
> > >  -Matthias
> > > -BEGIN PGP SIGNATURE-
> > > Comment: GPGTools - https://gpgtools.org
> > >
> > > iQIzBAEBCgAdFiEEeiQdEa0SVXokodP3DccxaWtLg18FAls7wQAACgkQDccxaWtL
> > > g1+b/g/+LjM5gh8u2wCVz7dhOstwvtaajRG7cG1QhZH3H9QquVs19aKiE9ZcvEcK
> > > eJkX0S7rWopXs2qQxy5fVCTWGw5yO4eFNWuWxSIffuxH8/3K2sKahPi/4IDgd5Tj
> > > ksmsxyXxWtGv/vEosJr+ZD7s1urPpkQ7DG6CT9wG9wj2ASq7sur/Eg7jfAnuIoTQ
> > > UvQenKXU0T+D+BZKpUiZs5e6VGya6bUzboAbPGiwsMH4/xj2IlOEjVAyf3ppnuiu
> > > /AW2LLqkFnbDB0IbveOu2+73CvVlahkaZ6nhPjkVpdpFw/SCAZHdkGdCafo8DKP8
> > > DKcmzta/QCEJ1uQUe7Rh8ndzYLzTaU0rqilA2WZUZvTx0gkviDGvQv/S97XP8lRJ
> > > SLn2xk166dxw0zpuIfzo0rr3S2Mz5PmAhrxiVxDG9ihaqBnABePspjp+cTXLhGhX
> > > 5zEhh1THiShjT03ZSPP8SEioQmj9LoQ9FH53/RXGmQ35O/nv4bAcvRvkqntFoF4Z
> > > iXE0bhQ2RyffQjBc70uJfdrpRbsmPqnNKSJ+60cB9y6jN+aQBuQdjB54ypu203mp
> > > x+yj7Fl+yf/EFbcs4aeAccAnx3J8uo6K1bKJmJtWrrBIIF28nNBrdBXGWh898rGe
> > > +m7teNKOm6WJXnuzASja82xJjul60WWOwAFLSOL1aAqo+At5Sps=
> > > =4xXe
> > > -END PGP SIGNATURE-
> > >
> >
>

Re: [VOTE] KIP-280: Enhanced log compaction

[Jason Gustafson]

Sorry to join the discussion late. Can you you add to the motivation the
use cases for header-based compaction. This seems not very clear to me.

Thanks,
Jason

On Mon, Jul 2, 2018 at 11:00 AM, Guozhang Wang  wrote:

> Hi Luis,
>
> I believe that compaction property is indeed overridable at per-topic
> level, as in
>
> https://github.com/apache/kafka/blob/0cacbcf30e0a90ab9fad7bc310e547
> 7cf959f1fd/clients/src/main/java/org/apache/kafka/common/
> config/TopicConfig.java#L116
>
> And also documented in https://kafka.apache.org/
> documentation/#topicconfigs
>
> Is that right?
>
>
>
> Guozhang
>
> On Mon, Jul 2, 2018 at 7:41 AM, Luís Cabral  >
> wrote:
>
> >  Hi Guozhang,
> >
> > You are right that it is not straightforward to add a dependent property
> > validation.
> > Though it is possible to re-design it to allow for this, that effort
> would
> > be better placed under its own KIP, if it really becomes useful for other
> > properties as well.
> > Given this, the fallback-to-offset behaviour currently documented will be
> > used.
> >
> > Also, while analyzing this, I noticed that the existing compaction
> > properties only exist globally, and not per topic.
> > I don't understand why this is, but it again feels like something out of
> > scope for this KIP.
> > Given this, the KIP was updated to only include the global configuration
> > properties, removing the per-topic configs.
> >
> > I'll soon update the PR according to the documentation, but I trust the
> > KIP doesn't need that to close, right?
> >
> > Cheers,
> > Luis
> >
> > On Monday, July 2, 2018, 2:00:08 PM GMT+2, Luís Cabral
> >  wrote:
> >
> >   Hi Guozhang,
> >
> > At the moment the KIP has your vote, Matthias' and Ted's.
> > Should I ask someone else to have a look?
> >
> > Cheers,
> > Luis
> >
> > On Monday, July 2, 2018, 12:16:48 PM GMT+2, Mickael Maison <
> > mickael.mai...@gmail.com> wrote:
> >
> >  +1 (non binding). Thanks for the KIP!
> >
> > On Sat, Jun 30, 2018 at 12:26 AM, Guozhang Wang 
> > wrote:
> > > Hi Luis,
> > >
> > > Regarding the minor suggest, I agree it would be better to make it as
> > > mandatory, but it might be a bit tricky because it is a conditional
> > > mandatory one depending on the other config's value. Would like to see
> > your
> > > updated PR.
> > >
> > > Regarding the KIP itself, both Matthias and myself can recast our votes
> > to
> > > the updated wiki, while we still need one more committer to vote
> > according
> > > to the bylaws.
> > >
> > >
> > > Guozhang
> > >
> > > On Thu, Jun 28, 2018 at 5:38 AM, Luís Cabral
> > 
> > > wrote:
> > >
> > >>  Hi,
> > >>
> > >> Thank you all for having a look!
> > >>
> > >> The KIP is now updated with the result of these late discussions,
> > though I
> > >> did take some liberty with this part:
> > >>
> > >>
> > >>- If the "compaction.strategy.header" configuration is not set (or
> is
> > >> blank), then the compaction strategy will fallback to "offset";
> > >>
> > >>
> > >> Alternatively, we can also set it to be a mandatory property when the
> > >> strategy is "header" and fail the application to start via a config
> > >> validation (I would honestly prefer this, but its up to your taste).
> > >>
> > >> Anyway, this is now a minute detail that can be adapted during the
> final
> > >> stage of this KIP, so are you all alright with me changing the status
> to
> > >> [ACCEPTED]?
> > >>
> > >> Cheers,
> > >> Luis
> > >>
> > >>
> > >>On Thursday, June 28, 2018, 2:08:11 PM GMT+2, Ted Yu <
> > >> yuzhih...@gmail.com> wrote:
> > >>
> > >>  +1
> > >>
> > >> On Thu, Jun 28, 2018 at 4:56 AM, Luís Cabral
> >  > >> >
> > >> wrote:
> > >>
> > >> > Hi Ted,
> > >> > Can I also get your input on this?
> > >> >
> > >> > bq. +1 from my side for using `compaction.strategy` with values
> > >> > "offset","timestamp" and "header" and `compaction.strategy.header`
> > >> > -Matthias
> > >> >
> > >> > bq. +1 from me as well.
> > >> > -Guozhang
> > >> >
> > >> >
> > >> > Cheers,
> > >> > Luis
> > >> >
> > >> >
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> >
>
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jason Gustafson]

; > > >> >
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Anna
> > > > >> >
> > > > >> >
> > > > >> > On Sat, Jul 7, 2018 at 1:36 AM Dong Lin 
> > > wrote:
> > > > >> >
> > > > >> > > Hey Anna,
> > > > >> > >
> > > > >> > > Thanks much for the thoughtful reply. It makes sense to
> > different
> > > > >> between
> > > > >> > > "seeking to a message" and "seeking to a position". I have to
> > > > >> questions
> > > > >> > > here:
> > > > >> > >
> > > > >> > > - For "seeking to a message" use-case, with the proposed
> > approach
> > > > user
> > > > >> > > needs to call findOffset(offset, leaderEpoch) followed by
> > > > >> seek(offset).
> > > > >> > If
> > > > >> > > message truncation and message append happen immediately after
> > > > >> > > findOffset(offset,
> > > > >> > > leaderEpoch) but before seek(offset), it seems that user will
> > seek
> > > > to
> > > > >> the
> > > > >> > > wrong message without knowing the truncation has happened.
> Would
> > > > this
> > > > >> be
> > > > >> > a
> > > > >> > > problem?
> > > > >> > >
> > > > >> > > - For "seeking to a position" use-case, it seems that there
> can
> > be
> > > > two
> > > > >> > > positions, i.e. earliest and latest. So these two cases can be
> > > > >> > > Consumer.fulfilled by seekToBeginning() and
> > Consumer.seekToEnd().
> > > > >> Then it
> > > > >> > > seems that user will only need to call position() and seek()
> for
> > > > >> "seeking
> > > > >> > > to a message" use-case?
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Dong
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, Jul 4, 2018 at 12:33 PM, Anna Povzner <
> > a...@confluent.io>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > Hi Jason and Dong,
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > I’ve been thinking about your suggestions and discussion
> > > regarding
> > > > >> > > > position(), seek(), and new proposed API.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > Here is my thought process why we should keep position() and
> > > > seek()
> > > > >> API
> > > > >> > > > unchanged.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > I think we should separate {offset, leader epoch} that
> > uniquely
> > > > >> > > identifies
> > > > >> > > > a message from an offset that is a position. In some cases,
> > > > offsets
> > > > >> > > > returned from position() could be actual consumed messages
> by
> > > this
> > > > >> > > consumer
> > > > >> > > > identified by {offset, leader epoch}. In other cases,
> > position()
> > > > >> > returns
> > > > >> > > > offset that was not actually consumed. Suppose, the user
> calls
> > > > >> > position()
> > > > >> > > > for the last offset. Suppose we return {offset, leader
> epoch}
> > of
> > > > the
> > > > >> > > > message currently in the log. Then, the message gets
> truncated
> > > > >> before
> > > > >> > > > consumer’s first poll(). It does not make sense for poll()
> to
> > > fail
> > > > >> in
> > > > >> > > this
> > > > >> > > > case, because the log truncation did not actually happen

Re: KIP-327: Add describe all topics API to AdminClient

[Jason Gustafson]

Hey Manikumar,

As Kafka begins to scale to larger and larger numbers of topics/partitions,
I'm a little concerned about the scalability of APIs such as this. The API
looks benign, but imagine you have have a few million partitions. We
already expose similar APIs in the producer and consumer, so probably not
much additional harm to expose it in the AdminClient, but it would be nice
to put a little thought into some longer term options. We should be giving
users an efficient way to select a smaller set of the topics they are
interested in. We have always discussed adding some filtering support to
the Metadata API. Perhaps now is a good time to reconsider this? We now
have a convention for wildcard ACLs, so perhaps we can do something
similar. Full regex support might be ideal given the consumer's
subscription API, but that is more challenging. What do you think?

Thanks,
Jason

On Thu, Jul 12, 2018 at 2:35 PM, Harsha  wrote:

> Very useful. LGTM.
>
> Thanks,
> Harsha
>
> On Thu, Jul 12, 2018, at 9:56 AM, Manikumar wrote:
> > Hi all,
> >
> > I have created a KIP to add describe all topics API to AdminClient .
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 327%3A+Add+describe+all+topics+API+to+AdminClient
> >
> > Please take a look.
> >
> > Thanks,
>

Re: [VOTE] KIP-338 Support to exclude the internal topics in kafka-topics.sh command

[Jason Gustafson]

Hi, thanks for the KIP. +1 from me.

I think eventually we'd want --exclude-internal-topics to be the default,
but it seems reasonable to keep the current behavior for compatibility.

-Jason

On Mon, Jul 16, 2018 at 2:23 PM, Dong Lin  wrote:

> +1
>
> Thanks!
> Dong
>
> On Mon, Jul 16, 2018 at 9:17 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > +1 (non-binding)
> >
> > Thanks!
> > --Vahid
> >
> >
> >
> >
> > From:   Chia-Ping Tsai 
> > To: 
> > Date:   07/16/2018 01:11 AM
> > Subject:[VOTE] KIP-338 Support to exclude the internal topics in
> > kafka-topics.sh command
> >
> >
> >
> > hi folks,
> >
> > The discussion[1] of KIP-338[2] did not get any objection for last 6 days
> > so it is time to start the voting thread.
> >
> > Thanks for your time!
> >
> > [1]
> > https://lists.apache.org/thread.html/9bd4e61b73c901b51132ada49743b9
> > b703d40b85fc4eeaa5c9099900@%3Cdev.kafka.apache.org%3E
> >
> >
> > [2]
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 338+Support+to+exclude+the+internal+topics+in+kafka-topics.sh+command
> >
> >
> > Cheers,
> > chia-ping
> >
> >
> >
> >
> >
> >
>

Re: [VOTE] 1.1.1 RC3

[Jason Gustafson]

+1

- Did basic quickstarts on the 2.11 and 2.12 artifacts
- Checked upgrade notes and documentation

Thanks for managing the release, Dong!


-Jason

On Thu, Jul 12, 2018 at 2:36 PM, Harsha  wrote:

> +1.
> 1. Ran unit tests
> 2. Ran 3 node cluster to run few tests.
>
> Thanks,
> Harsha
>
> On Thu, Jul 12, 2018, at 7:29 AM, Manikumar wrote:
> > +1 (non-binding)  Ran tests,  Verified quick start,  producer/consumer
> perf
> > tests
> >
> >
> >
> > On Thu, Jul 12, 2018 at 11:06 AM Brett Rann 
> > wrote:
> >
> > > +1 (non binding)
> > > rolling upgrade of shared staging multitenacy (200+ consumer groups)
> > > cluster from 1.1.0 to 1.1.1-rc3 using the kafka_2.11-1.1.1.tgz
> artifact.
> > > cluster looks healthy after upgrade. Lack of burrow lag suggests
> consumers
> > > are still happy, and incoming messages remains the same.
> > >
> > > On Mon, Jul 9, 2018 at 8:36 AM Dong Lin  wrote:
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the fourth candidate for release of Apache Kafka 1.1.1.
> > > >
> > > > Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was
> first
> > > > released with 1.1.0 about 3 months ago. We have fixed about 25 issues
> > > since
> > > > that release. A few of the more significant fixes include:
> > > >
> > > > KAFKA-6925  > > > > - Fix memory
> > > > leak in StreamsMetricsThreadImpl
> > > > KAFKA-6937  > > > > - In-sync
> > > > replica delayed during fetch if replica throttle is exceeded
> > > > KAFKA-6917  > > > > - Process txn
> > > > completion asynchronously to avoid deadlock
> > > > KAFKA-6893  > > > > - Create
> > > > processors before starting acceptor to avoid ArithmeticException
> > > > KAFKA-6870  > > > > -
> > > > Fix ConcurrentModificationException in SampledStat
> > > > KAFKA-6878  > > > > - Fix
> > > > NullPointerException when querying global state store
> > > > KAFKA-6879  > > > > - Invoke
> > > > session init callbacks outside lock to avoid Controller deadlock
> > > > KAFKA-6857  > > > > - Prevent
> > > > follower from truncating to the wrong offset if undefined leader
> epoch is
> > > > requested
> > > > KAFKA-6854  > > > > - Log cleaner
> > > > fails with transaction markers that are deleted during clean
> > > > KAFKA-6747  > > > > - Check
> > > > whether there is in-flight transaction before aborting transaction
> > > > KAFKA-6748  > > > > - Double
> > > > check before scheduling a new task after the punctuate call
> > > > KAFKA-6739  > > > > -
> > > > Fix IllegalArgumentException when down-converting from V2 to V0/V1
> > > > KAFKA-6728  > > > > -
> > > > Fix NullPointerException when instantiating the HeaderConverter
> > > >
> > > > Kafka 1.1.1 release plan:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
> > > >  Release+Plan+1.1.1>
> > > >
> > > > Release notes for the 1.1.1 release:
> > > > http://home.apache.org/~lindong/kafka-1.1.1-rc3/RELEASE_NOTES.html
> > > > 
> > > >
> > > > *** Please download, test and vote by Thursday, July 12, 12pm PT ***
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > http://kafka.apache.org/KEYS
> > > > 
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > http://home.apache.org/~lindong/kafka-1.1.1-rc3/
> > > > 
> > > >
> > > > * Maven artifacts to be voted upon:
> > > > https://repository.apache.org/content/groups/staging/
> > > > 
> > > 

Re: [VOTE] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

[Jason Gustafson]

+1. This is useful (though the naming inconsistencies in the tools are
vexing, as always).

-Jason

On Tue, Jul 17, 2018 at 12:24 PM, Dong Lin  wrote:

> Hi all,
>
> It seems that there is no further concern with the KIP-304. I guess it is
> time to start the voting thread.
>
> The KIP can be found at
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-340%
> 3A+Allow+kafka-reassign-partitions.sh+and+kafka-log-dirs.sh+
> to+take+admin+client+property+file
> . This KIP provides a way to allow kafka-reassign-partitions.sh and
> kafka-log-dirs.sh to talk to broker over SSL.
>
> Cheers,
> Dong
>

Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data Protocol

[Jason Gustafson]

Hey Vahid,

I'm with Mike that it seems simpler to just use the consumer group
generation. Even if you can figure out how to reason about the local
generation, it still seems confusing to have something called "generation"
which is not the consumer group generation. There doesn't seem to be much
downside to using the actual generation as far as I can tell and it is much
easier to reason about. I was thinking we could expose it in
PartitionAssignor.Assignment as an additional field. I think this wouldn't
impact existing assignors, though it would allow for other stateful
assignors to use the field in a similar way.

It's a pity we didn't add a version to this schema, by the way. There's not
much we can do if we can only add fields at the end of the schema, but I
guess we're hoping this is the last change ;-)

Thanks,
Jason

On Fri, Jul 13, 2018 at 10:07 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Mike,
>
>
>
> Thanks a lot for reviewing the KIP and sharing your feedback.
>
> I agree that such an issue could surface with option 1, but the existing
>
> PR (that currently implements this option) checks for such duplicate
>
> assignments and ignores one in favor of the other. So at the end there
>
> will be valid (non-duplicate) assignments to consumers, but they might
>
> deviate a bit from the ideal assignment.
>
> If rare scenarios like this are deemed troublesome and we want to avoid
>
> them, option 2 would probably be the way to go. In that case and in my
>
> opinion, option 2 would a better solution compared to introducing another
>
> field (e.g. timestamp).
>
>
>
> Regards.
>
> --Vahid
>
>
>
>
>
>
>
> From:   Mike Freyberger 
>
> To: "dev@kafka.apache.org" 
>
> Date:   07/13/2018 08:42 PM
>
> Subject:Re: [DISCUSS] KIP-341: Update Sticky Assignor's User Data
>
> Protocol
>
>
>
>
>
>
>
> This is great!
>
>
>
> For the client side implementation, I think it’s still possible for there
>
> to be a duplication. I’ll try to walk through the example here.
>
>
>
> Let’s says there are 2 consumers, 1 topic with 2 partitions.
>
>
>
> After the initial rebalance, generation 0:
>
> Consumer A has partition 0
>
> Consumer B has partition 1
>
>
>
> Let’s say consumer B leaves the group (long debug, GC pause). This leads
>
> to another rebalance. This rebalance will be considered generation 1 and
>
> will result in:
>
>
>
> Generation 1, Consumer A owns partition 0,1
>
>
>
> Now let’s say Consumer B is still out of the group and then Consumer A
>
> leaves as well. While Consumer A is out of the group, Consumer B rejoins
>
> the group. During this rebalance, the only previous state would be the
>
> initial generation 0 assignment. So this assignment would be considered
>
> generation 1 as well and would result in:
>
>
>
> Generation 1, Consumer B owns partition 0,1
>
>
>
> When A rejoins the group, both consumers would claim ownership of both
>
> partitions and they would report the assignment was from generation 1.
>
> This gets us back into the same issue as before because the generation
>
> number cannot help at all. You could add a timestamp in addition to the
>
> generation marker, but that’d still be vulnerable to clock skew.
>
>
>
> Would hooking into the existing generation marker protect the assignor for
>
> this kind of situation? We need to make sure the selected implantation is
>
> protected against the kind of failure mentioned above.
>
>
>
> Also, I have been working on KIP-315, which is another Sticky Assignor,
>
> which also requires some kind of epoch/generation marker to be protected
>
> against zombies. So, I’d be in favor of a generic solution here that other
>
> assignors can leverage.
>
>
>
> Best,
>
>
>
> Mike Freyberger
>
>
>
> > On Jul 13, 2018, at 6:15 PM, Vahid S Hashemian
>
>  wrote:
>
> >
>
> > Hi all,
>
> >
>
> > I create a short KIP to address an issue in Sticky Assignor assignment
>
> > logic:
>
> >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol
>
>
>
> > Please take a look and share your feedback / comments.
>
> >
>
> > In particular, there is a Generation Marker section (
>
> >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol#KIP-
> 341:UpdateStickyAssignor'sUserDataProtocol-GenerationMarker
>
>
>
> > ) that provides two methods for implementing the improvement to the
>
> > protocol. I'd like to know which option is more popular.
>
> >
>
> > Thanks!
>
> > --Vahid
>
> >
>
> >
>
>
>
>
>
>
>
>
>
>
>
>

Re: [VOTE] KIP-338 Support to exclude the internal topics in kafka-topics.sh command

[Jason Gustafson]

>
> (just a double check) the option name used in kip-338 is
> "--exclude-internal" rather than "--exclude-internal-topics". Do you prefer
> "--exclude-internal-topics"?


Nah, just a typo on my part. I think --exclude-internal is clear enough
given the name of the tool. Sorry for the confusion.

-Jason

On Tue, Jul 17, 2018 at 6:51 PM, Gwen Shapira  wrote:

> +1 (binding)
>
> Thank you for the improvement!
>
> On Mon, Jul 16, 2018 at 4:10 AM, Chia-Ping Tsai 
> wrote:
>
> > hi folks,
> >
> > The discussion[1] of KIP-338[2] did not get any objection for last 6 days
> > so it is time to start the voting thread.
> >
> > Thanks for your time!
> >
> > [1] https://lists.apache.org/thread.html/9bd4e61b73c901b51132ada49743b9
> > b703d40b85fc4eeaa5c9099900@%3Cdev.kafka.apache.org%3E
> >
> > [2] https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 338+Support+to+exclude+the+internal+topics+in+kafka-topics.sh+command
> >
> > Cheers,
> > chia-ping
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter  | blog
> 
>

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

[Jason Gustafson]

Hi Vahid,

Sorry for getting to this so late. I think there are two things here:

1. The use of "" as a groupId has always been a dubious practice at best.
We definitely ought to deprecate its use in the client. Perhaps in the next
major release, we can remove support completely. However, since older
clients depend on it, we may have to continue letting the broker support it
to some extent. Perhaps we just need to bump the OffsetCommit request API
and only accept the offset commit for older versions. You probably have to
do this anyway if you want to introduce the new error code since old
clients will not expect it.

2. There should be a way for the consumer to indicate that it has no group
id and will not commit offsets. This is an explicit instruction that the
consumer should not bother with coordinator lookup and such. We currently
have some brittle logic in place to let users avoid the coordinator lookup,
but it is a bit error-prone. I was hoping that we could change the default
value of group.id to be null so that the user had to take an explicit
action to opt into coordinator management (groups or offsets). However, it
is true that some users may be unknowingly depending on offset storage if
they are using both the default group.id and the default
enable.auto.commit. Perhaps one option is to disable enable.auto.commit
automatically if no group.id is specified? I am not sure if there are any
drawbacks, but my feeling is that implicit use of the empty group.id to
commit offsets is more likely to be causing users unexpected problems than
actually providing a useful capability.

Thoughts?

Thanks,
Jason




On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Viktor,
>
> Thanks for sharing your opinion.
> So you're in favor of disallowing the empty ("") group id altogether (even
> for fetching).
> Given that ideally no one should be using the empty group id (at least in
> a production setting) I think the impact would be minimal in either case.
>
> But as you said, let's hear what others think and I'd be happy to modify
> the KIP if needed.
>
> Regards.
> --Vahid
>
>
>
>
> From:   Viktor Somogyi 
> To: dev 
> Date:   05/28/2018 05:18 AM
> Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hi Vahid,
>
> (with the argument that using the default group id for offset commit
> should not be the user's intention in practice).
>
> Yea, so in my opinion too this use case doesn't seem too practical. Also I
> think breaking the offset commit is not smaller from this perspective than
> breaking fetch and offset fetch. If we suppose that someone uses the
> default group id and we break the offset commit then that might be harder
> to detect than breaking the whole thing altogether. (If we think about an
> upgrade situation.)
> So since we think it is not a practical use case, I think it would be
> better to break altogether but ofc that's just my 2 cents :). Let's gather
> other's input as well.
>
> Cheers,
> Viktor
>
> On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Hi Victor,
> >
> > Thanks for reviewing the KIP.
> >
> > Yes, to minimize the backward compatibility impact, there would be no
> harm
> > in letting a stand-alone consumer consume messages under a "" group id
> (as
> > long as there is no offset commit).
> > It would have to knowingly seek to an offset or rely on the
> > auto.offset.reset config for the starting offset.
> > This way the existing functionality would be preserved for the most part
> > (with the argument that using the default group id for offset commit
> > should not be the user's intention in practice).
> >
> > Does it seem reasonable?
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> > From:   Viktor Somogyi 
> > To: dev 
> > Date:   05/25/2018 04:56 AM
> > Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hi Vahid,
> >
> > When reading your KIP I coldn't fully understand why did you decide at
> > failing with "offset_commit" in case #2? Can't we fail with an empty
> group
> > id even in "fetch" or "fetch_offset"? What was the reason for deciding
> to
> > fail at "offset_commit"? Was it because of upgrade compatibility
> reasons?
> >
> > Thanks,
> > Viktor
> >
> > On Thu, May 24, 2018 at 12:06 AM, Ted Yu  wrote:
> >
> > > Looks good to me.
> > >  Original message From: Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com> Date: 5/23/18  11:19 AM  (GMT-08:00) To:
> > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289: Improve the
> default
> > > group id behavior in KafkaConsumer
> > > Hi Ted,
> > >
> > > Thanks for reviewing the KIP. I updated the KIP and introduced an
> error
> > > code for the scenario described.
> > >
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From:   Ted Yu 
> > > To: dev@kafka.apache.org
> > > Date:   04/27/2018 04:31 PM
> > > Subject:   

Re: [VOTE] KIP-322: Return new error code for DeleteTopics API when topic deletion disabled.

[Jason Gustafson]

Hey Manikumar,

This looks good. Do we have to keep the current behavior when handling old
DeleteTopics versions? I'm wondering if it would be better to return an
UNKNOWN_ERROR (say) and let the client raise that to the user instead of
just timing out.

Thanks,
Jason

On Fri, Jul 20, 2018 at 10:00 AM, Manikumar 
wrote:

> Bumping this thread.
>
> On Tue, Jul 10, 2018 at 10:33 PM Manikumar 
> wrote:
>
> > Waiting for one more binding vote to pass this minor KIP.  Appreciate
> your
> > vote.
> >
> > On Wed, Jul 4, 2018 at 7:03 PM Eno Thereska 
> > wrote:
> >
> >> +1 (non binding)
> >>
> >> On Wed, Jul 4, 2018 at 1:19 PM, Kamal Chandraprakash <
> >> kamal.chandraprak...@gmail.com> wrote:
> >>
> >> > +1 (non-binding)
> >> >
> >> > On Wed, Jul 4, 2018 at 5:22 PM Magnus Edenhill 
> >> wrote:
> >> >
> >> > > +1 (non-binding)
> >> > >
> >> > > 2018-07-04 13:40 GMT+02:00 Satish Duggana  >:
> >> > >
> >> > > > +1
> >> > > >
> >> > > > Thanks,
> >> > > > Satish.
> >> > > >
> >> > > > On Wed, Jul 4, 2018 at 4:11 PM, Daniele Ascione <
> >> ascion...@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > +1
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Daniele
> >> > > > >
> >> > > > > Il giorno mar 3 lug 2018 alle ore 23:55 Harsha  >
> >> ha
> >> > > > > scritto:
> >> > > > >
> >> > > > > > +1.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Harsha
> >> > > > > >
> >> > > > > > On Tue, Jul 3rd, 2018 at 9:22 AM, Ted Yu  >
> >> > > wrote:
> >> > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > +1
> >> > > > > > >
> >> > > > > > > On Tue, Jul 3, 2018 at 9:05 AM, Mickael Maison <
> >> > > > > > mickael.mai...@gmail.com >
> >> > > > > > >
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > +1 (non binding)
> >> > > > > > > > Thanks for the KIP
> >> > > > > > > >
> >> > > > > > > > On Tue, Jul 3, 2018 at 4:59 PM, Vahid S Hashemian
> >> > > > > > > > < vahidhashem...@us.ibm.com > wrote:
> >> > > > > > > > > +1 (non-binding)
> >> > > > > > > > >
> >> > > > > > > > > --Vahid
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > From: Gwen Shapira < g...@confluent.io >
> >> > > > > > > > > To: dev < dev@kafka.apache.org >
> >> > > > > > > > > Date: 07/03/2018 08:49 AM
> >> > > > > > > > > Subject: Re: [VOTE] KIP-322: Return new error code for
> >> > > > > > > > DeleteTopics
> >> > > > > > > > > API when topic deletion disabled.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > +1
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Jul 3, 2018 at 8:24 AM, Manikumar <
> >> > > > > > manikumar.re...@gmail.com >
> >> > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Manikumar < manikumar.re...@gmail.com >
> >> > > > > > > > >> Fri, Jun 29, 7:59 PM (4 days ago)
> >> > > > > > > > >> to dev
> >> > > > > > > > >> Hi All,
> >> > > > > > > > >>
> >> > > > > > > > >> I would like to start voting on KIP-322 which would
> >> return
> >> > new
> >> > > > > error
> >> > > > > > > > > code
> >> > > > > > > > >> for DeleteTopics API when topic deletion disabled.
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> >> > > > > > > > action?pageId=87295558
> >> > > > > > > > >
> >> > > > > > > > >>
> >> > > > > > > > >> Thanks,
> >> > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > --
> >> > > > > > > > > *Gwen Shapira*
> >> > > > > > > > > Product Manager | Confluent
> >> > > > > > > > > 650.450.2760 | @gwenshap
> >> > > > > > > > > Follow us: Twitter <
> >> > > > > > > > > https://twitter.com/ConfluentInc
> >> > > > > > > > >> | blog
> >> > > > > > > > > <
> >> > > > > > > > > http://www.confluent.io/blog
> >> > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [VOTE] KIP-322: Return new error code for DeleteTopics API when topic deletion disabled.

[Jason Gustafson]

After thinking about it, perhaps a better option is INVALID_REQUEST? I
think this is usually used for requests which are malformed in some way, so
it's a slight abuse, but at least it suggests an incompatibility between
what the client is requesting to do and what the server supports.
UNKNOWN_SERVER_ERROR, on the other hand, suggests an unexpected condition
on the broker.

-Jason

On Fri, Jul 20, 2018 at 10:57 AM, Manikumar 
wrote:

> Hi Jason,
>
> Yes, we can return UNKNOWN_SERVER_ERROR  for older versions.
> This will give better developer experience. Updated the KIP.
>
> Thanks,
>
> On Fri, Jul 20, 2018 at 11:02 PM Jason Gustafson 
> wrote:
>
> > Hey Manikumar,
> >
> > This looks good. Do we have to keep the current behavior when handling
> old
> > DeleteTopics versions? I'm wondering if it would be better to return an
> > UNKNOWN_ERROR (say) and let the client raise that to the user instead of
> > just timing out.
> >
> > Thanks,
> > Jason
> >
> > On Fri, Jul 20, 2018 at 10:00 AM, Manikumar 
> > wrote:
> >
> > > Bumping this thread.
> > >
> > > On Tue, Jul 10, 2018 at 10:33 PM Manikumar 
> > > wrote:
> > >
> > > > Waiting for one more binding vote to pass this minor KIP.  Appreciate
> > > your
> > > > vote.
> > > >
> > > > On Wed, Jul 4, 2018 at 7:03 PM Eno Thereska 
> > > > wrote:
> > > >
> > > >> +1 (non binding)
> > > >>
> > > >> On Wed, Jul 4, 2018 at 1:19 PM, Kamal Chandraprakash <
> > > >> kamal.chandraprak...@gmail.com> wrote:
> > > >>
> > > >> > +1 (non-binding)
> > > >> >
> > > >> > On Wed, Jul 4, 2018 at 5:22 PM Magnus Edenhill <
> mag...@edenhill.se>
> > > >> wrote:
> > > >> >
> > > >> > > +1 (non-binding)
> > > >> > >
> > > >> > > 2018-07-04 13:40 GMT+02:00 Satish Duggana <
> > satish.dugg...@gmail.com
> > > >:
> > > >> > >
> > > >> > > > +1
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Satish.
> > > >> > > >
> > > >> > > > On Wed, Jul 4, 2018 at 4:11 PM, Daniele Ascione <
> > > >> ascion...@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > +1
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > Daniele
> > > >> > > > >
> > > >> > > > > Il giorno mar 3 lug 2018 alle ore 23:55 Harsha <
> > ka...@harsha.io
> > > >
> > > >> ha
> > > >> > > > > scritto:
> > > >> > > > >
> > > >> > > > > > +1.
> > > >> > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > > Harsha
> > > >> > > > > >
> > > >> > > > > > On Tue, Jul 3rd, 2018 at 9:22 AM, Ted Yu <
> > yuzhih...@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > +1
> > > >> > > > > > >
> > > >> > > > > > > On Tue, Jul 3, 2018 at 9:05 AM, Mickael Maison <
> > > >> > > > > > mickael.mai...@gmail.com >
> > > >> > > > > > >
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > +1 (non binding)
> > > >> > > > > > > > Thanks for the KIP
> > > >> > > > > > > >
> > > >> > > > > > > > On Tue, Jul 3, 2018 at 4:59 PM, Vahid S Hashemian
> > > >> > > > > > > > < vahidhashem...@us.ibm.com > wrote:
> > > >> > > > > > > > > +1 (non-binding)
> > > >> > > > > > > > >
> > > >> > >

Re: [VOTE] KIP-322: Return new error code for DeleteTopics API when topic deletion disabled.

[Jason Gustafson]

Thanks, +1 from me.

On Fri, Jul 20, 2018 at 11:41 AM, Manikumar 
wrote:

> Thanks for the feedback, sounds good to me. Updated the KIP.
>
> On Sat, Jul 21, 2018 at 12:04 AM Jason Gustafson 
> wrote:
>
> > After thinking about it, perhaps a better option is INVALID_REQUEST? I
> > think this is usually used for requests which are malformed in some way,
> so
> > it's a slight abuse, but at least it suggests an incompatibility between
> > what the client is requesting to do and what the server supports.
> > UNKNOWN_SERVER_ERROR, on the other hand, suggests an unexpected condition
> > on the broker.
> >
> > -Jason
> >
> > On Fri, Jul 20, 2018 at 10:57 AM, Manikumar 
> > wrote:
> >
> > > Hi Jason,
> > >
> > > Yes, we can return UNKNOWN_SERVER_ERROR  for older versions.
> > > This will give better developer experience. Updated the KIP.
> > >
> > > Thanks,
> > >
> > > On Fri, Jul 20, 2018 at 11:02 PM Jason Gustafson 
> > > wrote:
> > >
> > > > Hey Manikumar,
> > > >
> > > > This looks good. Do we have to keep the current behavior when
> handling
> > > old
> > > > DeleteTopics versions? I'm wondering if it would be better to return
> an
> > > > UNKNOWN_ERROR (say) and let the client raise that to the user instead
> > of
> > > > just timing out.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Fri, Jul 20, 2018 at 10:00 AM, Manikumar <
> manikumar.re...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Bumping this thread.
> > > > >
> > > > > On Tue, Jul 10, 2018 at 10:33 PM Manikumar <
> > manikumar.re...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Waiting for one more binding vote to pass this minor KIP.
> > Appreciate
> > > > > your
> > > > > > vote.
> > > > > >
> > > > > > On Wed, Jul 4, 2018 at 7:03 PM Eno Thereska <
> > eno.there...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> +1 (non binding)
> > > > > >>
> > > > > >> On Wed, Jul 4, 2018 at 1:19 PM, Kamal Chandraprakash <
> > > > > >> kamal.chandraprak...@gmail.com> wrote:
> > > > > >>
> > > > > >> > +1 (non-binding)
> > > > > >> >
> > > > > >> > On Wed, Jul 4, 2018 at 5:22 PM Magnus Edenhill <
> > > mag...@edenhill.se>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > +1 (non-binding)
> > > > > >> > >
> > > > > >> > > 2018-07-04 13:40 GMT+02:00 Satish Duggana <
> > > > satish.dugg...@gmail.com
> > > > > >:
> > > > > >> > >
> > > > > >> > > > +1
> > > > > >> > > >
> > > > > >> > > > Thanks,
> > > > > >> > > > Satish.
> > > > > >> > > >
> > > > > >> > > > On Wed, Jul 4, 2018 at 4:11 PM, Daniele Ascione <
> > > > > >> ascion...@gmail.com>
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > +1
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > > Daniele
> > > > > >> > > > >
> > > > > >> > > > > Il giorno mar 3 lug 2018 alle ore 23:55 Harsha <
> > > > ka...@harsha.io
> > > > > >
> > > > > >> ha
> > > > > >> > > > > scritto:
> > > > > >> > > > >
> > > > > >> > > > > > +1.
> > > > > >> > > > > >
> > > > > >> > > > > > Thanks,
> > > > > >> > > > > > Harsha
> > > > > >> > > > > >
> > > > > >> > > > > > On Tue, Jul 3rd, 2018 at 9:22 AM, Ted Yu <
> > > > yuzhih...@gmail.com
> > > > > >
> > > > > >&

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

[Jason Gustafson]

Hey Vahid,

Thanks for the updates. Just to clarify, I was suggesting that we disable
enable.auto.commit only if no explicit group.id is configured. If an
explicit empty string is configured for the group id, then maybe we keep
the current behavior for compatibility. We can log a warning mentioning the
deprecation and we can use the old version of the OffsetCommit API that
allows the empty group id. In a later release, we can drop this support in
the client. Does that seem reasonable?

By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error code,
couldn't we use INVALID_GROUP_ID?

Thanks,
Jason



On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski  wrote:

> Hey Vahid,
>
> No I don't see an issue with it. I believe it to be the best approach.
>
> Best,
> Stanisav
>
> On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Hi Stanislav,
> >
> > Thanks for the feedback.
> > Do you see an issue with using `null` as the default group id (as
> > addressed by Jason in his response)?
> > This default group id would not support offset commits and consumers
> would
> > use `auto.offset.reset` config when there is no current offset.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> > From:   Stanislav Kozlovski 
> > To: dev@kafka.apache.org
> > Date:   07/20/2018 11:09 AM
> > Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > I agree with Jason's notion that
> > >  implicit use of the empty group.id to commit offsets is more likely
> to
> > be causing users unexpected problems than actually providing a useful
> > capability.
> > I was initially confused that this is the behavior when investigating a
> > new-ish JIRA issue <
> > https://issues.apache.org/jira/browse/KAFKA-6758
> > > about
> > the same topic.
> > So, +1 to deprecating "" as a group.id
> >
> > The question after that becomes what the *default* value should be -
> > should
> > we:
> > a) treat an unconfigured group.id consumer as a sort of intermittent
> > consumer where you don't store offsets at all (thereby making the user
> > explicitly sign up for them)
> > b) have a default value which makes use of them? I sort of like the
> > former.
> >
> > @Dhruvil, thinking about it at a high-level - yes. I can't think of a
> > situation where it makes sense to name something an empty string as far
> as
> > I'm aware - to me it seems like potential for confusion
> >
> >
> > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram  >
> > wrote:
> >
> > > +1 to deprecate use of "" as group.id since it is odd to have a
> resource
> > > name that you cannot set ACLs for. Agree, we have to support older
> > clients
> > > though.
> > >
> > > Thanks,
> > >
> > > Rajini
> > >
> > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson 
> > > wrote:
> > >
> > > > Hi Vahid,
> > > >
> > > > Sorry for getting to this so late. I think there are two things here:
> > > >
> > > > 1. The use of "" as a groupId has always been a dubious practice at
> > best.
> > > > We definitely ought to deprecate its use in the client. Perhaps in
> the
> > > next
> > > > major release, we can remove support completely. However, since older
> > > > clients depend on it, we may have to continue letting the broker
> > support
> > > it
> > > > to some extent. Perhaps we just need to bump the OffsetCommit request
> > API
> > > > and only accept the offset commit for older versions. You probably
> > have
> > > to
> > > > do this anyway if you want to introduce the new error code since old
> > > > clients will not expect it.
> > > >
> > > > 2. There should be a way for the consumer to indicate that it has no
> > > group
> > > > id and will not commit offsets. This is an explicit instruction that
> > the
> > > > consumer should not bother with coordinator lookup and such. We
> > currently
> > > > have some brittle logic in place to let users avoid the coordinator
> > > lookup,
> > > > but it is a bit error-prone. I was hoping that we could change the
> > > default
> > > > value of group.id to be null so that the user had to take an
> explicit
> > > >

***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

[Jason Gustafson]

Hey Vahid,

Sorry for the confusion. I think we all agree that going forward, we
shouldn't support the empty group id, so the question is just around
compatibility. I think we have to bump the OffsetCommit API version so that
old clients which are unknowingly depending on the default empty group id
will continue to work with new brokers. For new versions of the client, we
can either drop support for the empty group id immediately or we can give
users a grace period. I was thinking we would do the latter. We can change
the default group.id, but in the case that a user has explicitly configured
the empty group, then we can just use an old version of the OffsetCommit
API which still supports it. In a future release, we can drop this support
and only use the latest OffsetCommit version. Does that make sense?

Thanks,
Jason


On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Jason,
>
> Thanks for clarifying.
>
> So if we are going to continue supporting the empty group id as before
> (with only an addition of a deprecation warning), and disable
> enable.auto.commit for the new default (null) group id on the client side,
> do we really need to bump up the OffsetCommit version?
>
> You mentioned "If an explicit empty string is configured for the group id,
> then maybe we keep the current behavior for compatibility" which makes
> sense to me, but I find it in conflict with your earlier suggestion "we
> just need to bump the OffsetCommit request API and only accept the offset
> commit for older versions.". Maybe I'm missing something?
>
> Thanks!
> --Vahid
>
>
>
>
> From:   Jason Gustafson 
> To: dev 
> Date:   07/23/2018 10:52 PM
> Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hey Vahid,
>
> Thanks for the updates. Just to clarify, I was suggesting that we disable
> enable.auto.commit only if no explicit group.id is configured. If an
> explicit empty string is configured for the group id, then maybe we keep
> the current behavior for compatibility. We can log a warning mentioning
> the
> deprecation and we can use the old version of the OffsetCommit API that
> allows the empty group id. In a later release, we can drop this support in
> the client. Does that seem reasonable?
>
> By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error code,
> couldn't we use INVALID_GROUP_ID?
>
> Thanks,
> Jason
>
>
>
> On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
>  > wrote:
>
> > Hey Vahid,
> >
> > No I don't see an issue with it. I believe it to be the best approach.
> >
> > Best,
> > Stanisav
> >
> > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> >
> > > Hi Stanislav,
> > >
> > > Thanks for the feedback.
> > > Do you see an issue with using `null` as the default group id (as
> > > addressed by Jason in his response)?
> > > This default group id would not support offset commits and consumers
> > would
> > > use `auto.offset.reset` config when there is no current offset.
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From:   Stanislav Kozlovski 
> > > To: dev@kafka.apache.org
> > > Date:   07/20/2018 11:09 AM
> > > Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > I agree with Jason's notion that
> > > >  implicit use of the empty group.id to commit offsets is more likely
> > to
> > > be causing users unexpected problems than actually providing a useful
> > > capability.
> > > I was initially confused that this is the behavior when investigating
> a
> > > new-ish JIRA issue <
> > >
> https://issues.apache.org/jira/browse/KAFKA-6758
>
> > > > about
> > > the same topic.
> > > So, +1 to deprecating "" as a group.id
> > >
> > > The question after that becomes what the *default* value should be -
> > > should
> > > we:
> > > a) treat an unconfigured group.id consumer as a sort of intermittent
> > > consumer where you don't store offsets at all (thereby making the user
> > > explicitly sign up for them)
> > > b) have a default value which makes use of them? I sort of like the
> > > former.
> > >
> > > @Dhruvil, thinking about it at a high-level - yes. I can't think of a
>

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jason Gustafson]

Hi All,

I have made some updates to the KIP. As many of you know, a side project of
mine has been specifying the Kafka replication protocol in TLA. You can
check out the code here if you are interested:
https://github.com/hachikuji/kafka-specification. In addition to uncovering
a couple unknown bugs in the replication protocol (e.g.
https://issues.apache.org/jira/browse/KAFKA-7128), this has helped me
validate the behavior in this KIP. In fact, the original version I proposed
had a weakness. I initially suggested letting the leader validate the
expected epoch at the fetch offset. This made sense for the consumer in the
handling of unclean leader election, but it was not strong enough to
protect the follower in all cases. In order to make advancement of the high
watermark safe, for example, the leader actually needs to be sure that
every follower in the ISR matches its own epoch.

I attempted to fix this problem by treating the epoch in the fetch request
slightly differently for consumers and followers. For consumers, it would
be the expected epoch of the record at the fetch offset, and the leader
would raise a LOG_TRUNCATION error if the expectation failed. For
followers, it would be the current epoch and the leader would require that
it match its own epoch. This was unsatisfying both because of the
inconsistency in behavior and because the consumer was left with the weaker
fencing that we already knew was insufficient for the replicas. Ultimately
I decided that we should make the behavior consistent and that meant that
the consumer needed to act more like a following replica. Instead of
checking for truncation while fetching, the consumer should check for
truncation after leader changes. After checking for truncation, the
consumer can then use the current epoch when fetching and get the stronger
protection that it provides. What this means is that the Metadata API must
include the current leader epoch. Given the problems we have had around
stale metadata and how challenging they have been to debug, I'm convinced
that this is a good idea in any case and it resolves the inconsistent
behavior in the Fetch API. The downside is that there will be some
additional overhead upon leader changes, but I don't think it is a major
concern since leader changes are rare and the OffsetForLeaderEpoch request
is cheap.

This approach leaves the door open for some interesting follow up
improvements. For example, now that we have the leader epoch in the
Metadata request, we can implement similar fencing for the Produce API. And
now that the consumer can reason about truncation, we could consider having
a configuration to expose records beyond the high watermark. This would let
users trade lower end-to-end latency for weaker durability semantics. It is
sort of like having an acks=0 option for the consumer. Neither of these
options are included in this KIP, I am just mentioning them as potential
work for the future.

Finally, based on the discussion in this thread, I have added the
seekToCommitted API for the consumer. Please take a look and let me know
what you think.

Thanks,
Jason

On Fri, Jul 20, 2018 at 2:34 PM, Guozhang Wang  wrote:

> Hi Jason,
>
> The proposed API seems reasonable to me too. Could you please also update
> the wiki page (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 320%3A+Allow+fetchers+to+detect+and+handle+log+truncation)
> with a section say "workflow" on how the proposed API will be co-used with
> others to:
>
> 1. consumer callers handling a LogTruncationException.
> 2. consumer internals for handling a retriable UnknownLeaderEpochException.
>
>
> Guozhang
>
>
> On Tue, Jul 17, 2018 at 10:23 AM, Anna Povzner  wrote:
>
> > Hi Jason,
> >
> >
> > I also like your proposal and agree that KafkaConsumer#seekToCommitted()
> > is
> > more intuitive as a way to initialize both consumer's position and its
> > fetch state.
> >
> >
> > My understanding that KafkaConsumer#seekToCommitted() is purely for
> > clients
> > who store their offsets externally, right? And we are still going to
> > add KafkaConsumer#findOffsets()
> > in this KIP as we discussed, so that the client can handle
> > LogTruncationException?
> >
> >
> > Thanks,
> >
> > Anna
> >
> >
> > On Thu, Jul 12, 2018 at 3:57 PM Dong Lin  wrote:
> >
> > > Hey Jason,
> > >
> > > It is a great summary. The solution sounds good. I might have minor
> > > comments regarding the method name. But we can discuss that minor
> points
> > > later after we reach consensus on the high level API.
> > >
> > > Thanks,
> > > Dong
> > >
> > >
> > > On Thu, Jul 12, 2018 at 11:41 AM, Jason Gustafson 
> > > wrote:
> > >

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

[Jason Gustafson]

Hi Vahid,

I was thinking we'd only use the old API version if we had to. That is,
only if the user has explicitly configured "" as the group.id. Otherwise,
we'd just use the new one. Another option is to just drop support in the
client for the empty group id, but usually we allow a deprecation period
for changes like this.

-Jason

On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Jason,
>
> Thanks for additional clarification.
>
> So the next version of the OffsetCommit API will return an
> INVALID_GROUP_ID error for empty group ids; but on the client side we call
> the older version of the client until the next major release.
> The table below should summarize this.
>
>   +-+
>   | Client (group.id="")|
>   +-+
>   | pre-2.1 |   2.1  |   3.0|
> +-+---+-++--+
> | | V5 (cur.) | works   | works  | works|
> + API +---+-++--+
> | | V6| N/A | N/A (calls V5/warning) | INVALID_GROUP_ID |
> +-+---+-++--+
>
> Assumptions:
> * 2.1: The target release version for this KIP
> * 3.0: The next major release
>
> Please advise if you see an issue; otherwise, I'll update the KIP
> accordingly.
>
> Thanks!
> --Vahid
>
>
>
>
> From:   Jason Gustafson 
> To: dev 
> Date:   07/25/2018 12:08 AM
> Subject:***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve the default
> group idbehavior in KafkaConsumer
>
>
>
> Hey Vahid,
>
> Sorry for the confusion. I think we all agree that going forward, we
> shouldn't support the empty group id, so the question is just around
> compatibility. I think we have to bump the OffsetCommit API version so
> that
> old clients which are unknowingly depending on the default empty group id
> will continue to work with new brokers. For new versions of the client, we
> can either drop support for the empty group id immediately or we can give
> users a grace period. I was thinking we would do the latter. We can change
> the default group.id, but in the case that a user has explicitly
> configured
> the empty group, then we can just use an old version of the OffsetCommit
> API which still supports it. In a future release, we can drop this support
> and only use the latest OffsetCommit version. Does that make sense?
>
> Thanks,
> Jason
>
>
> On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Hi Jason,
> >
> > Thanks for clarifying.
> >
> > So if we are going to continue supporting the empty group id as before
> > (with only an addition of a deprecation warning), and disable
> > enable.auto.commit for the new default (null) group id on the client
> side,
> > do we really need to bump up the OffsetCommit version?
> >
> > You mentioned "If an explicit empty string is configured for the group
> id,
> > then maybe we keep the current behavior for compatibility" which makes
> > sense to me, but I find it in conflict with your earlier suggestion "we
> > just need to bump the OffsetCommit request API and only accept the
> offset
> > commit for older versions.". Maybe I'm missing something?
> >
> > Thanks!
> > --Vahid
> >
> >
> >
> >
> > From:   Jason Gustafson 
> > To: dev 
> > Date:   07/23/2018 10:52 PM
> > Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hey Vahid,
> >
> > Thanks for the updates. Just to clarify, I was suggesting that we
> disable
> > enable.auto.commit only if no explicit group.id is configured. If an
> > explicit empty string is configured for the group id, then maybe we keep
> > the current behavior for compatibility. We can log a warning mentioning
> > the
> > deprecation and we can use the old version of the OffsetCommit API that
> > allows the empty group id. In a later release, we can drop this support
> in
> > the client. Does that seem reasonable?
> >
> > By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error code,
> > couldn't we use INVALID_GROUP_ID?
> >
> > Thanks,
> > Jason
> >
> >
> >
> 

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

[Jason Gustafson]

Hey Colin,

The problem is both that the empty group id is the default value and that
it is actually accepted by the broker for offset commits. Combine that with
the fact that auto commit is enabled by default and you users get
surprising behavior. If you look at a random Kafka cluster, you'll probably
find a bunch of inadvertent offset commits for the empty group id. I was
hoping we could distinguish between users who are using the empty group id
as an accident of the default configuration and those who use it
intentionally. By default, there will be no group id and the consumer will
not commit offsets. If a user has actually intentionally used the empty
group id, however, it will continue to work. I actually think there are
probably very few people doing this (maybe even no one), but I thought we
might err on the side of compatibility.

The big incompatible change here is having brokers reject using assign(...)
> with empty / null group.id.


This is not correct. In the proposal, the broker will only reject the empty
group id for the new version of OffsetCommit. Older clients, which cannot
be changed, will continue to work because the old versions of the
OffsetCommit API still accept the empty group id. The null group id is
different from the empty group id: it is not allowed in any version of the
API. It is basically a way to indicate that the consumer has no dependence
on the coordinator at all, which we actually have a surprising number of
use cases for. Furthermore, if a user has an actual need for the empty
group id, it will still be allowed. We are just deprecating it.

-Jason

On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe  wrote:

> Sorry if this is a silly question, but what's the rationale for switching
> to using null for the default group id, rather than the empty string?
> Continuing to use the empty string seems like less churn.  And after all,
> we're not using the empty string group name for anything else.
>
> The big incompatible change here is having brokers reject using
> assign(...) with empty / null group.id.  If I understand correctly, the
> KIP proposes that this change be made on the brokers on the next
> incompatible Kafka release.  But that has nothing to do with client
> versions.  Why not just have a broker config which controls this?  Maybe "
> allow.assign.empty.group.id", or something like that.  At first, the
> default will be true, and then eventually we can flip it over to false.
>
> It seems like the main rationale for tying this behavior to the Kafka
> client version is to force people to stop using the empty group id so that
> they can upgrade their clients.  But it's also possible that people will
> stop upgrading their Kafka clients instead.  That would be pretty negative
> since  they'd miss out on any efficiency and feature improvements in the
> new clients and eventually have to do more protocol downgrading, etc.
>
> best,
> Colin
>
>
> On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> > Hi Jason,
> >
> > That makes sense.
> > I have updated the KIP based on the recent feedback.
> >
> > Thanks!
> > --Vahid
> >
> >
> >
> >
> > From:   Jason Gustafson 
> > To: dev 
> > Date:   07/25/2018 02:23 PM
> > Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> > behavior in KafkaConsumer
> >
> >
> >
> > Hi Vahid,
> >
> > I was thinking we'd only use the old API version if we had to. That is,
> > only if the user has explicitly configured "" as the group.id.
> Otherwise,
> > we'd just use the new one. Another option is to just drop support in the
> > client for the empty group id, but usually we allow a deprecation period
> > for changes like this.
> >
> > -Jason
> >
> > On Wed, Jul 25, 2018 at 12:49 PM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> >
> > > Hi Jason,
> > >
> > > Thanks for additional clarification.
> > >
> > > So the next version of the OffsetCommit API will return an
> > > INVALID_GROUP_ID error for empty group ids; but on the client side we
> > call
> > > the older version of the client until the next major release.
> > > The table below should summarize this.
> > >
> > > +-+
> > >   | Client (group.id="") |
> > > +-+
> > >   | pre-2.1 |   2.1  |   3.0 |
> > >
> > +-+---+-++--
> +
> > > | | V5 (c

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jason Gustafson]

Thanks Anna, you are right on both points. I updated the KIP.

-Jason

On Thu, Jul 26, 2018 at 2:08 PM, Anna Povzner  wrote:

> Hi Jason,
>
> Thanks for the update. I agree with the current proposal.
>
> Two minor comments:
> 1) In “API Changes” section, first paragraph says that “users can catch the
> more specific exception type and use the new `seekToNearest()` API defined
> below.”. Since LogTruncationException “will include the partitions that
> were truncated and the offset of divergence”., shouldn’t the client use
> seek(offset) to seek to the offset of divergence in response to the
> exception?
> 2) In “Protocol Changes” section, OffsetsForLeaderEpoch subsection says
> “Note
> that consumers will send a sentinel value (-1) for the current epoch and
> the broker will simply disregard that validation.”. Is that still true with
> MetadataResponse containing leader epoch?
>
> Thanks,
> Anna
>
> On Wed, Jul 25, 2018 at 1:44 PM Jason Gustafson 
> wrote:
>
> > Hi All,
> >
> > I have made some updates to the KIP. As many of you know, a side project
> of
> > mine has been specifying the Kafka replication protocol in TLA. You can
> > check out the code here if you are interested:
> > https://github.com/hachikuji/kafka-specification. In addition to
> > uncovering
> > a couple unknown bugs in the replication protocol (e.g.
> > https://issues.apache.org/jira/browse/KAFKA-7128), this has helped me
> > validate the behavior in this KIP. In fact, the original version I
> proposed
> > had a weakness. I initially suggested letting the leader validate the
> > expected epoch at the fetch offset. This made sense for the consumer in
> the
> > handling of unclean leader election, but it was not strong enough to
> > protect the follower in all cases. In order to make advancement of the
> high
> > watermark safe, for example, the leader actually needs to be sure that
> > every follower in the ISR matches its own epoch.
> >
> > I attempted to fix this problem by treating the epoch in the fetch
> request
> > slightly differently for consumers and followers. For consumers, it would
> > be the expected epoch of the record at the fetch offset, and the leader
> > would raise a LOG_TRUNCATION error if the expectation failed. For
> > followers, it would be the current epoch and the leader would require
> that
> > it match its own epoch. This was unsatisfying both because of the
> > inconsistency in behavior and because the consumer was left with the
> weaker
> > fencing that we already knew was insufficient for the replicas.
> Ultimately
> > I decided that we should make the behavior consistent and that meant that
> > the consumer needed to act more like a following replica. Instead of
> > checking for truncation while fetching, the consumer should check for
> > truncation after leader changes. After checking for truncation, the
> > consumer can then use the current epoch when fetching and get the
> stronger
> > protection that it provides. What this means is that the Metadata API
> must
> > include the current leader epoch. Given the problems we have had around
> > stale metadata and how challenging they have been to debug, I'm convinced
> > that this is a good idea in any case and it resolves the inconsistent
> > behavior in the Fetch API. The downside is that there will be some
> > additional overhead upon leader changes, but I don't think it is a major
> > concern since leader changes are rare and the OffsetForLeaderEpoch
> request
> > is cheap.
> >
> > This approach leaves the door open for some interesting follow up
> > improvements. For example, now that we have the leader epoch in the
> > Metadata request, we can implement similar fencing for the Produce API.
> And
> > now that the consumer can reason about truncation, we could consider
> having
> > a configuration to expose records beyond the high watermark. This would
> let
> > users trade lower end-to-end latency for weaker durability semantics. It
> is
> > sort of like having an acks=0 option for the consumer. Neither of these
> > options are included in this KIP, I am just mentioning them as potential
> > work for the future.
> >
> > Finally, based on the discussion in this thread, I have added the
> > seekToCommitted API for the consumer. Please take a look and let me know
> > what you think.
> >
> > Thanks,
> > Jason
> >
> > On Fri, Jul 20, 2018 at 2:34 PM, Guozhang Wang 
> wrote:
> >
> > > Hi Jason,
> > >
> > > The proposed API seems reasonable to me too. Could you pleas

Re: [VOTE] 2.0.0 RC3

[Jason Gustafson]

+1

I did the quickstart and verified the documentation (upgrade notes and
notable changes). Thanks Rajini!

On Fri, Jul 27, 2018 at 8:10 AM, Gwen Shapira  wrote:

> +1
>
> Quickstart on binaries, built from sources, verified signatures -- it
> really is Rajini. Thank you!
>
> Gwen
>
> On Thu, Jul 26, 2018 at 10:27 PM, Guozhang Wang 
> wrote:
>
> > +1.
> >
> > Validated the following:
> >
> > 1. quick start on binary (2.12)
> > 2. unit test on source
> > 3. javadoc
> > 4. web doc
> > 5. included jars (2.12).
> >
> >
> > Thanks Rajini!
> >
> >
> > Guozhang
> >
> >
> > On Wed, Jul 25, 2018 at 8:10 AM, Ron Dagostino 
> wrote:
> >
> > > +1 (non-binding)
> > >
> > > Built from source and exercised the new SASL/OAUTHBEARER functionality
> > with
> > > unsecured tokens.
> > >
> > > Thanks, Rajini -- apologies for KAFKA-7182.
> > >
> > > Ron
> > >
> > > On Tue, Jul 24, 2018 at 5:10 PM Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com>
> > > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Built from source and ran quickstart successfully with both Java 8
> and
> > > > Java 9 on Ubuntu.
> > > > Thanks Rajini!
> > > >
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > > From:   Rajini Sivaram 
> > > > To: dev , Users ,
> > > > kafka-clients 
> > > > Date:   07/24/2018 08:33 AM
> > > > Subject:[VOTE] 2.0.0 RC3
> > > >
> > > >
> > > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > >
> > > > This is the fourth candidate for release of Apache Kafka 2.0.0.
> > > >
> > > >
> > > > This is a major version release of Apache Kafka. It includes 40 new
> > KIPs
> > > > and
> > > >
> > > > several critical bug fixes. Please see the 2.0.0 release plan for
> more
> > > > details:
> > > >
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=80448820
> > > >
> > > >
> > > >
> > > > A few notable highlights:
> > > >
> > > >- Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for
> > CreateTopics
> > > >(KIP-277)
> > > >- SASL/OAUTHBEARER implementation (KIP-255)
> > > >- Improved quota communication and customization of quotas
> (KIP-219,
> > > >KIP-257)
> > > >- Efficient memory usage for down conversion (KIP-283)
> > > >- Fix log divergence between leader and follower during fast
> leader
> > > >failover (KIP-279)
> > > >- Drop support for Java 7 and remove deprecated code including old
> > > > scala
> > > >clients
> > > >- Connect REST extension plugin, support for externalizing secrets
> > and
> > > >improved error handling (KIP-285, KIP-297, KIP-298 etc.)
> > > >- Scala API for Kafka Streams and other Streams API improvements
> > > >(KIP-270, KIP-150, KIP-245, KIP-251 etc.)
> > > >
> > > >
> > > > Release notes for the 2.0.0 release:
> > > >
> > > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc3/RELEASE_NOTES.html
> > > >
> > > >
> > > >
> > > > *** Please download, test and vote by Friday July 27, 4pm PT.
> > > >
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > >
> > > > http://kafka.apache.org/KEYS
> > > >
> > > >
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > >
> > > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc3/
> > > >
> > > >
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> > > > https://repository.apache.org/content/groups/staging/
> > > >
> > > >
> > > >
> > > > * Javadoc:
> > > >
> > > > http://home.apache.org/~rsivaram/kafka-2.0.0-rc3/javadoc/
> > > >
> > > >
> > > >
> > > > * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
> > > >
> > > > https://github.com/apache/kafka/releases/tag/2.0.0-rc3
> > > >
> > > >
> > > > * Documentation:
> > > >
> > > > http://kafka.apache.org/20/documentation.html
> > > >
> > > >
> > > >
> > > > * Protocol:
> > > >
> > > > http://kafka.apache.org/20/protocol.html
> > > >
> > > >
> > > >
> > > > * Successful Jenkins builds for the 2.0 branch:
> > > >
> > > > Unit/integration tests:
> > > > https://builds.apache.org/job/kafka-2.0-jdk8/90/
> > > >
> > > >
> > > > System tests:
> > > > https://jenkins.confluent.io/job/system-test-kafka/job/2.0/41/
> > > >
> > > >
> > > >
> > > > /**
> > > >
> > > >
> > > > Thanks,
> > > >
> > > >
> > > >
> > > > Rajini
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter  | blog
> 
>

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

Hey Boyang,

Thanks for the KIP. I think my main question is in the same vein as James'.
The problem is that the coordinator needs to be able to identify which
instance of a particular memberId is the active one. For EOS, each
transactionalId gets an epoch. When a new producer is started, it bumps the
epoch which allows the transaction coordinator to fence off any zombie
instances which may try to continue doing work with the old epoch. It seems
like we need a similar protection for consumer members.

Suppose for example that we distinguish between a registration id which is
provided by the user and a member id which is assigned uniquely by the
coordinator. In the JoinGroup request, both the registration id and the
member id are provided. When a consumer is first started, it doesn't know
the memberId, so it it provides only the registration id. The coordinator
can then assign a new memberId and invalidate the previous one that was
associated with the registration id. This would then fence off the previous
instance which was still trying to use the member id.

Taking a little bit of a step back, I think the main observation in this
KIP is that applications with heavy local state need to have a strong bias
toward being able to reuse that state. It is a bit like Kafka itself in the
sense that a replica is not moved just because the broker is shutdown as
the cost of moving the log is extremely high. I'm wondering if we need to
think about streams applications in a similar way. Should there be a static
notion of the members of the group so that streams can make rebalancing
decisions more easily without depending so heavily on transient membership?
I feel the hacks we've put in place in some cases to avoid rebalances are a
bit brittle. Delaying group joining for example is an example of this. If
you knew ahead of time who the stable members of the group were, then this
would not be needed. Anyway, just a thought.

Thanks,
Jason



On Fri, Jul 27, 2018 at 1:58 PM, James Cheng  wrote:

> When you say that it will "break", what does this breakage look like? Will
> the consumer-group be non-functional? Will just those instances be
> non-functional? Or will the group be functional, but the rebalancing be
> non-optimal and require more round-trips/data-transfer? (similar to the
> current algorithm)
>
> I'm trying to assess the potential for user-error and the impact of
> user-error.
>
> -James
>
> > On Jul 27, 2018, at 11:25 AM, Boyang Chen  wrote:
> >
> > Hey James,
> >
> >
> > the algorithm is relying on client side to provide unique consumer
> member id. It will break unless we enforce some sort of validation (host +
> port) on the server side. To simplify the first version, we do not plan to
> enforce validation. A good comparison would be the EOS producer which is in
> charge of generating unique transaction id sequence. IMO for broker logic,
> the tolerance of client side error is not unlimited.
> >
> >
> > Thank you!
> >
> >
> > 
> > From: James Cheng 
> > Sent: Saturday, July 28, 2018 1:26 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
> >
> >
> >> On Jul 26, 2018, at 11:09 PM, Guozhang Wang  wrote:
> >>
> >> Hi Boyang,
> >>
> >> Thanks for the proposed KIP. I made a pass over the wiki and here are
> some
> >> comments / questions:
> >>
> >> 1. In order to preserve broker compatibility, we need to make sure the
> >> broker version discovery logic can be integrated with this new logic.
> I.e.
> >> if a newer versioned consumer is talking to an older versioned broker
> who
> >> does not recognize V4, the client needs to downgrade its
> JoinGroupRequest
> >> version to V3 and not setting the member-id specifically. You can take a
> >> look at the ApiVersionsRequest and see how to work with it.
> >>
> >> 2. There may exist some manners to validate that two different clients
> do
> >> not send with the same member id, for example if we pass along the
> >> host:port information from KafkaApis to the GroupCoordinator interface.
> But
> >> I think this is overly complicated the logic and may not worthwhile than
> >> relying on users to specify unique member ids.
> >
> > Boyang,
> >
> > Thanks for the KIP! How will the algorithm behave if multiple consumers
> provide the same member id?
> >
> > -James
> >
> >> 3. Minor: you would need to bumping up the version of JoinGroupResponse
> to
> >> V4 as well.
> >>
> >> 4. Minor: in the wiki page, you need to specify the actual string value
> for
> >> `MEMBER_ID`, for example "member.id".
> >>
> >> 5. When this additional config it specified by users, we should consider
> >> setting the default of internal `LEAVE_GROUP_ON_CLOSE_CONFIG` to false,
> >> since otherwise its effectiveness would be less.
> >>
> >>
> >> Guozhang
> >>
> >>
> >>
> >>> On Thu, Jul 26, 2018 at 9:20 PM, Boyang Chen 
> wrote:
> >>>
> >>> Hey friends,
> >>>
> >>>
> >>> I would like to open a di

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jason Gustafson]

Hey Dong,

Thanks for the detailed review. Responses below:

1/2: Thanks for noticing the inconsistency. Would it be reasonable to
simply call it LeaderEpoch for both APIs?

3: I agree it should be a map. I will update.

4: Fair point. I think we should always be able to identify an offset.
Let's remove the Optional for now and reconsider if we find an unhandled
case during implementation.

5: Yeah, I was thinking about this. The two error codes could be handled
similarly, so we might merge them. Mainly I was thinking that it will be
useful for consumers/replicas to know whether they are ahead or behind the
leader. For example, if a consumer sees UNKNOWN_LEADER_EPOCH, it need not
refresh metadata. Or if a replica sees a FENCED_LEADER_EPOCH error, it
could just stop fetching and await the LeaderAndIsr request that it is
missing. It probably also makes debugging a little bit easier. I guess I'm
a bit inclined to keep both error codes, but I'm open to reconsideration if
you feel strongly. Another point to consider is whether we should continue
using NOT_LEADER_FOR_PARTITION if a follower receives an unexpected fetch.
The leader epoch would be different in this case so we could use one of the
invalid epoch error codes instead since they contain more information.

6: I agree the name is not ideal in that scenario. What if we overloaded
`seek`?

7: Sure, I will mention this.


Thanks,
Jason

On Fri, Jul 27, 2018 at 6:17 PM, Dong Lin  wrote:

> Hey Jason,
>
> Thanks for the update! I agree with the current proposal overall. I have
> some minor comments related to naming etc.
>
> 1) I am not strong and will just leave it here for discussion. Would it be
> better to rename "CurrentLeaderEpoch" to "ExpectedLeaderEpoch" for the new
> field in the OffsetsForLeaderEpochRequest? The reason is that
> "CurrentLeaderEpoch" may not necessarily be true current leader epoch if
> the consumer has stale metadata. "ExpectedLeaderEpoch" shows that this
> epoch is what consumer expects on the broker which may or may not be the
> true value.
>
> 2) Currently we add the field "LeaderEpoch" to FetchRequest and the field
> "CurrentLeaderEpoch" to OffsetsForLeaderEpochRequest. Given that both
> fields are compared with the leaderEpoch in the broker, would it be better
> to give them the same name?
>
> 3) Currently LogTruncationException.truncationOffset() returns
> Optional to user. Should it return
> Optional> to handle the scenario
> where leaderEpoch of multiple partitions are different from the leaderEpoch
> in the broker?
>
> 4) Currently LogTruncationException.truncationOffset() returns an Optional
> value. Could you explain a bit more when it will return Optional.empty()? I
> am trying to understand whether it is simpler and reasonable to
> replace Optional.empty()
> with OffsetMetadata(offset=last_fetched_offset, leaderEpoch=-1).
>
> 5) Do we also need to add a new retriable exception for error code
> FENCED_LEADER_EPOCH? And do we need to define both FENCED_LEADER_EPOCH
> and UNKNOWN_LEADER_EPOCH.
> It seems that the current KIP uses these two error codes in the same way
> and the exception for these two error codes is not exposed to the user.
> Maybe we should combine them into one error, e.g. INVALID_LEADER_EPOCH?
>
> 6) For users who has turned off auto offset reset, when consumer.poll()
> throw LogTruncationException, it seems that user will most likely call
> seekToCommitted(offset,
> leaderEpoch) where offset and leaderEpoch are obtained from
> LogTruncationException.truncationOffset(). In this case, the offset used
> here is not committed, which is inconsistent from the method name
> seekToCommitted(...). Would it be better to rename the method to e.g.
> seekToLastConsumedMessage()?
>
> 7) Per point 3 in Jun's comment, would it be useful to explicitly specify
> in the KIP that we will log the truncation event if user has turned on auto
> offset reset policy?
>
>
> Thanks,
> Dong
>
>
> On Fri, Jul 27, 2018 at 12:39 PM, Jason Gustafson 
> wrote:
>
> > Thanks Anna, you are right on both points. I updated the KIP.
> >
> > -Jason
> >
> > On Thu, Jul 26, 2018 at 2:08 PM, Anna Povzner  wrote:
> >
> > > Hi Jason,
> > >
> > > Thanks for the update. I agree with the current proposal.
> > >
> > > Two minor comments:
> > > 1) In “API Changes” section, first paragraph says that “users can catch
> > the
> > > more specific exception type and use the new `seekToNearest()` API
> > defined
> > > below.”. Since LogTruncationException “will include the partitions that
> > > were truncated and the offset of divergence”., shouldn’t the client use
> >

Re: [DISCUSS] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Jason Gustafson]

Hey Stanislav,

Thanks for the KIP. I think the goal is to allow users to seek past a
records which cannot be parsed for whatever reason. However, it's a little
annoying that you need to catch two separate types to handle this. I'm
wondering if it makes sense to expose an interface like
`UnconsumableRecordException` or something like that. The consumer could
then have separate internal exception types which extend from
InvalidRecordException and SerializationException respectively and
implement `UnconsumableRecordException`. That would simplify the handling
and users could check the cause if they cared which case it was.

Another question for consideration. I'd imagine some users would find it
helpful to seek past failed messages automatically. If there is a corrupt
record, for example, there's almost nothing you can do except seek past it
anyway. I'm wondering if there should be a config for this or if users
should be able to install a callback of some sorts to handle failed
records. Not sure if this is that big of a problem for users, but
interested to hear others thoughts.

Thanks,
Jason

On Fri, Jul 20, 2018 at 6:32 PM, Stanislav Kozlovski  wrote:

> Hi Ted,
>
> I do plan to start one. When is the appropriate time? My reasoning was that
> people would like to view the changes first
>
> On Fri, Jul 20, 2018, 6:21 PM Ted Yu  wrote:
>
> > Hi, Stanislav:
> > Do you plan to start VOTE thread ?
> >
> > Cheers
> >
> > On Fri, Jul 20, 2018 at 6:11 PM Stanislav Kozlovski <
> > stanis...@confluent.io>
> > wrote:
> >
> > > Hey group,
> > >
> > > I added a Pull Request for this KIP - here it is
> > > https://github.com/apache/kafka/pull/5410
> > > Please take a look.
> > >
> > > Best,
> > > Stanislav
> > >
> > > On Thu, Jul 5, 2018 at 11:06 AM Ismael Juma  wrote:
> > >
> > > > Yes, the Scala consumers have been removed in 2.0.0, which simplifies
> > > some
> > > > of this. The following commit was an initial step in unifying the
> > > exception
> > > > handling:
> > > >
> > > >
> > > >
> > >
> > https://github.com/apache/kafka/commit/96bcfdfc7c9aac075635b2034e65e4
> 12a725672e
> > > >
> > > > But more can be done as you mentioned.
> > > >
> > > > Ismael
> > > >
> > > > On 5 Jul 2018 9:36 am, "Stanislav Kozlovski"  >
> > > > wrote:
> > > >
> > > > Hey Ismael,
> > > >
> > > > It is only slightly related - my PR would attach two new attributes
> and
> > > > also touch upon deserialization exceptions.
> > > >
> > > > But this PR did provide me with some insight:
> > > > Maybe the best approach would be to make `InvalidRecordException` a
> > > public
> > > > exception instead of introducing a new one - I did not realize it was
> > not
> > > > publicly exposed.
> > > > Does the following:
> > > >
> > > >  InvalidMessageException extends CorruptRecordException for temporary
> > > > compatibility with the old Scala clients.
> > > >  * We want to update the server side code to use and catch the new
> > > > CorruptRecordException.
> > > >  * Because ByteBufferMessageSet.scala and Message.scala are used in
> > > > both server and client code having
> > > >  * InvalidMessageException extend CorruptRecordException allows us to
> > > > change server code without affecting the client.
> > > >
> > > > still apply? I can see that the `ByteBufferMessageSet` and `Message`
> > > scala
> > > > classes are not present in the codebase anymore. AFAIA the old scala
> > > > clients were removed with 2.0.0 and we can thus update the server
> side
> > > code
> > > > to use the `CorruptRecordException` while changing and exposing
> > > > `InvalidRecordException` to the public. WDYT?
> > > >
> > > > I will also make sure to not expose the cause of the exception when
> not
> > > > needed, maybe I'll outright remove the `cause` attribute
> > > >
> > > >
> > > > On Thu, Jul 5, 2018 at 4:55 PM Ismael Juma 
> wrote:
> > > >
> > > > > Thanks for the KIP, Stanislav. The following PR looks related:
> > > > >
> > > > > https://github.com/apache/kafka/pull/4093/files
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Thu, Jul 5, 2018 at 8:44 AM Stanislav Kozlovski <
> > > > stanis...@confluent.io
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hey everybody,
> > > > > >
> > > > > > I just created a new KIP about exposing more information in
> > > exceptions
> > > > > > caused by consumer record deserialization/validation. Please
> have a
> > > > look
> > > > > at
> > > > > > it, it is a very short page.
> > > > > >
> > > > > > I am working under the assumption that all invalid record or
> > > > > > deserialization exceptions in the consumer pass through the
> > `Fetcher`
> > > > > > class. Please confirm if that is true, otherwise I might miss
> some
> > > > places
> > > > > > where the exceptions are raised in my implementation
> > > > > >
> > > > > > One concern I have is the name of the second exception -
> > > > > > `InoperativeRecordException`. I would have named it
> > > > > > `InvalidRecordException` but that is taken. The `Fetcher` class
> > >

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

ne the current architecture. We need also
> consider flexibility introduced by this KIP for cloud users to coordinate
> consumer/stream instances more freely. Honestly, I'm interested in Jason's
> registration id proposal and open to more voices, but I feel it would be
> more complex than the current KIP for user to understand. Hope this makes
> sense, Jason.
>
>
> Thank you again for the feedback!
>
>
> Best,
>
> Boyang
>
>
> 
> From: Jason Gustafson 
> Sent: Saturday, July 28, 2018 6:50 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hey Boyang,
>
> Thanks for the KIP. I think my main question is in the same vein as James'.
> The problem is that the coordinator needs to be able to identify which
> instance of a particular memberId is the active one. For EOS, each
> transactionalId gets an epoch. When a new producer is started, it bumps the
> epoch which allows the transaction coordinator to fence off any zombie
> instances which may try to continue doing work with the old epoch. It seems
> like we need a similar protection for consumer members.
>
> Suppose for example that we distinguish between a registration id which is
> provided by the user and a member id which is assigned uniquely by the
> coordinator. In the JoinGroup request, both the registration id and the
> member id are provided. When a consumer is first started, it doesn't know
> the memberId, so it it provides only the registration id. The coordinator
> can then assign a new memberId and invalidate the previous one that was
> associated with the registration id. This would then fence off the previous
> instance which was still trying to use the member id.
>
> Taking a little bit of a step back, I think the main observation in this
> KIP is that applications with heavy local state need to have a strong bias
> toward being able to reuse that state. It is a bit like Kafka itself in the
> sense that a replica is not moved just because the broker is shutdown as
> the cost of moving the log is extremely high. I'm wondering if we need to
> think about streams applications in a similar way. Should there be a static
> notion of the members of the group so that streams can make rebalancing
> decisions more easily without depending so heavily on transient membership?
> I feel the hacks we've put in place in some cases to avoid rebalances are a
> bit brittle. Delaying group joining for example is an example of this. If
> you knew ahead of time who the stable members of the group were, then this
> would not be needed. Anyway, just a thought.
>
> Thanks,
> Jason
>
>
>
> On Fri, Jul 27, 2018 at 1:58 PM, James Cheng  wrote:
>
> > When you say that it will "break", what does this breakage look like?
> Will
> > the consumer-group be non-functional? Will just those instances be
> > non-functional? Or will the group be functional, but the rebalancing be
> > non-optimal and require more round-trips/data-transfer? (similar to the
> > current algorithm)
> >
> > I'm trying to assess the potential for user-error and the impact of
> > user-error.
> >
> > -James
> >
> > > On Jul 27, 2018, at 11:25 AM, Boyang Chen  wrote:
> > >
> > > Hey James,
> > >
> > >
> > > the algorithm is relying on client side to provide unique consumer
> > member id. It will break unless we enforce some sort of validation (host
> +
> > port) on the server side. To simplify the first version, we do not plan
> to
> > enforce validation. A good comparison would be the EOS producer which is
> in
> > charge of generating unique transaction id sequence. IMO for broker
> logic,
> > the tolerance of client side error is not unlimited.
> > >
> > >
> > > Thank you!
> > >
> > >
> > > 
> > > From: James Cheng 
> > > Sent: Saturday, July 28, 2018 1:26 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> > specifying member id
> > >
> > >
> > >> On Jul 26, 2018, at 11:09 PM, Guozhang Wang 
> wrote:
> > >>
> > >> Hi Boyang,
> > >>
> > >> Thanks for the proposed KIP. I made a pass over the wiki and here are
> > some
> > >> comments / questions:
> > >>
> > >> 1. In order to preserve broker compatibility, we need to make sure the
> > >> broker version discovery logic can be integrated with this new logic.
> &g

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

[Jason Gustafson]

tarted, one
> rebalance is
> executed to shuffle all assignment, rather than just a "partial"
> shuffling
> of some of the members.
> 4. Application scale in: similarly to 3), when an existing instance
> gracefully shutdown, once rebalance is executed to shuffle all
> assignment.
> 5. Application instance bounce (upgrade, config change etc): one
> instance
> shut down and then restart, it will trigger two rebalances. NOTE that
> disabling leave-group is targeted for this issue.
> 6. Application instance failure: one instance failed, and probably a
> new
> instance start to take its assignment (e.g. k8s), it will trigger two
> rebalances. The different with 3) above is that new instance would not
> have
> local cached tasks.
>
>
> Among them, I think 1/2/5/6 could potentially be grouped together as
> "static membership"; 4/5 could be grouped as another category, of
> allowing
> "incremental rebalance" or "partial rebalance" than full-rebalance. Of
> course, having incremental rebalances can help on 1/2/5/6 as well to
> reduce
> the cost of each unnecessary rebalance, but ideally we want NO
> rebalances
> at all for these cases, which will be more true with k8s / etc
> integrations
> or static memberships.
>
> --
>
> So just to throw in a sketchy idea following this route for 1/2/5/6 for
> brainstorming kick-off:
>
>
> 1. We bump up the JoinGroupRequest with additional fields:
>
>   1.a) a flag indicating "static" or "dynamic" membership protocols.
>   1.b) with "static" membership, we also add the pre-defined member id.
>   1.c) with "static" membership, we also add an optional
> "group-change-timeout" value.
>
> 2. On the broker side, we enforce only one of the two protocols for all
> group members: we accept the protocol on the first joined member of the
> group, and if later joining members indicate a different membership
> protocol, we reject it. If the group-change-timeout value was
> different to
> the first joined member, we reject it as well.
>
> 3. With dynamic membership, nothing is changed; with static
> membership, we
> do the following:
>
>   3.a) never assign member ids, instead always expect the joining
> members
> to come with their own member id; we could do the fencing based on
> host /
> port here.
>   3.b) upon receiving the first join group request, use the
> "group-change-timeout" instead of the session-timeout as rebalance
> timeout
> to wait for other members to join. This is for 1) above.
>   3.c) upon receiving a leave-group request, use the
> "group-change-timeout"
> to wait for more members to leave group as well, or for the left
> members to
> re-join. After the timeout we trigger a rebalance with whatever have
> left
> in the members list. This is for all 2 (expecting other members to send
> leave-group) and 5/6 (expecting the left member to re-join).
>
> 4. As a result, we will deprecate KIP-134 and disable-on-leave-group as
> well.
>
>
> The key idea is that, with "static" membership, groups should be
> created or
> terminated as a whole, and dynamic member changes are not expected
> often.
> Hence we would not react to those membership-changing events
> immediately
> but wait for longer specified time expecting some other systems like
> k8s
> will resume the group members. WDYT?
>
>
> Guozhang
>
>
> On Mon, Jul 30, 2018 at 3:05 PM, Mike Freyberger <
> mfreyber...@appnexus.com>
> wrote:
>
> > Jason,
> >
> > I really appreciate the broader conversation that you are bringing
> up here.
> >
> > I've been working on an application that does streaming joins for a
> while
> > now, and we face a similar issue with group membership being
> dynamic. We
> > are currently using our own StickyAssignor and take special care
> during
> > rolling restarts to make sure consumer assignments do not change.
> >
> > I think a feature that allows for group membership to be fixed,
> along with
> > a CLI for adding or removing a node from the group be ideal. This
> reminds
> > me of some of the work by the DynamoDB team about 10 years back when
> they
> > differentiated transient failures from permanent failures to deal
> with this
> > problems like this.
> >
> >

Re: [DISCUSS] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Jason Gustafson]

Hey Stanislav,

Just to make sure I understood you right - you propose not exposing any new
> exception types but rather the interface itself only?


Yes, exactly. Our exception hierarchy is a bit of a mess to be honest.
Interfaces are more flexible and here it simplifies the error handling.

Regardless, I believe this is best left out for another KIP as I feel it
> would warrant a bigger discussion


Ok, that's fair. I thought I'd suggest it here just to see if there was any
interest in the community. At least with this KIP, users have a viable way
to skip past bad data if they wish.

-Jason

On Tue, Jul 31, 2018 at 2:42 AM, Stanislav Kozlovski  wrote:

> Hey Jason,
>
> Just to make sure I understood you right - you propose not exposing any new
> exception types but rather the interface itself only? So a sample code
> dealing with this would be something like:
>
> try {
> // ...
> } catch (KafkaException e) {
> if (e instanceof UnconsumableRecordException) {
>   // handle retry
> }
> }
>
> If that is the case, I like it better.
>
>
> In regards to automatic handling of unconsumable messages - I like that
> idea too. To me, a callback seems like the more straightforward approach. A
> config such as `seek.past.unconsumable.record` limits the behavior too
> much
> in my opinion, I believe giving them the option to implement a (or use the
> default) callback is better in that way.
> Regardless, I believe this is best left out for another KIP as I feel it
> would warrant a bigger discussion
>
> Best,
> Stanislav
>
> On Mon, Jul 30, 2018 at 9:34 PM Jason Gustafson 
> wrote:
>
> > Hey Stanislav,
> >
> > Thanks for the KIP. I think the goal is to allow users to seek past a
> > records which cannot be parsed for whatever reason. However, it's a
> little
> > annoying that you need to catch two separate types to handle this. I'm
> > wondering if it makes sense to expose an interface like
> > `UnconsumableRecordException` or something like that. The consumer could
> > then have separate internal exception types which extend from
> > InvalidRecordException and SerializationException respectively and
> > implement `UnconsumableRecordException`. That would simplify the handling
> > and users could check the cause if they cared which case it was.
> >
> > Another question for consideration. I'd imagine some users would find it
> > helpful to seek past failed messages automatically. If there is a corrupt
> > record, for example, there's almost nothing you can do except seek past
> it
> > anyway. I'm wondering if there should be a config for this or if users
> > should be able to install a callback of some sorts to handle failed
> > records. Not sure if this is that big of a problem for users, but
> > interested to hear others thoughts.
> >
> > Thanks,
> > Jason
> >
> > On Fri, Jul 20, 2018 at 6:32 PM, Stanislav Kozlovski <
> > stanis...@confluent.io
> > > wrote:
> >
> > > Hi Ted,
> > >
> > > I do plan to start one. When is the appropriate time? My reasoning was
> > that
> > > people would like to view the changes first
> > >
> > > On Fri, Jul 20, 2018, 6:21 PM Ted Yu  wrote:
> > >
> > > > Hi, Stanislav:
> > > > Do you plan to start VOTE thread ?
> > > >
> > > > Cheers
> > > >
> > > > On Fri, Jul 20, 2018 at 6:11 PM Stanislav Kozlovski <
> > > > stanis...@confluent.io>
> > > > wrote:
> > > >
> > > > > Hey group,
> > > > >
> > > > > I added a Pull Request for this KIP - here it is
> > > > > https://github.com/apache/kafka/pull/5410
> > > > > Please take a look.
> > > > >
> > > > > Best,
> > > > > Stanislav
> > > > >
> > > > > On Thu, Jul 5, 2018 at 11:06 AM Ismael Juma 
> > wrote:
> > > > >
> > > > > > Yes, the Scala consumers have been removed in 2.0.0, which
> > simplifies
> > > > > some
> > > > > > of this. The following commit was an initial step in unifying the
> > > > > exception
> > > > > > handling:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > https://github.com/apache/kafka/commit/
> 96bcfdfc7c9aac075635b2034e65e4
> > > 12a725672e
> > > > > >
> > > > > > But more can be done as you mentioned.
> > > > > >

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

[Jason Gustafson]

Hey Vahid,

I talked with Colin offline. I think specifically he felt the version bump
on the broker was overkill since the broker still has to support the empty
group id for older versions. I had thought that eventually we would be able
to remove those old versions, but it's true that this may not happen until
indefinitely far in the future. I think the main improvement here is
changing the default group.id to null instead of "". I could go either way
on whether bumping the protocol is useful. I do think it is helpful though
to signal clearly that it its use is deprecated and discouraged, especially
in light of the ACL problem. I guess we could just deprecate the use on the
client. What do you think?

Thanks,
Jason

On Wed, Aug 1, 2018 at 3:19 PM, Vahid S Hashemian  wrote:

> Thanks Jason for responding to Colin's concerns.
>
> If there are no other comment / feedback / objection I'll start a vote
> soon.
>
> Thanks.
> --Vahid
>
>
>
> From:   Jason Gustafson 
> To: dev 
> Date:   07/27/2018 10:38 AM
> Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hey Colin,
>
> The problem is both that the empty group id is the default value and that
> it is actually accepted by the broker for offset commits. Combine that
> with
> the fact that auto commit is enabled by default and you users get
> surprising behavior. If you look at a random Kafka cluster, you'll
> probably
> find a bunch of inadvertent offset commits for the empty group id. I was
> hoping we could distinguish between users who are using the empty group id
> as an accident of the default configuration and those who use it
> intentionally. By default, there will be no group id and the consumer will
> not commit offsets. If a user has actually intentionally used the empty
> group id, however, it will continue to work. I actually think there are
> probably very few people doing this (maybe even no one), but I thought we
> might err on the side of compatibility.
>
> The big incompatible change here is having brokers reject using
> assign(...)
> > with empty / null group.id.
>
>
> This is not correct. In the proposal, the broker will only reject the
> empty
> group id for the new version of OffsetCommit. Older clients, which cannot
> be changed, will continue to work because the old versions of the
> OffsetCommit API still accept the empty group id. The null group id is
> different from the empty group id: it is not allowed in any version of the
> API. It is basically a way to indicate that the consumer has no dependence
> on the coordinator at all, which we actually have a surprising number of
> use cases for. Furthermore, if a user has an actual need for the empty
> group id, it will still be allowed. We are just deprecating it.
>
> -Jason
>
> On Fri, Jul 27, 2018 at 9:56 AM, Colin McCabe  wrote:
>
> > Sorry if this is a silly question, but what's the rationale for
> switching
> > to using null for the default group id, rather than the empty string?
> > Continuing to use the empty string seems like less churn.  And after
> all,
> > we're not using the empty string group name for anything else.
> >
> > The big incompatible change here is having brokers reject using
> > assign(...) with empty / null group.id.  If I understand correctly, the
> > KIP proposes that this change be made on the brokers on the next
> > incompatible Kafka release.  But that has nothing to do with client
> > versions.  Why not just have a broker config which controls this?  Maybe
> "
> > allow.assign.empty.group.id", or something like that.  At first, the
> > default will be true, and then eventually we can flip it over to false.
> >
> > It seems like the main rationale for tying this behavior to the Kafka
> > client version is to force people to stop using the empty group id so
> that
> > they can upgrade their clients.  But it's also possible that people will
> > stop upgrading their Kafka clients instead.  That would be pretty
> negative
> > since  they'd miss out on any efficiency and feature improvements in the
> > new clients and eventually have to do more protocol downgrading, etc.
> >
> > best,
> > Colin
> >
> >
> > On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote:
> > > Hi Jason,
> > >
> > > That makes sense.
> > > I have updated the KIP based on the recent feedback.
> > >
> > > Thanks!
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From:   Jason Gustafson 
> > > To: dev 
> > > Date:   

Re: KAFKA-7137 - max time guarantees of compaction for GDPR et al

[Jason Gustafson]

Hey Brett,

I'd suggest going ahead with a KIP if you already have some ideas about it.
One of the complications you may run into is that the active segment is not
cleaned at present.

I went ahead and gave you wiki permission. I assumed your username is
brettrann, so let me know if it's different.

Thanks,
Jason

On Wed, Aug 1, 2018 at 5:38 PM, Brett Rann 
wrote:

> Ping on this. We're prepared to do the work i'm just not sure of the right
> way to start.  If nobody wants to discuss it now should we just write up a
> KIP?
>
> On Thu, Jul 19, 2018 at 10:57 AM Brett Rann  wrote:
>
> > re: https://issues.apache.org/jira/browse/KAFKA-7137
> >
> > My team is investigating what would be involved in code changes to give
> > some finer control over when compaction runs.  Detail in the ticket but
> > essentially the current way is that dirty.ratio is hijacked to be set to
> 0
> > to give guarantees that a tombstone record will result in previous record
> > being deleted within a set time.
> >
> > This results in a lot of unnecessary compaction, and we would like to add
> > the ability to provide a max delay for compaction to be used in
> combination
> > with a "healthy" dirty.ratio setting to meet this requirement instead.
> >
> > And to consider an API to trigger compaction (but that is looking
> > difficult because compaction isn't really triggered, it's evaluated by
> the
> > log cleaner thread).
> >
> > We'd like to get some discussion going around this.  Should we draft up a
> > KIP first then kick off a discussion?  (if yes, can I get KIP access for:
> > brettr...@gmail.com )
> >
> > Thanks!
> >
> > --
> >
> > Brett Rann
> >
> > Senior DevOps Engineer
> >
> >
> > Zendesk International Ltd
> >
> > 395 Collins Street, Melbourne VIC 3000 Australia
> >
> > Mobile: +61 (0) 418 826 017
> >
>
>
> --
>
> Brett Rann
>
> Senior DevOps Engineer
>
>
> Zendesk International Ltd
>
> 395 Collins Street, Melbourne VIC 3000 Australia
>
> Mobile: +61 (0) 418 826 017
>

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Jason Gustafson]

Hey Stanislav,

I should have noticed it earlier from your example, but I just realized
that interfaces don't mix well with exceptions. There is no way to catch
the interface type, which means you have to depend on instanceof checks,
which is not very conventional. At the moment, we raise
SerializationException if there is a problem parsing the error, and we
raise KafkaException if the record fails its CRC check. Since
SerializationException extends KafkaExeption, it seems like we can handle
both cases in a compatible way by handling both cases with a single type
that extends SerializationException. Maybe something like
RecordDeserializationException?

Thanks,
Jason

On Thu, Aug 2, 2018 at 5:45 AM, Ted Yu  wrote:

> +1
>  Original message From: Stanislav Kozlovski <
> stanis...@confluent.io> Date: 8/2/18  2:41 AM  (GMT-08:00) To:
> dev@kafka.apache.org Subject: [VOTE] KIP-334 Include partitions in
> exceptions raised during consumer record deserialization/validation
> Hey everybody,
>
> I'd like to start a vote thread for KIP-334 Include partitions in
> exceptions raised during consumer record deserialization/validation
>  >
>
> --
> Best,
> Stanislav
>

KIP-352: Distinguish URPs caused by reassignment

[Jason Gustafson]

Hey All,

Another day, another KIP. This one is hopefully straightforward:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment.
Have a look and let me know what you think!

Thanks,
Jason

Re: [DISCUSS] KIP-351: Add --critical-partitions option to describe topics command

[Jason Gustafson]

Hey Kevin,

Thanks for the KIP. I like Mickael's suggestion to
add --under-minisr-partitions since it fits with the metric we already
expose. It's also a good question whether there should be a separate
category for partitions which are right at min.isr. I'm reluctant to add
new categories, but I agree there might be a gap at the moment. Say you
have a replication factor of 3 and the min isr is set to 1. Our notion of
URP does not capture the difference between having an ISR down to a size of
1 and one down to a size of 2. The reason this might be significant is that
a shrink of the ISR down to 2 may just be caused by a rolling restart or a
transient network blip. A shrink to 1, on the other hand, might be
indicative of a more severe problem and could be cause for a call from
pagerduty.

-Jason

On Thu, Aug 2, 2018 at 9:28 AM, Kevin Lu  wrote:

> Hi Mickael,
>
> Thanks for the suggestion!
>
> Correct me if I am mistaken, but if a producer attempts to send to a
> partition that is under min ISR (and ack=all or -1) then the send will fail
> with a NotEnoughReplicas or NotEnoughReplicasAfterAppend exception? At this
> point, client-side has already suffered failure but the server-side is
> still fine for now?
>
> If the above is true, then this would be a FATAL case for producers.
>
> Would it be valuable to include the CRITICAL case where a topic partition
> has exactly min ISR so that Kafka operators can take action so it does not
> become FATAL? This could be in the same option or a new one.
>
> Thanks!
>
> Regards,
> Kevin
>
> On Thu, Aug 2, 2018 at 2:27 AM Mickael Maison 
> wrote:
>
> > What about also adding a --under-minisr-partitions option?
> >
> > That would match the
> > "kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount"
> > broker metric and it's usually pretty relevant when investigating
> > issues
> >
> > On Thu, Aug 2, 2018 at 8:54 AM, Kevin Lu  wrote:
> > > Hi friends!
> > >
> > > This thread is to discuss KIP-351
> > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-351%3A
> +Add+--critical-partitions+option+to+describe+topics+command
> > >
> > > !
> > >
> > > I am proposing to add a --critical-partitions option to the describe
> > topics
> > > command that will only list out topic partitions that have 1 ISR left
> > (RF >
> > > 1) as they would be in a critical state and need immediate
> > repartitioning.
> > >
> > > I wonder if the name "critical" is appropriate?
> > >
> > > Thoughts?
> > >
> > > Thanks!
> > >
> > > Regards,
> > > Kevin
> >
>

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

[Jason Gustafson]

Hey Vahid,

I think we're in agreement. Perhaps since none of us feel too strongly, we
should go for the more minimal change? I'll vote for the KIP either way as
long as we can change the default group.id.

Thanks,
Jason

On Thu, Aug 2, 2018 at 11:55 AM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Jason, Colin,
>
> Thanks for your feedback.
>
> If we are going to keep supporting the empty group id like before then it
> makes sense to do what you suggested.
>
> Though, IMHO if using the empty group id for committing offsets is a bad
> practice we should stop supporting it at some point; and it would be
> better to do it sooner before potentially more users adopt that.
> I understand that changing the default would reduce the chance of someone
> using it, but it doesn't stop them. Note that we did not receive any
> objections to dropping the support even right-away (a good sign).
> We could keep supporting the old API version in case someone must use this
> group id like before. But at least we won't expose new users to this bad
> practice.
>
> I, too, do now have a strong opinion about this. So if the current KIP
> should not move forward as is, I'm okay with just changing the default and
> deprecating the use on the client, as you suggested.
>
> Thanks.
> --Vahid
>
>
>
> From:   Colin McCabe 
> To: dev@kafka.apache.org
> Date:   08/02/2018 11:23 AM
> Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Thanks, Jason.  I don't have a very strong opinion on this.  But like you
> said, if we skip bumping the RPC versions, this would be a smaller change,
> which might be good.
>
> best,
> Colin
>
>
> On Wed, Aug 1, 2018, at 17:43, Jason Gustafson wrote:
> > Hey Vahid,
> >
> > I talked with Colin offline. I think specifically he felt the version
> bump
> > on the broker was overkill since the broker still has to support the
> empty
> > group id for older versions. I had thought that eventually we would be
> able
> > to remove those old versions, but it's true that this may not happen
> until
> > indefinitely far in the future. I think the main improvement here is
> > changing the default group.id to null instead of "". I could go either
> way
> > on whether bumping the protocol is useful. I do think it is helpful
> though
> > to signal clearly that it its use is deprecated and discouraged,
> especially
> > in light of the ACL problem. I guess we could just deprecate the use on
> the
> > client. What do you think?
> >
> > Thanks,
> > Jason
> >
> > On Wed, Aug 1, 2018 at 3:19 PM, Vahid S Hashemian
>  > > wrote:
> >
> > > Thanks Jason for responding to Colin's concerns.
> > >
> > > If there are no other comment / feedback / objection I'll start a vote
> > > soon.
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > > From:   Jason Gustafson 
> > > To: dev 
> > > Date:   07/27/2018 10:38 AM
> > > Subject:Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > Hey Colin,
> > >
> > > The problem is both that the empty group id is the default value and
> that
> > > it is actually accepted by the broker for offset commits. Combine that
> > > with
> > > the fact that auto commit is enabled by default and you users get
> > > surprising behavior. If you look at a random Kafka cluster, you'll
> > > probably
> > > find a bunch of inadvertent offset commits for the empty group id. I
> was
> > > hoping we could distinguish between users who are using the empty
> group id
> > > as an accident of the default configuration and those who use it
> > > intentionally. By default, there will be no group id and the consumer
> will
> > > not commit offsets. If a user has actually intentionally used the
> empty
> > > group id, however, it will continue to work. I actually think there
> are
> > > probably very few people doing this (maybe even no one), but I thought
> we
> > > might err on the side of compatibility.
> > >
> > > The big incompatible change here is having brokers reject using
> > > assign(...)
> > > > with empty / null group.id.
> > >
> > >
> > > This is not correct. In the proposal, the broker will only reject the
> > > empty
> > > group id for the n

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

[Jason Gustafson]

Hey All,

I think I've addressed all pending review. If there is no additional
feedback, I'll plan to start a vote thread next week.

Thanks,
Jason

On Tue, Jul 31, 2018 at 9:46 AM, Dong Lin  wrote:

> Hey Jason,
>
> Thanks for your reply. I will comment below.
>
> Regarding 1, we probably can not simply rename both to `LeaderEpoch`
> because we already have a LeaderEpoch field in OffsetsForLeaderEpoch.
>
> Regarding 5, I am not strong on this. I agree with the two benefits of
> having two error codes: 1) not having to refresh metadata when consumer
> sees UNKNOWN_LEADER_EPOCH and 2) provide more information in the log for
> debugging. Whether or not these two benefits are useful enough for one more
> error code may be subjective. I will let you and others determine this.
>
> Regarding 6, yeah overloading seek() looks good to me.
>
>
> Thanks,
> Dong
>
>
> On Mon, Jul 30, 2018 at 9:33 AM, Jason Gustafson 
> wrote:
>
> > Hey Dong,
> >
> > Thanks for the detailed review. Responses below:
> >
> > 1/2: Thanks for noticing the inconsistency. Would it be reasonable to
> > simply call it LeaderEpoch for both APIs?
> >
> > 3: I agree it should be a map. I will update.
> >
> > 4: Fair point. I think we should always be able to identify an offset.
> > Let's remove the Optional for now and reconsider if we find an unhandled
> > case during implementation.
> >
> > 5: Yeah, I was thinking about this. The two error codes could be handled
> > similarly, so we might merge them. Mainly I was thinking that it will be
> > useful for consumers/replicas to know whether they are ahead or behind
> the
> > leader. For example, if a consumer sees UNKNOWN_LEADER_EPOCH, it need not
> > refresh metadata. Or if a replica sees a FENCED_LEADER_EPOCH error, it
> > could just stop fetching and await the LeaderAndIsr request that it is
> > missing. It probably also makes debugging a little bit easier. I guess
> I'm
> > a bit inclined to keep both error codes, but I'm open to reconsideration
> if
> > you feel strongly. Another point to consider is whether we should
> continue
> > using NOT_LEADER_FOR_PARTITION if a follower receives an unexpected
> fetch.
> > The leader epoch would be different in this case so we could use one of
> the
> > invalid epoch error codes instead since they contain more information.
> >
> > 6: I agree the name is not ideal in that scenario. What if we overloaded
> > `seek`?
> >
> > 7: Sure, I will mention this.
> >
> >
> > Thanks,
> > Jason
> >
> > On Fri, Jul 27, 2018 at 6:17 PM, Dong Lin  wrote:
> >
> > > Hey Jason,
> > >
> > > Thanks for the update! I agree with the current proposal overall. I
> have
> > > some minor comments related to naming etc.
> > >
> > > 1) I am not strong and will just leave it here for discussion. Would it
> > be
> > > better to rename "CurrentLeaderEpoch" to "ExpectedLeaderEpoch" for the
> > new
> > > field in the OffsetsForLeaderEpochRequest? The reason is that
> > > "CurrentLeaderEpoch" may not necessarily be true current leader epoch
> if
> > > the consumer has stale metadata. "ExpectedLeaderEpoch" shows that this
> > > epoch is what consumer expects on the broker which may or may not be
> the
> > > true value.
> > >
> > > 2) Currently we add the field "LeaderEpoch" to FetchRequest and the
> field
> > > "CurrentLeaderEpoch" to OffsetsForLeaderEpochRequest. Given that both
> > > fields are compared with the leaderEpoch in the broker, would it be
> > better
> > > to give them the same name?
> > >
> > > 3) Currently LogTruncationException.truncationOffset() returns
> > > Optional to user. Should it return
> > > Optional> to handle the scenario
> > > where leaderEpoch of multiple partitions are different from the
> > leaderEpoch
> > > in the broker?
> > >
> > > 4) Currently LogTruncationException.truncationOffset() returns an
> > Optional
> > > value. Could you explain a bit more when it will return
> > Optional.empty()? I
> > > am trying to understand whether it is simpler and reasonable to
> > > replace Optional.empty()
> > > with OffsetMetadata(offset=last_fetched_offset, leaderEpoch=-1).
> > >
> > > 5) Do we also need to add a new retriable exception for error code
> > > FENCED_LEADER_EPOCH? And do we need to define both FENCED_LEADER_EPOCH
> > > and U

Re: [VOTE] KIP-341: Update Sticky Assignor's User Data Protocol

[Jason Gustafson]

+1 Thanks Vahid.

On Thu, Aug 2, 2018 at 1:27 PM, Vahid S Hashemian  wrote:

> Hi everyone,
>
> I believe the feedback on this KIP has been addressed so far. So I'd like
> to start a vote.
> The KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 341%3A+Update+Sticky+Assignor%27s+User+Data+Protocol
> Discussion thread:
> https://www.mail-archive.com/dev@kafka.apache.org/msg89733.html
>
> Thanks!
> --Vahid
>
>