Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

. Sax" <matth...@confluent.io> To: dev@kafka.apache.org Date: 04/04/2018 06:28 PM Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets I was just reading the whole KIP for the first time. Nice work! One minor comment. In the table of the sta

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

I was just reading the whole KIP for the first time. Nice work! One minor comment. In the table of the standalone consumer, the first line, first column says: > = Empty > (protocolType = Some("consumer")) I think this should be > = Empty > (protocolType != None) Note, that for example

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Hi all, Thanks for the feedback on this KIP so far. If there is no additional feedback, I'll start a vote on Wed. Thanks. --Vahid

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

coordinator change case. Thanks! --Vahid From: Jason Gustafson <ja...@confluent.io> To: dev@kafka.apache.org Date: 03/17/2018 11:50 AM Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets Hey Vahid, Sorry for the delay. I've re

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

se I've > missed something :) > > Thanks. > --Vahid > > > > From: Jason Gustafson <ja...@confluent.io> > To: dev@kafka.apache.org > Date: 03/05/2018 03:42 PM > Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of > Consumer Group Offsets &

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

time in case I've missed something :) Thanks. --Vahid From: Jason Gustafson <ja...@confluent.io> To: dev@kafka.apache.org Date: 03/05/2018 03:42 PM Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets Hey Vahid, On point #1 below:

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

ed to). This can be done as part of the offset cleanup > scheduled task that by default does not run very frequently. Were you > thinking of a different method for capturing the subscription change? > > Thanks. > --Vahid > > > > > From: Jason Gustafson <ja...@confluent.io

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

ption change? Thanks. --Vahid From: Jason Gustafson <ja...@confluent.io> To: dev@kafka.apache.org Date: 02/18/2018 01:16 PM Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets Hey Vahid, Sorry for the late response. The KIP looks goo

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

the update is ready for review. Apologies for the confusion. --Vahid From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> To: dev@kafka.apache.org Date: 03/01/2018 11:43 AM Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets H

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

'. Regarding the other JIRA you referred to, sure, I'll add that in the KIP. Thanks. --Vahid From: Jason Gustafson <ja...@confluent.io> To: dev@kafka.apache.org Date: 02/28/2018 12:10 PM Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

for deletion of offsets of specific topics > in > > the group. This is what the old ZooKeeper based group management > supported > > anyway, and we would just be leveling the group deletion features of the > > Kafka-based group management with the ZooKeeper-based on

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

. Thanks. --Vahid From: Jason Gustafson <ja...@confluent.io> To: dev@kafka.apache.org Date: 02/18/2018 01:16 PM Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets Hey Vahid, Sorry for the late response. The KIP looks good. A few comments:

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

gt; > From: James Cheng <wushuja...@gmail.com> > To: dev@kafka.apache.org > Date: 02/01/2018 12:37 AM > Subject:Re: [DISCUSS] KIP-211: Revise Expiration Semantics of > Consumer Group Offsets > > > > Vahid, > > Under rejected alternative

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

> To: dev@kafka.apache.org Date: 02/01/2018 12:37 AM Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets Vahid, Under rejected alternatives, we had decided that we did NOT want to do per-partition expiration, and instead we wait until the entire

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Vahid, Under rejected alternatives, we had decided that we did NOT want to do per-partition expiration, and instead we wait until the entire group is empty and then (after the right time has passed) expire the entire group at once. I thought of one scenario that might benefit from

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Bumping this as I'd like to see it land... It's one of the "features" that tends to catch Kafka n00bs unawares and typically results in message skippage/loss, vs the proposed solution is much more intuitive behavior. Plus it's more wire efficient because consumers no longer need to commit

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

There has been no further discussion on this KIP for about two months. So I thought I'd provide the scoop hoping it would spark additional feedback and move the KIP forward. The KIP proposes a method to preserve group offsets as long as the group is not in Empty state (even when offsets are

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

: long --Vahid From: James Cheng <wushuja...@gmail.com> To: dev@kafka.apache.org Date: 11/16/2017 12:01 AM Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets How fast does the in-memory cache grow? As a random datapoint... 10 months ago

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

> It would be great to hear feedback from others (and committers) on this. > > --Vahid > > > > > From: Jeff Widman <j...@jeffwidman.com> > To: dev@kafka.apache.org > Date: 11/15/2017 01:04 PM > Subject:Re: [DISCUSS] KIP-211: Revise Expiration

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

com> To: dev@kafka.apache.org Date: 11/15/2017 01:04 PM Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets I thought about this scenario as well. However, my conclusion was that because __consumer_offsets is a compacted topic, this extra clutte

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

I thought about this scenario as well. However, my conclusion was that because __consumer_offsets is a compacted topic, this extra clutter from short-lived consumer groups is negligible. The disk size is the product of the number of consumer groups and the number of partitions in the group's

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

I'm forwarding this feedback from John to the mailing list, and responding at the same time: John, thanks for the feedback. I agree that the scenario you described could lead to unnecessary long offset retention for other consumer groups. If we want to address that in this KIP we could either

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Thanks Jeff. I'll wait until EOD tomorrow (Wednesday), and then I'll start a vote. --Vahid From: Jeff Widman <j...@jeffwidman.com> To: dev@kafka.apache.org Date: 11/14/2017 11:35 AM Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Any other input on this? Otherwise Vahid what do you think about moving this to a vote? On Tue, Nov 7, 2017 at 2:34 PM, Jeff Widman wrote: > Any other feedback from folks on KIP-211? > > A prime benefit of this KIP is that it removes the need for the consumer > to commit

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Any other feedback from folks on KIP-211? A prime benefit of this KIP is that it removes the need for the consumer to commit offsets for partitions where the offset hasn't changed. Right now, if the consumer doesn't commit those offsets, they will be deleted, so the consumer keeps blindly

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

I support this as the proposed change seems both more intuitive and safer. Right now we've essentially hacked this at my day job by bumping the offset retention period really high, but this is a much cleaner solution. I don't have any use-cases that require custom retention periods on a

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Bump! From: Vahid S Hashemian/Silicon Valley/IBM To: dev Date: 10/18/2017 04:45 PM Subject:[DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets Hi all, I created a KIP to address the group offset expiration issue reported in

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Thanks Ted. I filled out that section. --Vahid From: Ted Yu <yuzhih...@gmail.com> To: dev@kafka.apache.org Date: 10/18/2017 04:59 PM Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets Please fill out 'Rejected Alternatives' section.

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Thanks Thomas for the suggestion. I updated the KIP to explicitly describe that situation. --Vahid From: Thomas Becker <thomas.bec...@tivo.com> To: "dev@kafka.apache.org" <dev@kafka.apache.org> Date: 10/19/2017 08:23 AM Subject: Re: [DISCUSS] KIP-211: Revis

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

I think it would be helpful to clarify what happens if consumers rejoin an empty group. I would presume that the expiration timer is stopped and reset back to offsets.retention.minutes when it is empty again but the KIP doesn't say. On Wed, 2017-10-18 at 16:45 -0700, Vahid S Hashemian wrote:

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Please fill out 'Rejected Alternatives' section. Thanks On Wed, Oct 18, 2017 at 4:45 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi all, > > I created a KIP to address the group offset expiration issue reported in > KAFKA-4682: >