Vahid, Under rejected alternatives, we had decided that we did NOT want to do per-partition expiration, and instead we wait until the entire group is empty and then (after the right time has passed) expire the entire group at once.
I thought of one scenario that might benefit from per-partition expiration. Let's say I have topics A B C... Z. So, I have 26 topics, all of them single partition, so 26 partitions. Let's say I have mirrormaker mirroring those 26 topics. The group will then have 26 committed offsets. Let's say I then change the whitelist on mirrormaker so that it only mirrors topic Z, but I keep the same consumer group name. (I imagine that is a common thing to do?) With the proposed design for this KIP, the committed offsets for topics A through Y will stay around as long as this mirroring group name exists. In the current implementation that already exists (prior to this KIP), I belive that committed offsets for topics A through Y will expire. How much do we care about this case? -James > On Jan 23, 2018, at 11:44 PM, Jeff Widman <j...@jeffwidman.com> wrote: > > Bumping this as I'd like to see it land... > > It's one of the "features" that tends to catch Kafka n00bs unawares and > typically results in message skippage/loss, vs the proposed solution is > much more intuitive behavior. > > Plus it's more wire efficient because consumers no longer need to commit > offsets for partitions that have no new messages just to keep those offsets > alive. > > On Fri, Jan 12, 2018 at 10:21 AM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > >> There has been no further discussion on this KIP for about two months. >> So I thought I'd provide the scoop hoping it would spark additional >> feedback and move the KIP forward. >> >> The KIP proposes a method to preserve group offsets as long as the group >> is not in Empty state (even when offsets are committed very rarely), and >> start the offset expiration of the group as soon as the group becomes >> Empty. >> It suggests dropping the `retention_time` field from the `OffsetCommit` >> request and, instead, enforcing it via the broker config >> `offsets.retention.minutes` for all groups. In other words, all groups >> will have the same retention time. >> The KIP presumes that this global retention config would suffice common >> use cases and does not lead to, e.g., unmanageable offset cache size (for >> groups that don't need to stay around that long). It suggests opening >> another KIP if this global retention setting proves to be problematic in >> the future. It was suggested earlier in the discussion thread that the KIP >> should propose a per-group retention config to circumvent this risk. >> >> I look forward to hearing your thoughts. Thanks! >> >> --Vahid >> >> >> >> >> From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> >> To: dev <dev@kafka.apache.org> >> Date: 10/18/2017 04:45 PM >> Subject: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer >> Group Offsets >> >> >> >> Hi all, >> >> I created a KIP to address the group offset expiration issue reported in >> KAFKA-4682: >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki. >> apache.org_confluence_display_KAFKA_KIP-2D211-253A-2BRevise- >> 2BExpiration-2BSemantics-2Bof-2BConsumer-2BGroup-2BOffsets& >> d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj- >> kjJc7uSVcviKUc&m=JkzH_2jfSMhCUPMk3rUasrjDAId6xbAEmX7_shSYdU4&s= >> UBu7D2Obulg0fterYxL5m8xrDWkF_O2kGlygTCWsfFc&e= >> >> >> Your feedback is welcome! >> >> Thanks. >> --Vahid >> >> >> >> >> >> > > > -- > > *Jeff Widman* > jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265) > <><
