Hi, Andrew,
Thanks for the updates. Looks reasonable to me.
I don't see ShareConsumer.acknowledge(String, int, long, AcknowledgeType)
in the KIP or the code. There are only the following two.
void acknowledge(ConsumerRecord<K, V> record);
void acknowledge(ConsumerRecord<K, V> record, AcknowledgeType type);
Thanks,
Jun
On Fri, Jul 11, 2025 at 2:30 AM Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:
> Hi,
> As we've written the code for KIP-932, we've made some small changes to
> the KIP
> to correct mistakes, align with other accepted KIPs and fix some omissions
> picked
> up during PR reviews. We also have experience of people writing
> applications
> against the ShareConsumer interface and have corrected a couple of
> problems.
>
> * Using share.version=1 as the switch to enable share groups. This aligns
> with group.version=1 for KIP-848 and streams.version=1 for KIP-1071.
> * Aligned with KIP-1043 by using a common GroupState and removing
> Admin.listShareGroups (because Admin.listGroups handles all group types).
> * Corrected the topic authorizations on admin requests.
> * Added SHARE_SESSION_LIMIT_REACHED error code to give proper
> consumer error handling when the limit is reached.
> * Adjusted ShareFetch RPC request to remove PartitionMaxBytes and
> add BatchSize to improve efficiency of fetching and state management.
> * Changed the rules for explicit acknowledgement because, as originally
> written, it was simple to write an application that went into an infinite
> loop.
> Now, when using explicit acknowledgement, it is mandatory to acknowledge
> every single record, which allows an exception to be thrown if the
> application
> does not do this properly.
> * Added a method overload with signature
> ShareConsumer.acknowledge(String, int, long, AcknowledgeType) to go with
> ShareConsumer.acknowledge(ConsumerRecord, AcknowledgeType).
> This is because the KIP said it would be possible to choose the acknowledge
> type in the event of RecordDeserializationException. However, because the
> application does not have the ConsumerRecord instance when the
> deserialization
> fails, but it does have topic name, partition and offset, adding an
> overloaded
> method enables the application to choose the acknowledge type.
> * The SimpleAssignor no longer uses hashing as part of making assignments.
> In practice, this hindered the assignment computation. SimpleAssignor
> achieves
> the balance and stickiness described previously, just without hashing.
> * Added
> org.apache.kafka.coordinator.group.api.assignor.GroupSpec.isPartitionAssignable
> so that partitions which are in the process of being initialized in the
> share coordinator
> can be omitted from assignment computation.
> * Removed the ShareGroupPartitionMetadata record type (KIP-1101).
> * Added timestamps to ShareSnapshot records so it is possible to manage
> pruning of records correctly.
> * Adjusted some of the config defaults and limits such as maximum for lock
> duration which didn't make sense.
>
> KIP-932 mentions some future work and I have started to create new KIPs
> for this.
>
> Thanks,
> Andrew
> ________________________________________
> From: Jun Rao <j...@confluent.io.INVALID>
> Sent: 15 August 2024 19:07
> To: dev@kafka.apache.org <dev@kafka.apache.org>
> Subject: Re: [VOTE] KIP-932: Queues for Kafka
>
> Hi, Andrew,
>
> Thanks for the clarification. Sounds good.
>
> Jun
>
> On Thu, Aug 15, 2024 at 2:53 AM Andrew Schofield <
> andrew_schofi...@live.com>
> wrote:
>
> > Hi Jun,
> > Thanks for the questions.
> >
> > 1. The PartitionMetadata is only stored if there is rack ID information.
> > So, there are situations in which it’s redundant, but almost always the
> > PartitionMetadata will be empty.
> >
> > 2. They are for subtly different purposes. ShareGroupPartitionMetadata
> > is essentially a persistent cache of the metadata for topic-partitions
> > which
> > can be assigned to group members. ShareGroupStatePartitionMetadata
> > is keeping track of which partitions have share-group state in the share
> > coordinator. I prefer having them separate, in particular because it
> means
> > the ShareGroupPartitionMetadata matches nicely with consumer groups,
> > and also because share groups are able to work without a share
> coordinator
> > (the persister is pluggable).
> >
> > I propose that as we develop the code in the group coordinator which uses
> > the various ShareGroupState RPCs to talk to the share coordinator,
> > we will confirm whether the record schemas are optimal
> > and make a further refinement if required. Let me know if that’s OK.
> >
> > Thanks,
> > Andrew
> >
> >
> > > On 14 Aug 2024, at 00:00, Jun Rao <j...@confluent.io.INVALID> wrote:
> > >
> > > Hi, Andrew,
> > >
> > > Thanks for the update. A couple of followup questions.
> > >
> > > 1. ShareGroupPartitionMetadataValue: Is NumPartitions redundant given
> > > PartitionMetadata?
> > >
> > > 2. I assume that ShareGroupPartitionMetadataValue contains initialized
> > > partitions. Does that
> > > make ShareGroupStatePartitionMetadataValue.InitializedTopics redundant?
> > >
> > > Jun
> > >
> > > On Tue, Aug 13, 2024 at 11:56 AM Andrew Schofield <
> > andrew_schofi...@live.com>
> > > wrote:
> > >
> > >> Hi,
> > >> During the implementation of the group management code for KIP-932,
> > >> we learnt that the decision not to persist assignments for share
> groups
> > was
> > >> making it difficult to fit with the coordinator framework, in
> particular
> > >> the
> > >> use of timeline structures. As a result, I decided to change the KIP
> to
> > >> follow the consumer group record scheme much more closely.
> > >>
> > >> Now, share groups do persist assignments and member epochs which
> > >> resolves the problems we were experiencing managing soft state in the
> > >> coordinator and making group membership reliable across coordinator
> > >> restarts.
> > >>
> > >> I have updated the KIP accordingly and the revised code has been
> merged.
> > >> If anyone has any comments, please let me know.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >>
> > >>> On 21 May 2024, at 15:35, Andrew Schofield <
> andrew_schofi...@live.com>
> > >> wrote:
> > >>>
> > >>> Hi Jun,
> > >>> All the client metrics are standard. None are required.
> > >>>
> > >>> I’ve updated the KIP accordingly.
> > >>>
> > >>> Thanks,
> > >>> Andrew
> > >>>
> > >>>> On 15 May 2024, at 21:36, Jun Rao <j...@confluent.io.INVALID> wrote:
> > >>>>
> > >>>> Hi, Andrew,
> > >>>>
> > >>>> Thanks for the update. Should we mark whether those metrics are
> > >>>> standard/required for KIP-714?
> > >>>>
> > >>>> Jun
> > >>>>
> > >>>> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <
> > >> andrew_schofi...@live.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi,
> > >>>>> I have made a small update to the KIP as a result of testing the
> new
> > >>>>> share consumer with client telemetry (KIP-714).
> > >>>>>
> > >>>>> I’ve added telemetry metric names to the table of client metrics
> and
> > >>>>> also updated the metric group names so that the resulting client
> > >> metrics
> > >>>>> sent to the broker have consistent names.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Andrew
> > >>>>>
> > >>>>>> On 8 May 2024, at 12:51, Manikumar <manikumar.re...@gmail.com>
> > wrote:
> > >>>>>>
> > >>>>>> Hi Andrew,
> > >>>>>>
> > >>>>>> Thanks for the KIP. Great write-up!
> > >>>>>>
> > >>>>>> +1 (binding)
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> On Wed, May 8, 2024 at 12:17 PM Satish Duggana <
> > >> satish.dugg...@gmail.com>
> > >>>>> wrote:
> > >>>>>>>
> > >>>>>>> Hi Andrew,
> > >>>>>>> Thanks for the nice KIP, it will allow other messaging use cases
> to
> > >> be
> > >>>>>>> onboarded to Kafka.
> > >>>>>>>
> > >>>>>>> +1 from me.
> > >>>>>>>
> > >>>>>>> Satish.
> > >>>>>>>
> > >>>>>>> On Tue, 7 May 2024 at 03:41, Jun Rao <j...@confluent.io.invalid>
> > >> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi, Andrew,
> > >>>>>>>>
> > >>>>>>>> Thanks for the KIP. +1
> > >>>>>>>>
> > >>>>>>>> Jun
> > >>>>>>>>
> > >>>>>>>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <
> > >> edoardli...@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Thanks Andrew,
> > >>>>>>>>>
> > >>>>>>>>> +1 (binding)
> > >>>>>>>>>
> > >>>>>>>>> Edo
> > >>>>>>>>>
> > >>>>>>>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> > >>>>>>>>> <kevers...@cloudflare.com.invalid> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> Hi Andrew
> > >>>>>>>>>>
> > >>>>>>>>>> + 1 (Non-Binding)
> > >>>>>>>>>>
> > >>>>>>>>>> This will be great addition to Kafka
> > >>>>>>>>>>
> > >>>>>>>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
> > >>>>> apoorvmitta...@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi Andrew,
> > >>>>>>>>>>> Thanks for writing the KIP. This is indeed going to be a
> > valuable
> > >>>>>>>>> addition
> > >>>>>>>>>>> to the Kafka, excited to see the KIP.
> > >>>>>>>>>>>
> > >>>>>>>>>>> + 1 (Non-Binding)
> > >>>>>>>>>>>
> > >>>>>>>>>>> Regards,
> > >>>>>>>>>>> Apoorv Mittal
> > >>>>>>>>>>> +44 7721681581
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> > >>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi,
> > >>>>>>>>>>>> I’ve been working to complete KIP-932 over the past few
> months
> > >> and
> > >>>>>>>>>>>> discussions have quietened down.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I’d like to open the voting for KIP-932:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>> Andrew
> >
> >
> >
>
