Hi Chia-Ping / David, Thanks for the review and suggestions.
>> I don't fully follow your comment so please excuse me if I say something >> wrong. I think Chia-Ping wants to say: originally, the default protocol in classic group is “consumer”. After we have a new consumer group, the protocol of a classic member in consumer group is “classic”. The MemberDescription is used by both ClassicGroupDescription and ConsumerGroupDescription. If we add the protocol field to MemberDescription, the protocol of a classic member in classic group is “consumer”, but it’s “classic” in consumer group. To avoid misleading word, we can add a new collection of member protocols to ConsumerGroupDescription only. The original goal is to show member protocol during consumer migration, so we don’t need to add this information to ClassicGroupDescription. Please correct me if I misunderstood anything. Thanks, PoAn > On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <chia7...@gmail.com> wrote: > > hi DJ > >> How about we use another Jira to add “SHARE” to GroupProtocol and change > return type of ModernGroup#protocolType? > > This is not my comment ... sorry for the incorrect quote :( > >> A member in a classic group should report "classic". > > I completely agree. However, in this case, ClassicGroupDescription#protocol > returns 'consumer', while ClassicGroupDescription#members[x].protocol > returns 'classic'. This inconsistency feels a bit strange to me. > > Best, > Chia-Ping > > > David Jacot <dja...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道： > >> Hi Chia, >> >>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but I >> would like to align the format with kafka-groups.sh. >> How about we use another Jira to add “SHARE” to GroupProtocol and change >> return type of ModernGroup#protocolType? >> >> My understanding is that `GroupProtocol` is used to drive the config of the >> `Consumer`. Hence adding `Share` seems wrong to me. Or are you referring to >> another GroupProtocol enum? Perhaps GroupType enum? >> >>> In fact, I'm wondering if we should even add 'protocol' to >> MemberDescription, >> since the field has different values between classic and consumer groups—it >> shows 'consumer' in classic groups, whereas it shows 'classic' in consumer >> groups. Yes, it seems like a typo, but it's true. This inconsistency feels >> confusing to me. For example, users call Admin#describeConsumerGroups and >> see that a member has the 'consumer' protocol when it's in a classic group. >> Then, if the group is upgraded to a consumer group, calling >> Admin#describeConsumerGroups again shows the protocol as 'classic' for the >> same member. >> >> I don't fully follow your comment so please excuse me if I say something >> wrong. >> >> I think that it should work as follows: >> * A member in a classic group should report "classic". >> * A member using the consumer protocol in a consumer group should report >> "consumer". >> * A member using the classic protocol in a consumer group should report >> "classic". >> * A member in a share group should report "share". >> >>> If all we want to do is distinguish classic members in the >> ConsumerGroupDescription, then we shouldn't modify the common >> MemberDescription structure. Instead, we could add a collection to >> ConsumerGroupDescription to trace members using the classic protocol, which >> would allow us to enrich the output of command-line tools. >> >> Personally, I believe that using different collections per protocol types >> is not necessary. A field in the member is more than enough. >> >> Best, >> DJ >> >> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <chia7...@gmail.com> >> wrote: >> >>> hi PoAn >>> >>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but I >>> would like to align the format with kafka-groups.sh. >>> How about we use another Jira to add “SHARE” to GroupProtocol and change >>> return type of ModernGroup#protocolType? >>> >>> MemberDescription is also used by ClassicGroupDescription, and in classic >>> groups, the protocol can be an arbitrary string, so we can't use an enum. >>> >>> In fact, I'm wondering if we should even add 'protocol' to >>> MemberDescription, >>> since the field has different values between classic and consumer >> groups—it >>> shows 'consumer' in classic groups, whereas it shows 'classic' in >> consumer >>> groups. Yes, it seems like a typo, but it's true. This inconsistency >> feels >>> confusing to me. For example, users call Admin#describeConsumerGroups and >>> see that a member has the 'consumer' protocol when it's in a classic >> group. >>> Then, if the group is upgraded to a consumer group, calling >>> Admin#describeConsumerGroups again shows the protocol as 'classic' for >> the >>> same member. >>> >>> If all we want to do is distinguish classic members in the >>> ConsumerGroupDescription, then we shouldn't modify the common >>> MemberDescription structure. Instead, we could add a collection to >>> ConsumerGroupDescription to trace members using the classic protocol, >> which >>> would allow us to enrich the output of command-line tools. >>> Best, >>> Chia-Ping >>> >>> PoAn Yang <yangp...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道： >>> >>>> Hi Lianet / Jeff, >>>> >>>> Thanks for the review. >>>> >>>> LM5: The kafka-share-groups.sh also uses “—describe” to show offsets >>>> information. >>>> Change to use “—describe —state —verbose” to show group level >> information >>>> in kafka-share-groups.sh. >>>> >>>> LM6: Update the example. >>>> >>>> LM7: Add a description to mention “—verbose” is a new option in >>>> ShareGroupCommandOptions. >>>> >>>> JK2: Yes, add the statement about format change for ASSIGNMENT value. >>>> >>>> Thanks, >>>> PoAn >>>> >>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <jeffkb...@apache.org> wrote: >>>>> >>>>> Hi PoAn, >>>>> >>>>> JK2: Can we state in the KIP that we will reformat the existing >>>>> classic group's `--members --verbose` as well, specifically the >>>>> "ASSIGNMENT" column to include the topic name? >>>>> >>>>> Thanks! >>>>> >>>>> On 2024/11/21 14:33:37 "Lianet M." wrote: >>>>>> Thanks for the updates PoAn! >>>>>> >>>>>> LM6. Just a nit, under the --describe --state --verbose the header >> is >>>> fine >>>>>> but the example is still missing the --state argument. >>>>>> >>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, should we >>>>>> clearly mention that the KIP adds a new --verbose option? I think >> it's >>>>>> confusing because it's presented as if the option already exists and >>>> we're >>>>>> only changing the output (which is the case for >> kafka-consumer-groups >>>> but >>>>>> not for kafka-share-groups) >>>>>> >>>>>> That's all on my side. Thanks! >>>>>> >>>>>> Lianet >>>>>> >>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <yangp...@gmail.com> >>> wrote: >>>>>> >>>>>>> Hi Lianet / Jeff, >>>>>>> >>>>>>> Thanks for review and suggestions. >>>>>>> >>>>>>> LM4: At group level, the protocol field is from >>>> ModernGroup#protocolType. >>>>>>> It’s a string. >>>>>>> The value is defined in different places like >>> ShareGroup#PROTOCOL_TYPE >>>> and >>>>>>> ConsumerProtocol#PROTOCOL_TYPE. >>>>>>> In KIP-1043 < >>>>>>> >>>> >>> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh >>>>> , >>>>>>> the kafka-groups.sh also shows protocol like “classic” / >> “consumer” / >>>>>>> “share”. >>>>>>> I agree it’s better to use a type like GroupProtocol enum, but I >>> would >>>>>>> like to align the format with kafka-groups.sh. >>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and >>>> change >>>>>>> return type of ModernGroup#protocolType? >>>>>>> >>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in the >>>> title. >>>>>>> You’re correct. I would like to use `—describe —state —verbose` to >>> show >>>>>>> group level information. >>>>>>> For `—describe —verbose`, it will show output as same as `—describe >>>>>>> —offsets —verbose`, >>>>>>> because `—describe` also shows output as same as `—describe >>> —offsets`. >>>>>>> >>>>>>> JK2: Yes, I only use consumer group in the sample output, but the >>>> classic >>>>>>> group will use the same format as well. >>>>>>> >>>>>>> Thanks, >>>>>>> PoAn >>>>>>> >>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <jeffkb...@apache.org> >> wrote: >>>>>>>> >>>>>>>> Hi PoAn, >>>>>>>> >>>>>>>> Thanks for the KIP. I have some questions: >>>>>>>> >>>>>>>> JK1: (Realized this is similar to LM5) On `--describe --verbose` >>>>>>> proposed changes - doesn't `--describe $group` default to printing >>> the >>>>>>> offsets? Perhaps you're referring to the `--state` argument? Also, >>>> would >>>>>>> that mean the default `--describe $group --verbose` command would >> not >>>> print >>>>>>> the added field to `--offsets verbose` (leader-epoch) or would it? >>>>>>>> >>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing >> "ASSIGNMENT" >>>>>>> column under `--members --verbose` does group the topic partitions >> by >>>> topic >>>>>>> but does not prefix the grouping with the topic name like you're >>>> proposal: >>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format >> for >>>> the >>>>>>> classic group as well? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote: >>>>>>>>> Hello PoAn, just of couple more minor comments: >>>>>>>>> >>>>>>>>> LM4. Regarding the new “protocol” field added to >> MemberDescription. >>>>>>> Should >>>>>>>>> we consider reusing the existing GroupProtocol enum instead of >>>> String? >>>>>>>>> (It’s the one we use from the consumer side to refer to the >>> protocol >>>> in >>>>>>>>> use, just missing Share I notice). >>>>>>>>> >>>>>>>>> LM5. Regarding the change to the output for >> kafka-consumer-groups, >>>> the >>>>>>>>> command shown does not include the —state option, but the output >>>> shows >>>>>>>>> state info (state, #members, epochs). I would guess that we want >> to >>>>>>> modify >>>>>>>>> the output only when we describe a group with the —state —verbose >>>>>>> option, >>>>>>>>> is my understanding right? If my understanding is right we’re >> just >>>>>>> missing >>>>>>>>> adding the —state in the example, and the KIP does not introduce >>> any >>>>>>>>> changes to the —describe —verbose option. (If not, it would mean >> a >>>>>>> bigger >>>>>>>>> change to the output of —describe —verbose which I expect is not >>> the >>>>>>>>> intention?) >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> Lianet >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <yangp...@gmail.com> >>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Chia-Ping, >>>>>>>>>> >>>>>>>>>> Thanks for the review and suggestions. >>>>>>>>>> >>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in >>>>>>>>>> ConsumerGroupDescribeRequest to 0-1. >>>>>>>>>> >>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to >> check >>>>>>> whether >>>>>>>>>> a member in consumer group uses “classic” or “consumer” >> protocol. >>>>>>>>>> >>>>>>>>>> CI2: Yes, a member in share group always uses “share” protocol. >>>>>>>>>> >>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”, and >>>> “share” >>>>>>>>>> protocol. >>>>>>>>>> >>>>>>>>>> BTW, the vote thread is in >>>>>>>>>> >> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> PoAn >>>>>>>>>> >>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai < >>> chia7...@apache.org> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> hi PoAn >>>>>>>>>>> >>>>>>>>>>> CI0: We have to bump the version of >> ConsumerGroupDescribeRequest >>>> as >>>>>>>>>> well, so server can distinguish the new and old behavior. >>>>>>>>>>> >>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan to >> use >>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return >> either >>>>>>>>>> "classic" or "consumer", right? >>>>>>>>>>> >>>>>>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription` >> too, >>> so >>>>>>> the >>>>>>>>>> new filed protocol in shared group is always "shared", right? >>>>>>>>>>> >>>>>>>>>>> CI3: Could you consider adding a table to show the value of the >>>>>>> protocol >>>>>>>>>> field in each case? Andrew has a beautiful table in KIP-1043 >> that >>>>>>> lists all >>>>>>>>>> possible protocol names. >>>>>>>>>>> >>>>>>>>>>> [0] >>>>>>>>>> >>>>>>> >>>> >>> >> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454 >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Chia-Ping >>>>>>>>>>> >>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote: >>>>>>>>>>>> Hi David, >>>>>>>>>>>> >>>>>>>>>>>> Thanks for the reminder. >>>>>>>>>>>> >>>>>>>>>>>> DJ4: Add related content to the KIP. >>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”. >>>>>>>>>>>> Add a new field “Protocol” to ConsumerGroupDescribeResponse. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> PoAn >>>>>>>>>>>> >>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot >>>>>>> <dja...@confluent.io.INVALID> >>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>> >>>>>>>>>>>>> DJ4: Could you please also include the required changes to >> the >>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface section? >> We >>>>>>>>>> basically >>>>>>>>>>>>> need to bump the version, add the new field, etc. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> David >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang < >> yangp...@gmail.com> >>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi David, >>>>>>>>>>>>>> >>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription, >>>>>>>>>>>>>> so the command line tool can show protocol information when >>>> users >>>>>>>>>> describe >>>>>>>>>>>>>> members. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If there is no further suggestion, I will start a vote >> thread >>>>>>> today. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot >>>>>>>>>> <dja...@confluent.io.INVALID> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> DJ3: That seems reasonable to me. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change the >>>> output >>>>>>> of >>>>>>>>>> the >>>>>>>>>>>>>>> tool too many times. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> David >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang < >>> yangp...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi David / Andrew, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up >>>>>>>>>> kafka-share-groups.sh >>>>>>>>>>>>>>>> implementation. >>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that >>>> using >>>>>>> new >>>>>>>>>>>>>> format >>>>>>>>>>>>>>>> is more clear. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The new format will be like >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>> >>> >> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs. >>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same topic >>> name. >>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote >>> thread >>>>>>> next >>>>>>>>>>>>>> Monday. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield < >>>>>>>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a more >>>>>>>>>> appropriate >>>>>>>>>>>>>>>> missing value, but >>>>>>>>>>>>>>>>> you got there first. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include >>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the >>>>>>>>>>>>>>>>> command line. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the >>>>>>>>>> implementation of >>>>>>>>>>>>>>>> the share groups >>>>>>>>>>>>>>>>> part of this if it helps. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Andrew >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>>> From: Frank Yang <yangp...@gmail.com> >>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48 >>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> >>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend >>> kafka-consumer-groups >>>>>>>>>> command >>>>>>>>>>>>>>>> line tool to support new consumer group >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi David, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like to get >>>> this >>>>>>> in >>>>>>>>>> AK >>>>>>>>>>>>>> 4.0 >>>>>>>>>>>>>>>> as well. I will do my best. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and >>>> TARGET-ASSIGNMENT-EPOCH >>>>>>>>>> before >>>>>>>>>>>>>>>> #MEMBERS. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value “-“. >>>>>>>>>> (reference < >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>> >>> >> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92 >>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT >>>>>>>>>> TARGET-EPOCH >>>>>>>>>>>>>>>> TARGET-ASSIGNMENT. >>>>>>>>>>>>>>>>> Remove GROUP-EPOCH. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> For assignment value, it follows current output >> (reference >>> < >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>> >>> >> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418 >>>>>>>>>>>>>>> ). >>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more clear. >>>>>>>>>>>>>>>>> If we would like to use this form, I will update both >>> output >>>> in >>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has >>>> protocol >>>>>>>>>> type at >>>>>>>>>>>>>>>> group level. >>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and >>>>>>>>>> ConsumerGroupDescribeResponseData >>>>>>>>>>>>>>>> don’t have protocol at member level. >>>>>>>>>>>>>>>>> Could we use a followup to add it? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before >> CURRENT-OFFSET. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot >>>>>>>>>> <dja...@confluent.io.INVALID >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor >>> comments/suggestions: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would >>>> suggest >>>>>>>>>> putting >>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before >>>> `#MEMBERS`. >>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will >> print >>>> out >>>>>>> N/A >>>>>>>>>> for >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this correct? >>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members --verbose`, >> I >>>>>>> wonder >>>>>>>>>> if we >>>>>>>>>>>>>>>>>> should use the following order and terms: CURRENT-EPOCH >>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT >>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the >>>> GROUP-EPOCH >>>>>>>>>>>>>> because >>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>> is already in the group description. The value `(0)` >> used >>>> for >>>>>>> the >>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will >> print >>>> out >>>>>>> the >>>>>>>>>>>>>> list >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`. >>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should also >>> add >>>>>>> the >>>>>>>>>>>>>>>> protocol >>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is >> possible >>> to >>>>>>> have >>>>>>>>>>>>>>>> `classic` >>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` group >>> during >>>> an >>>>>>>>>> online >>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer >>> protocol. >>>>>>> Having >>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>> information may be handy for administrators. What do you >>>> think? >>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets --verbose`, >> I >>>> would >>>>>>>>>>>>>> suggest >>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0 as >> it >>>>>>>>>> changes the >>>>>>>>>>>>>>>>>> output of the command. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> DJ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang < >>>> yangp...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when >>> users >>>>>>> give >>>>>>>>>>>>>>>> —verbose >>>>>>>>>>>>>>>>>>> option. >>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target >>> assignment >>>>>>>>>> epoch. >>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above information, >>>> member >>>>>>>>>> epoch, >>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> target assignment. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 >>>>>>> definition. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use >>>>>>> Optional >>>>>>>>>>>>>>>> fields in >>>>>>>>>>>>>>>>>>> ConsumerGroupDescription. >>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have >>> epoch, >>>> so >>>>>>> I >>>>>>>>>> use >>>>>>>>>>>>>> int >>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level >>>> information >>>>>>> by >>>>>>>>>>>>>>>> —describe >>>>>>>>>>>>>>>>>>> —verbose. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe >> —offsets >>>>>>>>>> —verbose. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. < >>> liane...@gmail.com >>>>> >>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of >> moving >>>> this >>>>>>>>>> into a >>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only attempt >> to >>>> make >>>>>>>>>> sense >>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying to >>> view >>>> the >>>>>>>>>> group >>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense to >>> me, >>>>>>> even >>>>>>>>>> more >>>>>>>>>>>>>>>> if we >>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both >> issues, >>>> my >>>>>>>>>> take is >>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect >>> member-epoch, >>>>>>>>>> whoever >>>>>>>>>>>>>> is >>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs at >> all. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we >>>> expect >>>>>>> to >>>>>>>>>>>>>> always >>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side for >>> the >>>>>>> group >>>>>>>>>>>>>> ones, >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> client side for the member) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in the >>>>>>> proposed >>>>>>>>>>>>>>>> changes? >>>>>>>>>>>>>>>>>>> ( >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>> >>> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state >>>>>>>>>>>>>>>>>>>> ). >>>>>>>>>>>>>>>>>>>> I get that the output would change in that example, >> but >>>> it’s >>>>>>> not >>>>>>>>>>>>>>>> because >>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's because >> of >>>> the >>>>>>>>>> change >>>>>>>>>>>>>>>>>>> proposed >>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), and the >>>>>>> changes >>>>>>>>>> to >>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems >> confusing, >>>> got >>>>>>> me >>>>>>>>>>>>>>>> looking >>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed unrelated). >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are >>>> conceptually >>>>>>>>>> at the >>>>>>>>>>>>>>>>>>> group >>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member level. >> So >>>>>>> wonder >>>>>>>>>> if we >>>>>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose option) >>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target >>>> assignment >>>>>>>>>> epoch >>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member >>> epoch >>>>>>>>>> (along >>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Lianet >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield < >>>>>>>>>>>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are most >>>>>>> useful to >>>>>>>>>>>>>>>> people >>>>>>>>>>>>>>>>>>>>> trying to understand >>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail. The >>>>>>>>>> information >>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>> really not understandable >>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of >>>>>>> KIP-848/932. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that >>> you >>>>>>> add a >>>>>>>>>>>>>>>>>>> --members >>>>>>>>>>>>>>>>>>>>> --verbose option >>>>>>>>>>>>>>>>>>>>> and only include the new information in the output >> for >>>> that >>>>>>>>>> option, >>>>>>>>>>>>>>>>>>> rather >>>>>>>>>>>>>>>>>>>>> than changing the >>>>>>>>>>>>>>>>>>>>> non-verbose --members output. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for >>>>>>> kafka-consumer-groups.sh >>>>>>>>>>>>>>>> --members >>>>>>>>>>>>>>>>>>>>> and only add the >>>>>>>>>>>>>>>>>>>>> new information for the --verbose output. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH >>>> instead of >>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH. >>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the >>>> terminology of >>>>>>>>>> the >>>>>>>>>>>>>>>> KIPs. >>>>>>>>>>>>>>>>>>> For >>>>>>>>>>>>>>>>>>>>> one thing, >>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client such as >>>>>>>>>>>>>>>> MemberDescription. >>>>>>>>>>>>>>>>>>>>> It's not a >>>>>>>>>>>>>>>>>>>>> new term. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>> Andrew >>>>>>>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pay...@apache.org> >>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55 >>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> >>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend >>>>>>> kafka-consumer-groups >>>>>>>>>>>>>> command >>>>>>>>>>>>>>>>>>> line >>>>>>>>>>>>>>>>>>>>> tool to support new consumer group >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Lucas, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks for the review! >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for >> kafka-share-groups.sh >>> to >>>>>>> the >>>>>>>>>> KIP. >>>>>>>>>>>>>>>> Could >>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use >>>> MEMBER-EPOCH >>>>>>>>>> here, >>>>>>>>>>>>>>>> users >>>>>>>>>>>>>>>>>>> may >>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and >> MEMBER. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote: >>>>>>>>>>>>>>>>>>>>>> Hi Frank, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> thanks for the KIP! >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for >>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? >>>> Even >>>>>>> if >>>>>>>>>> we do >>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group >>> implementation >>>>>>> may >>>>>>>>>> still >>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in the >>> KIP. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? >> That >>>> would >>>>>>>>>> seem >>>>>>>>>>>>>>>> more >>>>>>>>>>>>>>>>>>>>>> consistent. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>>>>>> Lucas >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang < >>>>>>>>>> yangp...@gmail.com> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of >> KIP-1099. >>>> This >>>>>>> KIP >>>>>>>>>>>>>>>> enhances >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state which >> is >>>>>>>>>> introduced >>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>> KIP-848. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line >>>> tool >>>>>>> to >>>>>>>>>>>>>> support >>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software >>>> Foundation >>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>> >>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org >>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>> >>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico] >>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>> >>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>> >>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >>>> >>> >>
