Hi David, Thanks for the feedback.
Yes, I agree that the “UPGRADED” column name is better. I updated the KIP. Best, PoAn > On Nov 25, 2024, at 11:11 PM, PoAn Yang <yangp...@gmail.com> wrote: > > Hi Andrew, > > Thanks for the feedback. > > AS4: Change TARGET-ASSIGNMENT-EPOCH to ASSIGNMENT-EPOCH for “—describe —state > —verbose" in kafka-share-groups.sh. > > AS5: Remove TARGET-EPOCH and TARGET-ASSIGNMENT. > Change CURRENT-EPOCH to MEMBER-EPOCH and CURRENT-ASSIGNMENT to ASSIGNMENT for > “—describe —member —verbose” in kafka-share-groups.sh. > > Thanks, > PoAn > >> On Nov 25, 2024, at 10:48 PM, Andrew Schofield >> <andrew_schofield_j...@outlook.com> wrote: >> >> Hi PoAn, >> I've taken another pass over the KIP and have some more comments around the >> share >> group support. >> >> AS4: Share groups do not have the concept of a target assignment. As a >> result, >> the verbose state TARGET-ASSIGNMENT-EPOCH is a bit misleading from my >> point of view. ASSIGNMENT-EPOCH is better. >> >> AS5: Similarly for the verbose members output, the TARGET-EPOCH and >> TARGET-ASSIGNMENT are not really appropriate. Personally, I would prefer >> just MEMBER-EPOCH and ASSIGNMENT, and remove the "target" fields. >> >> Thanks, >> Andrew >> >> ________________________________________ >> From: PoAn Yang <yangp...@gmail.com> >> Sent: 24 November 2024 15:08 >> To: dev@kafka.apache.org <dev@kafka.apache.org> >> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line >> tool to support new consumer group >> >> Hi Chia-Ping, >> >> Thanks for the suggestion. >> >> The “isClassic” field in ConsumerGroupDescribeResponse is a boolean value, >> so we can’t use it as the protocol string. >> I change the “PROTOCOL” column to “IS-CLASSIC” in kafka-consumer-groups.sh. >> It returns “true” if a member is in a classic group or it’s a classic member >> in a consumer group. >> In other cases, it returns “false”. >> >> I have a draft implementation in https://github.com/apache/kafka/pull/17872. >> >> Thanks, >> PoAn >> >>> On Nov 23, 2024, at 1:05 PM, Chia-Ping Tsai <chia7...@gmail.com> wrote: >>> >>> hi PoAn >>> >>> `isClassic` is good to me :) >>> >>> Best, >>> Chia-Ping >>> >>> PoAn Yang <yangp...@gmail.com> 於 2024年11月23日 週六 下午12:35寫道： >>> >>>> Hi Chia-Ping / David, >>>> >>>> Thanks for the great suggestion. >>>> >>>> How about using “isClassic”? >>>> The term like “isLegacy” may be equivocal in the future. >>>> For example, if we have a newer version of consumer / share consumer, the >>>> meaning will be different for different cases. >>>> If we use “isClassic”, for MemberDescription in ClassicGroupDescription, >>>> the value can be always “true”. >>>> In ConsumerGroupDescription, it’s “true” when it’s a classic member. >>>> In ShareGroupDescription, it’s always “false”. >>>> >>>> Thanks, >>>> PoAn >>>> >>>>> On Nov 23, 2024, at 1:02 AM, Chia-Ping Tsai <chia7...@gmail.com> wrote: >>>>> >>>>> hi DJ >>>>> >>>>>> My goal was to have a way to tell the user that his (new) consumer group >>>>> may have non-upgraded members yet. I wonder if we could use a boolean to >>>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column which >>>>> is only displayed when they are non upgraded members. What do you think? >>>>> >>>>> Yes, this is a good approach, but the downside is that the new isLegacy >>>>> field in MemberDescription would be irrelevant for classic and shared >>>>> groups. This would require thorough documentation to clarify its purpose. >>>>> >>>>> Additionally, we could use a boolean instead of a string for the new >>>> field >>>>> in the RPC to reduce the request size. >>>>> Best, >>>>> Chia-Ping >>>>> >>>>> >>>>> >>>>> David Jacot <dja...@confluent.io.invalid> 於 2024年11月23日 週六 上午12:41寫道： >>>>> >>>>>> Hi both, >>>>>> >>>>>> Ah, I understand what you mean now. You're saying that we have two >>>> protocol >>>>>> fields. One at the group level and one at the member level (the new >>>> one). >>>>>> The two fields are actually two different things. The former is the >>>> name of >>>>>> the embedded protocol used in the classic protocol whereas the latter is >>>>>> the name of the protocol/api used. This is indeed confusing. >>>>>> >>>>>> My goal was to have a way to tell the user that his (new) consumer group >>>>>> may have non-upgraded members yet. I wonder if we could use a boolean to >>>>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column >>>> which >>>>>> is only displayed when they are non upgraded members. What do you think? >>>>>> >>>>>> If we cannot find a good way to represent this, we could drop it from >>>> this >>>>>> KIP too. >>>>>> >>>>>> Best, >>>>>> David >>>>>> >>>>>> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <yangp...@gmail.com> wrote: >>>>>> >>>>>>> Hi Chia-Ping / David, >>>>>>> >>>>>>> Thanks for the review and suggestions. >>>>>>> >>>>>>>>> I don't fully follow your comment so please excuse me if I say >>>>>> something >>>>>>>>> wrong. >>>>>>> >>>>>>> I think Chia-Ping wants to say: originally, the default protocol in >>>>>>> classic group is “consumer”. >>>>>>> After we have a new consumer group, the protocol of a classic member in >>>>>>> consumer group is “classic”. >>>>>>> The MemberDescription is used by both ClassicGroupDescription and >>>>>>> ConsumerGroupDescription. >>>>>>> If we add the protocol field to MemberDescription, the protocol of a >>>>>>> classic member in classic group is “consumer”, but it’s “classic” in >>>>>>> consumer group. >>>>>>> To avoid misleading word, we can add a new collection of member >>>> protocols >>>>>>> to ConsumerGroupDescription only. >>>>>>> The original goal is to show member protocol during consumer migration, >>>>>> so >>>>>>> we don’t need to add this information to ClassicGroupDescription. >>>>>>> >>>>>>> Please correct me if I misunderstood anything. >>>>>>> >>>>>>> Thanks, >>>>>>> PoAn >>>>>>> >>>>>>>> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <chia7...@gmail.com> >>>>>> wrote: >>>>>>>> >>>>>>>> hi DJ >>>>>>>> >>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and >>>>>> change >>>>>>>> return type of ModernGroup#protocolType? >>>>>>>> >>>>>>>> This is not my comment ... sorry for the incorrect quote :( >>>>>>>> >>>>>>>>> A member in a classic group should report "classic". >>>>>>>> >>>>>>>> I completely agree. However, in this case, >>>>>>> ClassicGroupDescription#protocol >>>>>>>> returns 'consumer', while ClassicGroupDescription#members[x].protocol >>>>>>>> returns 'classic'. This inconsistency feels a bit strange to me. >>>>>>>> >>>>>>>> Best, >>>>>>>> Chia-Ping >>>>>>>> >>>>>>>> >>>>>>>> David Jacot <dja...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道： >>>>>>>> >>>>>>>>> Hi Chia, >>>>>>>>> >>>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but >>>>>> I >>>>>>>>> would like to align the format with kafka-groups.sh. >>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and >>>>>> change >>>>>>>>> return type of ModernGroup#protocolType? >>>>>>>>> >>>>>>>>> My understanding is that `GroupProtocol` is used to drive the config >>>>>> of >>>>>>> the >>>>>>>>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you >>>>>>> referring to >>>>>>>>> another GroupProtocol enum? Perhaps GroupType enum? >>>>>>>>> >>>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to >>>>>>>>> MemberDescription, >>>>>>>>> since the field has different values between classic and consumer >>>>>>> groups—it >>>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in >>>>>>> consumer >>>>>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency >>>>>>> feels >>>>>>>>> confusing to me. For example, users call Admin#describeConsumerGroups >>>>>>> and >>>>>>>>> see that a member has the 'consumer' protocol when it's in a classic >>>>>>> group. >>>>>>>>> Then, if the group is upgraded to a consumer group, calling >>>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic' >>>> for >>>>>>> the >>>>>>>>> same member. >>>>>>>>> >>>>>>>>> I don't fully follow your comment so please excuse me if I say >>>>>> something >>>>>>>>> wrong. >>>>>>>>> >>>>>>>>> I think that it should work as follows: >>>>>>>>> * A member in a classic group should report "classic". >>>>>>>>> * A member using the consumer protocol in a consumer group should >>>>>> report >>>>>>>>> "consumer". >>>>>>>>> * A member using the classic protocol in a consumer group should >>>>>> report >>>>>>>>> "classic". >>>>>>>>> * A member in a share group should report "share". >>>>>>>>> >>>>>>>>>> If all we want to do is distinguish classic members in the >>>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common >>>>>>>>> MemberDescription structure. Instead, we could add a collection to >>>>>>>>> ConsumerGroupDescription to trace members using the classic protocol, >>>>>>> which >>>>>>>>> would allow us to enrich the output of command-line tools. >>>>>>>>> >>>>>>>>> Personally, I believe that using different collections per protocol >>>>>>> types >>>>>>>>> is not necessary. A field in the member is more than enough. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> DJ >>>>>>>>> >>>>>>>>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <chia7...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> hi PoAn >>>>>>>>>> >>>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but >>>>>> I >>>>>>>>>> would like to align the format with kafka-groups.sh. >>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and >>>>>>> change >>>>>>>>>> return type of ModernGroup#protocolType? >>>>>>>>>> >>>>>>>>>> MemberDescription is also used by ClassicGroupDescription, and in >>>>>>> classic >>>>>>>>>> groups, the protocol can be an arbitrary string, so we can't use an >>>>>>> enum. >>>>>>>>>> >>>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to >>>>>>>>>> MemberDescription, >>>>>>>>>> since the field has different values between classic and consumer >>>>>>>>> groups—it >>>>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in >>>>>>>>> consumer >>>>>>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency >>>>>>>>> feels >>>>>>>>>> confusing to me. For example, users call >>>> Admin#describeConsumerGroups >>>>>>> and >>>>>>>>>> see that a member has the 'consumer' protocol when it's in a classic >>>>>>>>> group. >>>>>>>>>> Then, if the group is upgraded to a consumer group, calling >>>>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic' >>>>>> for >>>>>>>>> the >>>>>>>>>> same member. >>>>>>>>>> >>>>>>>>>> If all we want to do is distinguish classic members in the >>>>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common >>>>>>>>>> MemberDescription structure. Instead, we could add a collection to >>>>>>>>>> ConsumerGroupDescription to trace members using the classic >>>> protocol, >>>>>>>>> which >>>>>>>>>> would allow us to enrich the output of command-line tools. >>>>>>>>>> Best, >>>>>>>>>> Chia-Ping >>>>>>>>>> >>>>>>>>>> PoAn Yang <yangp...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道： >>>>>>>>>> >>>>>>>>>>> Hi Lianet / Jeff, >>>>>>>>>>> >>>>>>>>>>> Thanks for the review. >>>>>>>>>>> >>>>>>>>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show >>>> offsets >>>>>>>>>>> information. >>>>>>>>>>> Change to use “—describe —state —verbose” to show group level >>>>>>>>> information >>>>>>>>>>> in kafka-share-groups.sh. >>>>>>>>>>> >>>>>>>>>>> LM6: Update the example. >>>>>>>>>>> >>>>>>>>>>> LM7: Add a description to mention “—verbose” is a new option in >>>>>>>>>>> ShareGroupCommandOptions. >>>>>>>>>>> >>>>>>>>>>> JK2: Yes, add the statement about format change for ASSIGNMENT >>>>>> value. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> PoAn >>>>>>>>>>> >>>>>>>>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <jeffkb...@apache.org> >>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>> >>>>>>>>>>>> JK2: Can we state in the KIP that we will reformat the existing >>>>>>>>>>>> classic group's `--members --verbose` as well, specifically the >>>>>>>>>>>> "ASSIGNMENT" column to include the topic name? >>>>>>>>>>>> >>>>>>>>>>>> Thanks! >>>>>>>>>>>> >>>>>>>>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote: >>>>>>>>>>>>> Thanks for the updates PoAn! >>>>>>>>>>>>> >>>>>>>>>>>>> LM6. Just a nit, under the --describe --state --verbose the >>>> header >>>>>>>>> is >>>>>>>>>>> fine >>>>>>>>>>>>> but the example is still missing the --state argument. >>>>>>>>>>>>> >>>>>>>>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, should >>>>>> we >>>>>>>>>>>>> clearly mention that the KIP adds a new --verbose option? I think >>>>>>>>> it's >>>>>>>>>>>>> confusing because it's presented as if the option already exists >>>>>> and >>>>>>>>>>> we're >>>>>>>>>>>>> only changing the output (which is the case for >>>>>>>>> kafka-consumer-groups >>>>>>>>>>> but >>>>>>>>>>>>> not for kafka-share-groups) >>>>>>>>>>>>> >>>>>>>>>>>>> That's all on my side. Thanks! >>>>>>>>>>>>> >>>>>>>>>>>>> Lianet >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <yangp...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Lianet / Jeff, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for review and suggestions. >>>>>>>>>>>>>> >>>>>>>>>>>>>> LM4: At group level, the protocol field is from >>>>>>>>>>> ModernGroup#protocolType. >>>>>>>>>>>>>> It’s a string. >>>>>>>>>>>>>> The value is defined in different places like >>>>>>>>>> ShareGroup#PROTOCOL_TYPE >>>>>>>>>>> and >>>>>>>>>>>>>> ConsumerProtocol#PROTOCOL_TYPE. >>>>>>>>>>>>>> In KIP-1043 < >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh >>>>>>>>>>>> , >>>>>>>>>>>>>> the kafka-groups.sh also shows protocol like “classic” / >>>>>>>>> “consumer” / >>>>>>>>>>>>>> “share”. >>>>>>>>>>>>>> I agree it’s better to use a type like GroupProtocol enum, but I >>>>>>>>>> would >>>>>>>>>>>>>> like to align the format with kafka-groups.sh. >>>>>>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol >>>> and >>>>>>>>>>> change >>>>>>>>>>>>>> return type of ModernGroup#protocolType? >>>>>>>>>>>>>> >>>>>>>>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in >>>> the >>>>>>>>>>> title. >>>>>>>>>>>>>> You’re correct. I would like to use `—describe —state —verbose` >>>>>> to >>>>>>>>>> show >>>>>>>>>>>>>> group level information. >>>>>>>>>>>>>> For `—describe —verbose`, it will show output as same as >>>>>> `—describe >>>>>>>>>>>>>> —offsets —verbose`, >>>>>>>>>>>>>> because `—describe` also shows output as same as `—describe >>>>>>>>>> —offsets`. >>>>>>>>>>>>>> >>>>>>>>>>>>>> JK2: Yes, I only use consumer group in the sample output, but >>>> the >>>>>>>>>>> classic >>>>>>>>>>>>>> group will use the same format as well. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <jeffkb...@apache.org> >>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for the KIP. I have some questions: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe >>>> --verbose` >>>>>>>>>>>>>> proposed changes - doesn't `--describe $group` default to >>>>>> printing >>>>>>>>>> the >>>>>>>>>>>>>> offsets? Perhaps you're referring to the `--state` argument? >>>>>> Also, >>>>>>>>>>> would >>>>>>>>>>>>>> that mean the default `--describe $group --verbose` command >>>> would >>>>>>>>> not >>>>>>>>>>> print >>>>>>>>>>>>>> the added field to `--offsets verbose` (leader-epoch) or would >>>>>> it? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing >>>>>>>>> "ASSIGNMENT" >>>>>>>>>>>>>> column under `--members --verbose` does group the topic >>>>>> partitions >>>>>>>>> by >>>>>>>>>>> topic >>>>>>>>>>>>>> but does not prefix the grouping with the topic name like you're >>>>>>>>>>> proposal: >>>>>>>>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format >>>>>>>>> for >>>>>>>>>>> the >>>>>>>>>>>>>> classic group as well? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote: >>>>>>>>>>>>>>>> Hello PoAn, just of couple more minor comments: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> LM4. Regarding the new “protocol” field added to >>>>>>>>> MemberDescription. >>>>>>>>>>>>>> Should >>>>>>>>>>>>>>>> we consider reusing the existing GroupProtocol enum instead of >>>>>>>>>>> String? >>>>>>>>>>>>>>>> (It’s the one we use from the consumer side to refer to the >>>>>>>>>> protocol >>>>>>>>>>> in >>>>>>>>>>>>>>>> use, just missing Share I notice). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> LM5. Regarding the change to the output for >>>>>>>>> kafka-consumer-groups, >>>>>>>>>>> the >>>>>>>>>>>>>>>> command shown does not include the —state option, but the >>>>>> output >>>>>>>>>>> shows >>>>>>>>>>>>>>>> state info (state, #members, epochs). I would guess that we >>>>>> want >>>>>>>>> to >>>>>>>>>>>>>> modify >>>>>>>>>>>>>>>> the output only when we describe a group with the —state >>>>>> —verbose >>>>>>>>>>>>>> option, >>>>>>>>>>>>>>>> is my understanding right? If my understanding is right we’re >>>>>>>>> just >>>>>>>>>>>>>> missing >>>>>>>>>>>>>>>> adding the —state in the example, and the KIP does not >>>>>> introduce >>>>>>>>>> any >>>>>>>>>>>>>>>> changes to the —describe —verbose option. (If not, it would >>>>>> mean >>>>>>>>> a >>>>>>>>>>>>>> bigger >>>>>>>>>>>>>>>> change to the output of —describe —verbose which I expect is >>>>>> not >>>>>>>>>> the >>>>>>>>>>>>>>>> intention?) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Lianet >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <yangp...@gmail.com >>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Chia-Ping, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks for the review and suggestions. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in >>>>>>>>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to >>>>>>>>> check >>>>>>>>>>>>>> whether >>>>>>>>>>>>>>>>> a member in consumer group uses “classic” or “consumer” >>>>>>>>> protocol. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> CI2: Yes, a member in share group always uses “share” >>>>>> protocol. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”, >>>> and >>>>>>>>>>> “share” >>>>>>>>>>>>>>>>> protocol. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> BTW, the vote thread is in >>>>>>>>>>>>>>>>> >>>>>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai < >>>>>>>>>> chia7...@apache.org> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> hi PoAn >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> CI0: We have to bump the version of >>>>>>>>> ConsumerGroupDescribeRequest >>>>>>>>>>> as >>>>>>>>>>>>>>>>> well, so server can distinguish the new and old behavior. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan to >>>>>>>>> use >>>>>>>>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return >>>>>>>>> either >>>>>>>>>>>>>>>>> "classic" or "consumer", right? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription` >>>>>>>>> too, >>>>>>>>>> so >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> new filed protocol in shared group is always "shared", right? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> CI3: Could you consider adding a table to show the value of >>>>>> the >>>>>>>>>>>>>> protocol >>>>>>>>>>>>>>>>> field in each case? Andrew has a beautiful table in KIP-1043 >>>>>>>>> that >>>>>>>>>>>>>> lists all >>>>>>>>>>>>>>>>> possible protocol names. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [0] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote: >>>>>>>>>>>>>>>>>>> Hi David, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks for the reminder. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> DJ4: Add related content to the KIP. >>>>>>>>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to >>>>>> “0-1”. >>>>>>>>>>>>>>>>>>> Add a new field “Protocol” to >>>> ConsumerGroupDescribeResponse. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot >>>>>>>>>>>>>> <dja...@confluent.io.INVALID> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> DJ4: Could you please also include the required changes to >>>>>>>>> the >>>>>>>>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface >>>> section? >>>>>>>>> We >>>>>>>>>>>>>>>>> basically >>>>>>>>>>>>>>>>>>>> need to bump the version, add the new field, etc. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>> David >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang < >>>>>>>>> yangp...@gmail.com> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi David, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription, >>>>>>>>>>>>>>>>>>>>> so the command line tool can show protocol information >>>>>> when >>>>>>>>>>> users >>>>>>>>>>>>>>>>> describe >>>>>>>>>>>>>>>>>>>>> members. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote >>>>>>>>> thread >>>>>>>>>>>>>> today. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot >>>>>>>>>>>>>>>>> <dja...@confluent.io.INVALID> >>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> DJ3: That seems reasonable to me. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change >>>> the >>>>>>>>>>> output >>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> tool too many times. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>> David >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang < >>>>>>>>>> yangp...@gmail.com> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi David / Andrew, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up >>>>>>>>>>>>>>>>> kafka-share-groups.sh >>>>>>>>>>>>>>>>>>>>>>> implementation. >>>>>>>>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think >>>>>> that >>>>>>>>>>> using >>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>> format >>>>>>>>>>>>>>>>>>>>>>> is more clear. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> The new format will be like >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs. >>>>>>>>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same >>>> topic >>>>>>>>>> name. >>>>>>>>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote >>>>>>>>>> thread >>>>>>>>>>>>>> next >>>>>>>>>>>>>>>>>>>>> Monday. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield < >>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a >>>>>> more >>>>>>>>>>>>>>>>> appropriate >>>>>>>>>>>>>>>>>>>>>>> missing value, but >>>>>>>>>>>>>>>>>>>>>>>> you got there first. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include >>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the >>>>>>>>>>>>>>>>>>>>>>>> command line. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the >>>>>>>>>>>>>>>>> implementation of >>>>>>>>>>>>>>>>>>>>>>> the share groups >>>>>>>>>>>>>>>>>>>>>>>> part of this if it helps. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>> Andrew >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>>>>>>>>>> From: Frank Yang <yangp...@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48 >>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> >>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend >>>>>>>>>> kafka-consumer-groups >>>>>>>>>>>>>>>>> command >>>>>>>>>>>>>>>>>>>>>>> line tool to support new consumer group >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi David, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like to >>>>>> get >>>>>>>>>>> this >>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>> AK >>>>>>>>>>>>>>>>>>>>> 4.0 >>>>>>>>>>>>>>>>>>>>>>> as well. I will do my best. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and >>>>>>>>>>> TARGET-ASSIGNMENT-EPOCH >>>>>>>>>>>>>>>>> before >>>>>>>>>>>>>>>>>>>>>>> #MEMBERS. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value >>>>>> “-“. >>>>>>>>>>>>>>>>> (reference < >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92 >>>>>>>>>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH >>>> CURRENT-ASSIGNMENT >>>>>>>>>>>>>>>>> TARGET-EPOCH >>>>>>>>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT. >>>>>>>>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> For assignment value, it follows current output >>>>>>>>> (reference >>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418 >>>>>>>>>>>>>>>>>>>>>> ). >>>>>>>>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more >>>> clear. >>>>>>>>>>>>>>>>>>>>>>>> If we would like to use this form, I will update both >>>>>>>>>> output >>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has >>>>>>>>>>> protocol >>>>>>>>>>>>>>>>> type at >>>>>>>>>>>>>>>>>>>>>>> group level. >>>>>>>>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and >>>>>>>>>>>>>>>>> ConsumerGroupDescribeResponseData >>>>>>>>>>>>>>>>>>>>>>> don’t have protocol at member level. >>>>>>>>>>>>>>>>>>>>>>>> Could we use a followup to add it? >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before >>>>>>>>> CURRENT-OFFSET. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot >>>>>>>>>>>>>>>>> <dja...@confluent.io.INVALID >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor >>>>>>>>>> comments/suggestions: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would >>>>>>>>>>> suggest >>>>>>>>>>>>>>>>> putting >>>>>>>>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before >>>>>>>>>>> `#MEMBERS`. >>>>>>>>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will >>>>>>>>> print >>>>>>>>>>> out >>>>>>>>>>>>>> N/A >>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this >>>>>> correct? >>>>>>>>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members >>>>>> --verbose`, >>>>>>>>> I >>>>>>>>>>>>>> wonder >>>>>>>>>>>>>>>>> if we >>>>>>>>>>>>>>>>>>>>>>>>> should use the following order and terms: >>>>>> CURRENT-EPOCH >>>>>>>>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT >>>>>>>>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the >>>>>>>>>>> GROUP-EPOCH >>>>>>>>>>>>>>>>>>>>> because >>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>> is already in the group description. The value `(0)` >>>>>>>>> used >>>>>>>>>>> for >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will >>>>>>>>> print >>>>>>>>>>> out >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> list >>>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`. >>>>>>>>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should >>>>>> also >>>>>>>>>> add >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> protocol >>>>>>>>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is >>>>>>>>> possible >>>>>>>>>> to >>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>>>> `classic` >>>>>>>>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` group >>>>>>>>>> during >>>>>>>>>>> an >>>>>>>>>>>>>>>>> online >>>>>>>>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer >>>>>>>>>> protocol. >>>>>>>>>>>>>> Having >>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>> information may be handy for administrators. What do >>>>>> you >>>>>>>>>>> think? >>>>>>>>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets >>>>>> --verbose`, >>>>>>>>> I >>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>> suggest >>>>>>>>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0 >>>>>> as >>>>>>>>> it >>>>>>>>>>>>>>>>> changes the >>>>>>>>>>>>>>>>>>>>>>>>> output of the command. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>> DJ >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang < >>>>>>>>>>> yangp...@gmail.com> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information >>>> when >>>>>>>>>> users >>>>>>>>>>>>>> give >>>>>>>>>>>>>>>>>>>>>>> —verbose >>>>>>>>>>>>>>>>>>>>>>>>>> option. >>>>>>>>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target >>>>>>>>>> assignment >>>>>>>>>>>>>>>>> epoch. >>>>>>>>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above >>>> information, >>>>>>>>>>> member >>>>>>>>>>>>>>>>> epoch, >>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>> target assignment. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with >>>> KIP-848 >>>>>>>>>>>>>> definition. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I >>>>>> use >>>>>>>>>>>>>> Optional >>>>>>>>>>>>>>>>>>>>>>> fields in >>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription. >>>>>>>>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have >>>>>>>>>> epoch, >>>>>>>>>>> so >>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>> use >>>>>>>>>>>>>>>>>>>>> int >>>>>>>>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level >>>>>>>>>>> information >>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>> —describe >>>>>>>>>>>>>>>>>>>>>>>>>> —verbose. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe >>>>>>>>> —offsets >>>>>>>>>>>>>>>>> —verbose. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. < >>>>>>>>>> liane...@gmail.com >>>>>>>>>>>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of >>>>>>>>> moving >>>>>>>>>>> this >>>>>>>>>>>>>>>>> into a >>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only >>>>>> attempt >>>>>>>>> to >>>>>>>>>>> make >>>>>>>>>>>>>>>>> sense >>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying >>>> to >>>>>>>>>> view >>>>>>>>>>> the >>>>>>>>>>>>>>>>> group >>>>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense >>>>>> to >>>>>>>>>> me, >>>>>>>>>>>>>> even >>>>>>>>>>>>>>>>> more >>>>>>>>>>>>>>>>>>>>>>> if we >>>>>>>>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both >>>>>>>>> issues, >>>>>>>>>>> my >>>>>>>>>>>>>>>>> take is >>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect >>>>>>>>>> member-epoch, >>>>>>>>>>>>>>>>> whoever >>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs at >>>>>>>>> all. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't >>>>>> we >>>>>>>>>>> expect >>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> always >>>>>>>>>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side >>>>>> for >>>>>>>>>> the >>>>>>>>>>>>>> group >>>>>>>>>>>>>>>>>>>>> ones, >>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>> client side for the member) >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in >>>>>> the >>>>>>>>>>>>>> proposed >>>>>>>>>>>>>>>>>>>>>>> changes? >>>>>>>>>>>>>>>>>>>>>>>>>> ( >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state >>>>>>>>>>>>>>>>>>>>>>>>>>> ). >>>>>>>>>>>>>>>>>>>>>>>>>>> I get that the output would change in that example, >>>>>>>>> but >>>>>>>>>>> it’s >>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>> because >>>>>>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's >>>> because >>>>>>>>> of >>>>>>>>>>> the >>>>>>>>>>>>>>>>> change >>>>>>>>>>>>>>>>>>>>>>>>>> proposed >>>>>>>>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), and >>>>>> the >>>>>>>>>>>>>> changes >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems >>>>>>>>> confusing, >>>>>>>>>>> got >>>>>>>>>>>>>> me >>>>>>>>>>>>>>>>>>>>>>> looking >>>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed >>>>>> unrelated). >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are >>>>>>>>>>> conceptually >>>>>>>>>>>>>>>>> at the >>>>>>>>>>>>>>>>>>>>>>>>>> group >>>>>>>>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member >>>> level. >>>>>>>>> So >>>>>>>>>>>>>> wonder >>>>>>>>>>>>>>>>> if we >>>>>>>>>>>>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose >>>>>> option) >>>>>>>>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target >>>>>>>>>>> assignment >>>>>>>>>>>>>>>>> epoch >>>>>>>>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member >>>>>>>>>> epoch >>>>>>>>>>>>>>>>> (along >>>>>>>>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch) >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Lianet >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield < >>>>>>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are >>>>>> most >>>>>>>>>>>>>> useful to >>>>>>>>>>>>>>>>>>>>>>> people >>>>>>>>>>>>>>>>>>>>>>>>>>>> trying to understand >>>>>>>>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail. >>>>>> The >>>>>>>>>>>>>>>>> information >>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>> really not understandable >>>>>>>>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of >>>>>>>>>>>>>> KIP-848/932. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh >>>>>> that >>>>>>>>>> you >>>>>>>>>>>>>> add a >>>>>>>>>>>>>>>>>>>>>>>>>> --members >>>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option >>>>>>>>>>>>>>>>>>>>>>>>>>>> and only include the new information in the output >>>>>>>>> for >>>>>>>>>>> that >>>>>>>>>>>>>>>>> option, >>>>>>>>>>>>>>>>>>>>>>>>>> rather >>>>>>>>>>>>>>>>>>>>>>>>>>>> than changing the >>>>>>>>>>>>>>>>>>>>>>>>>>>> non-verbose --members output. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for >>>>>>>>>>>>>> kafka-consumer-groups.sh >>>>>>>>>>>>>>>>>>>>>>> --members >>>>>>>>>>>>>>>>>>>>>>>>>>>> and only add the >>>>>>>>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH >>>>>>>>>>> instead of >>>>>>>>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH. >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the >>>>>>>>>>> terminology of >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> KIPs. >>>>>>>>>>>>>>>>>>>>>>>>>> For >>>>>>>>>>>>>>>>>>>>>>>>>>>> one thing, >>>>>>>>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client such >>>>>> as >>>>>>>>>>>>>>>>>>>>>>> MemberDescription. >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's not a >>>>>>>>>>>>>>>>>>>>>>>>>>>> new term. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>> Andrew >>>>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pay...@apache.org> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55 >>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend >>>>>>>>>>>>>> kafka-consumer-groups >>>>>>>>>>>>>>>>>>>>> command >>>>>>>>>>>>>>>>>>>>>>>>>> line >>>>>>>>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lucas, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review! >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for >>>>>>>>> kafka-share-groups.sh >>>>>>>>>> to >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> KIP. >>>>>>>>>>>>>>>>>>>>>>> Could >>>>>>>>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use >>>>>>>>>>> MEMBER-EPOCH >>>>>>>>>>>>>>>>> here, >>>>>>>>>>>>>>>>>>>>>>> users >>>>>>>>>>>>>>>>>>>>>>>>>> may >>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and >>>>>>>>> MEMBER. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Frank, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP! >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for >>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, >>>>>> etc. ? >>>>>>>>>>> Even >>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>> we do >>>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group >>>>>>>>>> implementation >>>>>>>>>>>>>> may >>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in >>>> the >>>>>>>>>> KIP. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? >>>>>>>>> That >>>>>>>>>>> would >>>>>>>>>>>>>>>>> seem >>>>>>>>>>>>>>>>>>>>>>> more >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consistent. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lucas >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang < >>>>>>>>>>>>>>>>> yangp...@gmail.com> >>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of >>>>>>>>> KIP-1099. >>>>>>>>>>> This >>>>>>>>>>>>>> KIP >>>>>>>>>>>>>>>>>>>>>>> enhances >>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state >>>>>> which >>>>>>>>> is >>>>>>>>>>>>>>>>> introduced >>>>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-848. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command >>>>>> line >>>>>>>>>>> tool >>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> support >>>>>>>>>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software >>>>>>>>>>> Foundation >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >>>> >>>> >
