Hi Andrew, Thanks for the feedback.
AS4: Change TARGET-ASSIGNMENT-EPOCH to ASSIGNMENT-EPOCH for “—describe —state —verbose" in kafka-share-groups.sh. AS5: Remove TARGET-EPOCH and TARGET-ASSIGNMENT. Change CURRENT-EPOCH to MEMBER-EPOCH and CURRENT-ASSIGNMENT to ASSIGNMENT for “—describe —member —verbose” in kafka-share-groups.sh. Thanks, PoAn > On Nov 25, 2024, at 10:48 PM, Andrew Schofield > <andrew_schofield_j...@outlook.com> wrote: > > Hi PoAn, > I've taken another pass over the KIP and have some more comments around the > share > group support. > > AS4: Share groups do not have the concept of a target assignment. As a result, > the verbose state TARGET-ASSIGNMENT-EPOCH is a bit misleading from my > point of view. ASSIGNMENT-EPOCH is better. > > AS5: Similarly for the verbose members output, the TARGET-EPOCH and > TARGET-ASSIGNMENT are not really appropriate. Personally, I would prefer > just MEMBER-EPOCH and ASSIGNMENT, and remove the "target" fields. > > Thanks, > Andrew > > ________________________________________ > From: PoAn Yang <yangp...@gmail.com> > Sent: 24 November 2024 15:08 > To: dev@kafka.apache.org <dev@kafka.apache.org> > Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line > tool to support new consumer group > > Hi Chia-Ping, > > Thanks for the suggestion. > > The “isClassic” field in ConsumerGroupDescribeResponse is a boolean value, so > we can’t use it as the protocol string. > I change the “PROTOCOL” column to “IS-CLASSIC” in kafka-consumer-groups.sh. > It returns “true” if a member is in a classic group or it’s a classic member > in a consumer group. > In other cases, it returns “false”. > > I have a draft implementation in https://github.com/apache/kafka/pull/17872. > > Thanks, > PoAn > >> On Nov 23, 2024, at 1:05 PM, Chia-Ping Tsai <chia7...@gmail.com> wrote: >> >> hi PoAn >> >> `isClassic` is good to me :) >> >> Best, >> Chia-Ping >> >> PoAn Yang <yangp...@gmail.com> 於 2024年11月23日 週六 下午12:35寫道： >> >>> Hi Chia-Ping / David, >>> >>> Thanks for the great suggestion. >>> >>> How about using “isClassic”? >>> The term like “isLegacy” may be equivocal in the future. >>> For example, if we have a newer version of consumer / share consumer, the >>> meaning will be different for different cases. >>> If we use “isClassic”, for MemberDescription in ClassicGroupDescription, >>> the value can be always “true”. >>> In ConsumerGroupDescription, it’s “true” when it’s a classic member. >>> In ShareGroupDescription, it’s always “false”. >>> >>> Thanks, >>> PoAn >>> >>>> On Nov 23, 2024, at 1:02 AM, Chia-Ping Tsai <chia7...@gmail.com> wrote: >>>> >>>> hi DJ >>>> >>>>> My goal was to have a way to tell the user that his (new) consumer group >>>> may have non-upgraded members yet. I wonder if we could use a boolean to >>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column which >>>> is only displayed when they are non upgraded members. What do you think? >>>> >>>> Yes, this is a good approach, but the downside is that the new isLegacy >>>> field in MemberDescription would be irrelevant for classic and shared >>>> groups. This would require thorough documentation to clarify its purpose. >>>> >>>> Additionally, we could use a boolean instead of a string for the new >>> field >>>> in the RPC to reduce the request size. >>>> Best, >>>> Chia-Ping >>>> >>>> >>>> >>>> David Jacot <dja...@confluent.io.invalid> 於 2024年11月23日 週六 上午12:41寫道： >>>> >>>>> Hi both, >>>>> >>>>> Ah, I understand what you mean now. You're saying that we have two >>> protocol >>>>> fields. One at the group level and one at the member level (the new >>> one). >>>>> The two fields are actually two different things. The former is the >>> name of >>>>> the embedded protocol used in the classic protocol whereas the latter is >>>>> the name of the protocol/api used. This is indeed confusing. >>>>> >>>>> My goal was to have a way to tell the user that his (new) consumer group >>>>> may have non-upgraded members yet. I wonder if we could use a boolean to >>>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column >>> which >>>>> is only displayed when they are non upgraded members. What do you think? >>>>> >>>>> If we cannot find a good way to represent this, we could drop it from >>> this >>>>> KIP too. >>>>> >>>>> Best, >>>>> David >>>>> >>>>> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <yangp...@gmail.com> wrote: >>>>> >>>>>> Hi Chia-Ping / David, >>>>>> >>>>>> Thanks for the review and suggestions. >>>>>> >>>>>>>> I don't fully follow your comment so please excuse me if I say >>>>> something >>>>>>>> wrong. >>>>>> >>>>>> I think Chia-Ping wants to say: originally, the default protocol in >>>>>> classic group is “consumer”. >>>>>> After we have a new consumer group, the protocol of a classic member in >>>>>> consumer group is “classic”. >>>>>> The MemberDescription is used by both ClassicGroupDescription and >>>>>> ConsumerGroupDescription. >>>>>> If we add the protocol field to MemberDescription, the protocol of a >>>>>> classic member in classic group is “consumer”, but it’s “classic” in >>>>>> consumer group. >>>>>> To avoid misleading word, we can add a new collection of member >>> protocols >>>>>> to ConsumerGroupDescription only. >>>>>> The original goal is to show member protocol during consumer migration, >>>>> so >>>>>> we don’t need to add this information to ClassicGroupDescription. >>>>>> >>>>>> Please correct me if I misunderstood anything. >>>>>> >>>>>> Thanks, >>>>>> PoAn >>>>>> >>>>>>> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <chia7...@gmail.com> >>>>> wrote: >>>>>>> >>>>>>> hi DJ >>>>>>> >>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and >>>>> change >>>>>>> return type of ModernGroup#protocolType? >>>>>>> >>>>>>> This is not my comment ... sorry for the incorrect quote :( >>>>>>> >>>>>>>> A member in a classic group should report "classic". >>>>>>> >>>>>>> I completely agree. However, in this case, >>>>>> ClassicGroupDescription#protocol >>>>>>> returns 'consumer', while ClassicGroupDescription#members[x].protocol >>>>>>> returns 'classic'. This inconsistency feels a bit strange to me. >>>>>>> >>>>>>> Best, >>>>>>> Chia-Ping >>>>>>> >>>>>>> >>>>>>> David Jacot <dja...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道： >>>>>>> >>>>>>>> Hi Chia, >>>>>>>> >>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but >>>>> I >>>>>>>> would like to align the format with kafka-groups.sh. >>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and >>>>> change >>>>>>>> return type of ModernGroup#protocolType? >>>>>>>> >>>>>>>> My understanding is that `GroupProtocol` is used to drive the config >>>>> of >>>>>> the >>>>>>>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you >>>>>> referring to >>>>>>>> another GroupProtocol enum? Perhaps GroupType enum? >>>>>>>> >>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to >>>>>>>> MemberDescription, >>>>>>>> since the field has different values between classic and consumer >>>>>> groups—it >>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in >>>>>> consumer >>>>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency >>>>>> feels >>>>>>>> confusing to me. For example, users call Admin#describeConsumerGroups >>>>>> and >>>>>>>> see that a member has the 'consumer' protocol when it's in a classic >>>>>> group. >>>>>>>> Then, if the group is upgraded to a consumer group, calling >>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic' >>> for >>>>>> the >>>>>>>> same member. >>>>>>>> >>>>>>>> I don't fully follow your comment so please excuse me if I say >>>>> something >>>>>>>> wrong. >>>>>>>> >>>>>>>> I think that it should work as follows: >>>>>>>> * A member in a classic group should report "classic". >>>>>>>> * A member using the consumer protocol in a consumer group should >>>>> report >>>>>>>> "consumer". >>>>>>>> * A member using the classic protocol in a consumer group should >>>>> report >>>>>>>> "classic". >>>>>>>> * A member in a share group should report "share". >>>>>>>> >>>>>>>>> If all we want to do is distinguish classic members in the >>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common >>>>>>>> MemberDescription structure. Instead, we could add a collection to >>>>>>>> ConsumerGroupDescription to trace members using the classic protocol, >>>>>> which >>>>>>>> would allow us to enrich the output of command-line tools. >>>>>>>> >>>>>>>> Personally, I believe that using different collections per protocol >>>>>> types >>>>>>>> is not necessary. A field in the member is more than enough. >>>>>>>> >>>>>>>> Best, >>>>>>>> DJ >>>>>>>> >>>>>>>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <chia7...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> hi PoAn >>>>>>>>> >>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol enum, but >>>>> I >>>>>>>>> would like to align the format with kafka-groups.sh. >>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and >>>>>> change >>>>>>>>> return type of ModernGroup#protocolType? >>>>>>>>> >>>>>>>>> MemberDescription is also used by ClassicGroupDescription, and in >>>>>> classic >>>>>>>>> groups, the protocol can be an arbitrary string, so we can't use an >>>>>> enum. >>>>>>>>> >>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to >>>>>>>>> MemberDescription, >>>>>>>>> since the field has different values between classic and consumer >>>>>>>> groups—it >>>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in >>>>>>>> consumer >>>>>>>>> groups. Yes, it seems like a typo, but it's true. This inconsistency >>>>>>>> feels >>>>>>>>> confusing to me. For example, users call >>> Admin#describeConsumerGroups >>>>>> and >>>>>>>>> see that a member has the 'consumer' protocol when it's in a classic >>>>>>>> group. >>>>>>>>> Then, if the group is upgraded to a consumer group, calling >>>>>>>>> Admin#describeConsumerGroups again shows the protocol as 'classic' >>>>> for >>>>>>>> the >>>>>>>>> same member. >>>>>>>>> >>>>>>>>> If all we want to do is distinguish classic members in the >>>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common >>>>>>>>> MemberDescription structure. Instead, we could add a collection to >>>>>>>>> ConsumerGroupDescription to trace members using the classic >>> protocol, >>>>>>>> which >>>>>>>>> would allow us to enrich the output of command-line tools. >>>>>>>>> Best, >>>>>>>>> Chia-Ping >>>>>>>>> >>>>>>>>> PoAn Yang <yangp...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道： >>>>>>>>> >>>>>>>>>> Hi Lianet / Jeff, >>>>>>>>>> >>>>>>>>>> Thanks for the review. >>>>>>>>>> >>>>>>>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show >>> offsets >>>>>>>>>> information. >>>>>>>>>> Change to use “—describe —state —verbose” to show group level >>>>>>>> information >>>>>>>>>> in kafka-share-groups.sh. >>>>>>>>>> >>>>>>>>>> LM6: Update the example. >>>>>>>>>> >>>>>>>>>> LM7: Add a description to mention “—verbose” is a new option in >>>>>>>>>> ShareGroupCommandOptions. >>>>>>>>>> >>>>>>>>>> JK2: Yes, add the statement about format change for ASSIGNMENT >>>>> value. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> PoAn >>>>>>>>>> >>>>>>>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <jeffkb...@apache.org> >>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi PoAn, >>>>>>>>>>> >>>>>>>>>>> JK2: Can we state in the KIP that we will reformat the existing >>>>>>>>>>> classic group's `--members --verbose` as well, specifically the >>>>>>>>>>> "ASSIGNMENT" column to include the topic name? >>>>>>>>>>> >>>>>>>>>>> Thanks! >>>>>>>>>>> >>>>>>>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote: >>>>>>>>>>>> Thanks for the updates PoAn! >>>>>>>>>>>> >>>>>>>>>>>> LM6. Just a nit, under the --describe --state --verbose the >>> header >>>>>>>> is >>>>>>>>>> fine >>>>>>>>>>>> but the example is still missing the --state argument. >>>>>>>>>>>> >>>>>>>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, should >>>>> we >>>>>>>>>>>> clearly mention that the KIP adds a new --verbose option? I think >>>>>>>> it's >>>>>>>>>>>> confusing because it's presented as if the option already exists >>>>> and >>>>>>>>>> we're >>>>>>>>>>>> only changing the output (which is the case for >>>>>>>> kafka-consumer-groups >>>>>>>>>> but >>>>>>>>>>>> not for kafka-share-groups) >>>>>>>>>>>> >>>>>>>>>>>> That's all on my side. Thanks! >>>>>>>>>>>> >>>>>>>>>>>> Lianet >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <yangp...@gmail.com> >>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Lianet / Jeff, >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for review and suggestions. >>>>>>>>>>>>> >>>>>>>>>>>>> LM4: At group level, the protocol field is from >>>>>>>>>> ModernGroup#protocolType. >>>>>>>>>>>>> It’s a string. >>>>>>>>>>>>> The value is defined in different places like >>>>>>>>> ShareGroup#PROTOCOL_TYPE >>>>>>>>>> and >>>>>>>>>>>>> ConsumerProtocol#PROTOCOL_TYPE. >>>>>>>>>>>>> In KIP-1043 < >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh >>>>>>>>>>> , >>>>>>>>>>>>> the kafka-groups.sh also shows protocol like “classic” / >>>>>>>> “consumer” / >>>>>>>>>>>>> “share”. >>>>>>>>>>>>> I agree it’s better to use a type like GroupProtocol enum, but I >>>>>>>>> would >>>>>>>>>>>>> like to align the format with kafka-groups.sh. >>>>>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol >>> and >>>>>>>>>> change >>>>>>>>>>>>> return type of ModernGroup#protocolType? >>>>>>>>>>>>> >>>>>>>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in >>> the >>>>>>>>>> title. >>>>>>>>>>>>> You’re correct. I would like to use `—describe —state —verbose` >>>>> to >>>>>>>>> show >>>>>>>>>>>>> group level information. >>>>>>>>>>>>> For `—describe —verbose`, it will show output as same as >>>>> `—describe >>>>>>>>>>>>> —offsets —verbose`, >>>>>>>>>>>>> because `—describe` also shows output as same as `—describe >>>>>>>>> —offsets`. >>>>>>>>>>>>> >>>>>>>>>>>>> JK2: Yes, I only use consumer group in the sample output, but >>> the >>>>>>>>>> classic >>>>>>>>>>>>> group will use the same format as well. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> PoAn >>>>>>>>>>>>> >>>>>>>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <jeffkb...@apache.org> >>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for the KIP. I have some questions: >>>>>>>>>>>>>> >>>>>>>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe >>> --verbose` >>>>>>>>>>>>> proposed changes - doesn't `--describe $group` default to >>>>> printing >>>>>>>>> the >>>>>>>>>>>>> offsets? Perhaps you're referring to the `--state` argument? >>>>> Also, >>>>>>>>>> would >>>>>>>>>>>>> that mean the default `--describe $group --verbose` command >>> would >>>>>>>> not >>>>>>>>>> print >>>>>>>>>>>>> the added field to `--offsets verbose` (leader-epoch) or would >>>>> it? >>>>>>>>>>>>>> >>>>>>>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing >>>>>>>> "ASSIGNMENT" >>>>>>>>>>>>> column under `--members --verbose` does group the topic >>>>> partitions >>>>>>>> by >>>>>>>>>> topic >>>>>>>>>>>>> but does not prefix the grouping with the topic name like you're >>>>>>>>>> proposal: >>>>>>>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format >>>>>>>> for >>>>>>>>>> the >>>>>>>>>>>>> classic group as well? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote: >>>>>>>>>>>>>>> Hello PoAn, just of couple more minor comments: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> LM4. Regarding the new “protocol” field added to >>>>>>>> MemberDescription. >>>>>>>>>>>>> Should >>>>>>>>>>>>>>> we consider reusing the existing GroupProtocol enum instead of >>>>>>>>>> String? >>>>>>>>>>>>>>> (It’s the one we use from the consumer side to refer to the >>>>>>>>> protocol >>>>>>>>>> in >>>>>>>>>>>>>>> use, just missing Share I notice). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> LM5. Regarding the change to the output for >>>>>>>> kafka-consumer-groups, >>>>>>>>>> the >>>>>>>>>>>>>>> command shown does not include the —state option, but the >>>>> output >>>>>>>>>> shows >>>>>>>>>>>>>>> state info (state, #members, epochs). I would guess that we >>>>> want >>>>>>>> to >>>>>>>>>>>>> modify >>>>>>>>>>>>>>> the output only when we describe a group with the —state >>>>> —verbose >>>>>>>>>>>>> option, >>>>>>>>>>>>>>> is my understanding right? If my understanding is right we’re >>>>>>>> just >>>>>>>>>>>>> missing >>>>>>>>>>>>>>> adding the —state in the example, and the KIP does not >>>>> introduce >>>>>>>>> any >>>>>>>>>>>>>>> changes to the —describe —verbose option. (If not, it would >>>>> mean >>>>>>>> a >>>>>>>>>>>>> bigger >>>>>>>>>>>>>>> change to the output of —describe —verbose which I expect is >>>>> not >>>>>>>>> the >>>>>>>>>>>>>>> intention?) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Lianet >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <yangp...@gmail.com >>>> >>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Chia-Ping, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for the review and suggestions. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in >>>>>>>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to >>>>>>>> check >>>>>>>>>>>>> whether >>>>>>>>>>>>>>>> a member in consumer group uses “classic” or “consumer” >>>>>>>> protocol. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> CI2: Yes, a member in share group always uses “share” >>>>> protocol. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> CI3: Add a table to show meaning of “classic”, “consumer”, >>> and >>>>>>>>>> “share” >>>>>>>>>>>>>>>> protocol. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> BTW, the vote thread is in >>>>>>>>>>>>>>>> >>>>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai < >>>>>>>>> chia7...@apache.org> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> hi PoAn >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> CI0: We have to bump the version of >>>>>>>> ConsumerGroupDescribeRequest >>>>>>>>>> as >>>>>>>>>>>>>>>> well, so server can distinguish the new and old behavior. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> CI1: the type of new filed is string, so I guess you plan to >>>>>>>> use >>>>>>>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return >>>>>>>> either >>>>>>>>>>>>>>>> "classic" or "consumer", right? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription` >>>>>>>> too, >>>>>>>>> so >>>>>>>>>>>>> the >>>>>>>>>>>>>>>> new filed protocol in shared group is always "shared", right? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> CI3: Could you consider adding a table to show the value of >>>>> the >>>>>>>>>>>>> protocol >>>>>>>>>>>>>>>> field in each case? Andrew has a beautiful table in KIP-1043 >>>>>>>> that >>>>>>>>>>>>> lists all >>>>>>>>>>>>>>>> possible protocol names. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [0] >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>> https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>> Chia-Ping >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote: >>>>>>>>>>>>>>>>>> Hi David, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks for the reminder. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> DJ4: Add related content to the KIP. >>>>>>>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to >>>>> “0-1”. >>>>>>>>>>>>>>>>>> Add a new field “Protocol” to >>> ConsumerGroupDescribeResponse. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot >>>>>>>>>>>>> <dja...@confluent.io.INVALID> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> DJ4: Could you please also include the required changes to >>>>>>>> the >>>>>>>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface >>> section? >>>>>>>> We >>>>>>>>>>>>>>>> basically >>>>>>>>>>>>>>>>>>> need to bump the version, add the new field, etc. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> David >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang < >>>>>>>> yangp...@gmail.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi David, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription, >>>>>>>>>>>>>>>>>>>> so the command line tool can show protocol information >>>>> when >>>>>>>>>> users >>>>>>>>>>>>>>>> describe >>>>>>>>>>>>>>>>>>>> members. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote >>>>>>>> thread >>>>>>>>>>>>> today. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot >>>>>>>>>>>>>>>> <dja...@confluent.io.INVALID> >>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> DJ3: That seems reasonable to me. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to change >>> the >>>>>>>>>> output >>>>>>>>>>>>> of >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> tool too many times. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>> David >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang < >>>>>>>>> yangp...@gmail.com> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi David / Andrew, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up >>>>>>>>>>>>>>>> kafka-share-groups.sh >>>>>>>>>>>>>>>>>>>>>> implementation. >>>>>>>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think >>>>> that >>>>>>>>>> using >>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>> format >>>>>>>>>>>>>>>>>>>>>> is more clear. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> The new format will be like >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>> <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs. >>>>>>>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same >>> topic >>>>>>>>> name. >>>>>>>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a vote >>>>>>>>> thread >>>>>>>>>>>>> next >>>>>>>>>>>>>>>>>>>> Monday. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield < >>>>>>>>>>>>>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a >>>>> more >>>>>>>>>>>>>>>> appropriate >>>>>>>>>>>>>>>>>>>>>> missing value, but >>>>>>>>>>>>>>>>>>>>>>> you got there first. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include >>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the >>>>>>>>>>>>>>>>>>>>>>> command line. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the >>>>>>>>>>>>>>>> implementation of >>>>>>>>>>>>>>>>>>>>>> the share groups >>>>>>>>>>>>>>>>>>>>>>> part of this if it helps. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>> Andrew >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>>>>>>>>> From: Frank Yang <yangp...@gmail.com> >>>>>>>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48 >>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> >>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend >>>>>>>>> kafka-consumer-groups >>>>>>>>>>>>>>>> command >>>>>>>>>>>>>>>>>>>>>> line tool to support new consumer group >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi David, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would like to >>>>> get >>>>>>>>>> this >>>>>>>>>>>>> in >>>>>>>>>>>>>>>> AK >>>>>>>>>>>>>>>>>>>> 4.0 >>>>>>>>>>>>>>>>>>>>>> as well. I will do my best. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and >>>>>>>>>> TARGET-ASSIGNMENT-EPOCH >>>>>>>>>>>>>>>> before >>>>>>>>>>>>>>>>>>>>>> #MEMBERS. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column value >>>>> “-“. >>>>>>>>>>>>>>>> (reference < >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92 >>>>>>>>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH >>> CURRENT-ASSIGNMENT >>>>>>>>>>>>>>>> TARGET-EPOCH >>>>>>>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT. >>>>>>>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> For assignment value, it follows current output >>>>>>>> (reference >>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>> https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418 >>>>>>>>>>>>>>>>>>>>> ). >>>>>>>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more >>> clear. >>>>>>>>>>>>>>>>>>>>>>> If we would like to use this form, I will update both >>>>>>>>> output >>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has >>>>>>>>>> protocol >>>>>>>>>>>>>>>> type at >>>>>>>>>>>>>>>>>>>>>> group level. >>>>>>>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and >>>>>>>>>>>>>>>> ConsumerGroupDescribeResponseData >>>>>>>>>>>>>>>>>>>>>> don’t have protocol at member level. >>>>>>>>>>>>>>>>>>>>>>> Could we use a followup to add it? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before >>>>>>>> CURRENT-OFFSET. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot >>>>>>>>>>>>>>>> <dja...@confluent.io.INVALID >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor >>>>>>>>> comments/suggestions: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would >>>>>>>>>> suggest >>>>>>>>>>>>>>>> putting >>>>>>>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before >>>>>>>>>> `#MEMBERS`. >>>>>>>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will >>>>>>>> print >>>>>>>>>> out >>>>>>>>>>>>> N/A >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this >>>>> correct? >>>>>>>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members >>>>> --verbose`, >>>>>>>> I >>>>>>>>>>>>> wonder >>>>>>>>>>>>>>>> if we >>>>>>>>>>>>>>>>>>>>>>>> should use the following order and terms: >>>>> CURRENT-EPOCH >>>>>>>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT >>>>>>>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the >>>>>>>>>> GROUP-EPOCH >>>>>>>>>>>>>>>>>>>> because >>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>> is already in the group description. The value `(0)` >>>>>>>> used >>>>>>>>>> for >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will >>>>>>>> print >>>>>>>>>> out >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> list >>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`. >>>>>>>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should >>>>> also >>>>>>>>> add >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> protocol >>>>>>>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is >>>>>>>> possible >>>>>>>>> to >>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>>> `classic` >>>>>>>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` group >>>>>>>>> during >>>>>>>>>> an >>>>>>>>>>>>>>>> online >>>>>>>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer >>>>>>>>> protocol. >>>>>>>>>>>>> Having >>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>> information may be handy for administrators. What do >>>>> you >>>>>>>>>> think? >>>>>>>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets >>>>> --verbose`, >>>>>>>> I >>>>>>>>>> would >>>>>>>>>>>>>>>>>>>> suggest >>>>>>>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0 >>>>> as >>>>>>>> it >>>>>>>>>>>>>>>> changes the >>>>>>>>>>>>>>>>>>>>>>>> output of the command. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>> DJ >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang < >>>>>>>>>> yangp...@gmail.com> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information >>> when >>>>>>>>> users >>>>>>>>>>>>> give >>>>>>>>>>>>>>>>>>>>>> —verbose >>>>>>>>>>>>>>>>>>>>>>>>> option. >>>>>>>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target >>>>>>>>> assignment >>>>>>>>>>>>>>>> epoch. >>>>>>>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above >>> information, >>>>>>>>>> member >>>>>>>>>>>>>>>> epoch, >>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>> target assignment. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with >>> KIP-848 >>>>>>>>>>>>> definition. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I >>>>> use >>>>>>>>>>>>> Optional >>>>>>>>>>>>>>>>>>>>>> fields in >>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription. >>>>>>>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have >>>>>>>>> epoch, >>>>>>>>>> so >>>>>>>>>>>>> I >>>>>>>>>>>>>>>> use >>>>>>>>>>>>>>>>>>>> int >>>>>>>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level >>>>>>>>>> information >>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>> —describe >>>>>>>>>>>>>>>>>>>>>>>>> —verbose. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe >>>>>>>> —offsets >>>>>>>>>>>>>>>> —verbose. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. < >>>>>>>>> liane...@gmail.com >>>>>>>>>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of >>>>>>>> moving >>>>>>>>>> this >>>>>>>>>>>>>>>> into a >>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only >>>>> attempt >>>>>>>> to >>>>>>>>>> make >>>>>>>>>>>>>>>> sense >>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying >>> to >>>>>>>>> view >>>>>>>>>> the >>>>>>>>>>>>>>>> group >>>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense >>>>> to >>>>>>>>> me, >>>>>>>>>>>>> even >>>>>>>>>>>>>>>> more >>>>>>>>>>>>>>>>>>>>>> if we >>>>>>>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both >>>>>>>> issues, >>>>>>>>>> my >>>>>>>>>>>>>>>> take is >>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect >>>>>>>>> member-epoch, >>>>>>>>>>>>>>>> whoever >>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the epochs at >>>>>>>> all. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't >>>>> we >>>>>>>>>> expect >>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>> always >>>>>>>>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side >>>>> for >>>>>>>>> the >>>>>>>>>>>>> group >>>>>>>>>>>>>>>>>>>> ones, >>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>> client side for the member) >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in >>>>> the >>>>>>>>>>>>> proposed >>>>>>>>>>>>>>>>>>>>>> changes? >>>>>>>>>>>>>>>>>>>>>>>>> ( >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state >>>>>>>>>>>>>>>>>>>>>>>>>> ). >>>>>>>>>>>>>>>>>>>>>>>>>> I get that the output would change in that example, >>>>>>>> but >>>>>>>>>> it’s >>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>> because >>>>>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's >>> because >>>>>>>> of >>>>>>>>>> the >>>>>>>>>>>>>>>> change >>>>>>>>>>>>>>>>>>>>>>>>> proposed >>>>>>>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), and >>>>> the >>>>>>>>>>>>> changes >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems >>>>>>>> confusing, >>>>>>>>>> got >>>>>>>>>>>>> me >>>>>>>>>>>>>>>>>>>>>> looking >>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed >>>>> unrelated). >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are >>>>>>>>>> conceptually >>>>>>>>>>>>>>>> at the >>>>>>>>>>>>>>>>>>>>>>>>> group >>>>>>>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member >>> level. >>>>>>>> So >>>>>>>>>>>>> wonder >>>>>>>>>>>>>>>> if we >>>>>>>>>>>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose >>>>> option) >>>>>>>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target >>>>>>>>>> assignment >>>>>>>>>>>>>>>> epoch >>>>>>>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member >>>>>>>>> epoch >>>>>>>>>>>>>>>> (along >>>>>>>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch) >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Lianet >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield < >>>>>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn, >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are >>>>> most >>>>>>>>>>>>> useful to >>>>>>>>>>>>>>>>>>>>>> people >>>>>>>>>>>>>>>>>>>>>>>>>>> trying to understand >>>>>>>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail. >>>>> The >>>>>>>>>>>>>>>> information >>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>> really not understandable >>>>>>>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of >>>>>>>>>>>>> KIP-848/932. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh >>>>> that >>>>>>>>> you >>>>>>>>>>>>> add a >>>>>>>>>>>>>>>>>>>>>>>>> --members >>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option >>>>>>>>>>>>>>>>>>>>>>>>>>> and only include the new information in the output >>>>>>>> for >>>>>>>>>> that >>>>>>>>>>>>>>>> option, >>>>>>>>>>>>>>>>>>>>>>>>> rather >>>>>>>>>>>>>>>>>>>>>>>>>>> than changing the >>>>>>>>>>>>>>>>>>>>>>>>>>> non-verbose --members output. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for >>>>>>>>>>>>> kafka-consumer-groups.sh >>>>>>>>>>>>>>>>>>>>>> --members >>>>>>>>>>>>>>>>>>>>>>>>>>> and only add the >>>>>>>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH >>>>>>>>>> instead of >>>>>>>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH. >>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the >>>>>>>>>> terminology of >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> KIPs. >>>>>>>>>>>>>>>>>>>>>>>>> For >>>>>>>>>>>>>>>>>>>>>>>>>>> one thing, >>>>>>>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client such >>>>> as >>>>>>>>>>>>>>>>>>>>>> MemberDescription. >>>>>>>>>>>>>>>>>>>>>>>>>>> It's not a >>>>>>>>>>>>>>>>>>>>>>>>>>> new term. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>> Andrew >>>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pay...@apache.org> >>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55 >>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> >>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend >>>>>>>>>>>>> kafka-consumer-groups >>>>>>>>>>>>>>>>>>>> command >>>>>>>>>>>>>>>>>>>>>>>>> line >>>>>>>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lucas, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review! >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for >>>>>>>> kafka-share-groups.sh >>>>>>>>> to >>>>>>>>>>>>> the >>>>>>>>>>>>>>>> KIP. >>>>>>>>>>>>>>>>>>>>>> Could >>>>>>>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use >>>>>>>>>> MEMBER-EPOCH >>>>>>>>>>>>>>>> here, >>>>>>>>>>>>>>>>>>>>>> users >>>>>>>>>>>>>>>>>>>>>>>>> may >>>>>>>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and >>>>>>>> MEMBER. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Frank, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP! >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for >>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, >>>>> etc. ? >>>>>>>>>> Even >>>>>>>>>>>>> if >>>>>>>>>>>>>>>> we do >>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group >>>>>>>>> implementation >>>>>>>>>>>>> may >>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in >>> the >>>>>>>>> KIP. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? >>>>>>>> That >>>>>>>>>> would >>>>>>>>>>>>>>>> seem >>>>>>>>>>>>>>>>>>>>>> more >>>>>>>>>>>>>>>>>>>>>>>>>>>> consistent. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>>>>>>>>>>>> Lucas >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang < >>>>>>>>>>>>>>>> yangp...@gmail.com> >>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of >>>>>>>> KIP-1099. >>>>>>>>>> This >>>>>>>>>>>>> KIP >>>>>>>>>>>>>>>>>>>>>> enhances >>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state >>>>> which >>>>>>>> is >>>>>>>>>>>>>>>> introduced >>>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-848. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command >>>>> line >>>>>>>>>> tool >>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>> support >>>>>>>>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software >>>>>>>>>> Foundation >>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico] >>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>> >>> >>>
