hi DJ > How about we use another Jira to add “SHARE” to GroupProtocol and change return type of ModernGroup#protocolType?
This is not my comment ... sorry for the incorrect quote :( > A member in a classic group should report "classic". I completely agree. However, in this case, ClassicGroupDescription#protocol returns 'consumer', while ClassicGroupDescription#members[x].protocol returns 'classic'. This inconsistency feels a bit strange to me. Best, Chia-Ping David Jacot <dja...@confluent.io.invalid> 於 2024年11月22日 週五 下午6:51寫道： > Hi Chia, > > > > LM4: I agree it’s better to use a type like GroupProtocol enum, but I > would like to align the format with kafka-groups.sh. > How about we use another Jira to add “SHARE” to GroupProtocol and change > return type of ModernGroup#protocolType? > > My understanding is that `GroupProtocol` is used to drive the config of the > `Consumer`. Hence adding `Share` seems wrong to me. Or are you referring to > another GroupProtocol enum? Perhaps GroupType enum? > > > In fact, I'm wondering if we should even add 'protocol' to > MemberDescription, > since the field has different values between classic and consumer groups—it > shows 'consumer' in classic groups, whereas it shows 'classic' in consumer > groups. Yes, it seems like a typo, but it's true. This inconsistency feels > confusing to me. For example, users call Admin#describeConsumerGroups and > see that a member has the 'consumer' protocol when it's in a classic group. > Then, if the group is upgraded to a consumer group, calling > Admin#describeConsumerGroups again shows the protocol as 'classic' for the > same member. > > I don't fully follow your comment so please excuse me if I say something > wrong. > > I think that it should work as follows: > * A member in a classic group should report "classic". > * A member using the consumer protocol in a consumer group should report > "consumer". > * A member using the classic protocol in a consumer group should report > "classic". > * A member in a share group should report "share". > > > If all we want to do is distinguish classic members in the > ConsumerGroupDescription, then we shouldn't modify the common > MemberDescription structure. Instead, we could add a collection to > ConsumerGroupDescription to trace members using the classic protocol, which > would allow us to enrich the output of command-line tools. > > Personally, I believe that using different collections per protocol types > is not necessary. A field in the member is more than enough. > > Best, > DJ > > On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai <chia7...@gmail.com> > wrote: > > > hi PoAn > > > > > LM4: I agree it’s better to use a type like GroupProtocol enum, but I > > would like to align the format with kafka-groups.sh. > > How about we use another Jira to add “SHARE” to GroupProtocol and change > > return type of ModernGroup#protocolType? > > > > MemberDescription is also used by ClassicGroupDescription, and in classic > > groups, the protocol can be an arbitrary string, so we can't use an enum. > > > > In fact, I'm wondering if we should even add 'protocol' to > > MemberDescription, > > since the field has different values between classic and consumer > groups—it > > shows 'consumer' in classic groups, whereas it shows 'classic' in > consumer > > groups. Yes, it seems like a typo, but it's true. This inconsistency > feels > > confusing to me. For example, users call Admin#describeConsumerGroups and > > see that a member has the 'consumer' protocol when it's in a classic > group. > > Then, if the group is upgraded to a consumer group, calling > > Admin#describeConsumerGroups again shows the protocol as 'classic' for > the > > same member. > > > > If all we want to do is distinguish classic members in the > > ConsumerGroupDescription, then we shouldn't modify the common > > MemberDescription structure. Instead, we could add a collection to > > ConsumerGroupDescription to trace members using the classic protocol, > which > > would allow us to enrich the output of command-line tools. > > Best, > > Chia-Ping > > > > PoAn Yang <yangp...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道： > > > > > Hi Lianet / Jeff, > > > > > > Thanks for the review. > > > > > > LM5: The kafka-share-groups.sh also uses “—describe” to show offsets > > > information. > > > Change to use “—describe —state —verbose” to show group level > information > > > in kafka-share-groups.sh. > > > > > > LM6: Update the example. > > > > > > LM7: Add a description to mention “—verbose” is a new option in > > > ShareGroupCommandOptions. > > > > > > JK2: Yes, add the statement about format change for ASSIGNMENT value. > > > > > > Thanks, > > > PoAn > > > > > > > On Nov 22, 2024, at 7:35 AM, Jeff Kim <jeffkb...@apache.org> wrote: > > > > > > > > Hi PoAn, > > > > > > > > JK2: Can we state in the KIP that we will reformat the existing > > > > classic group's `--members --verbose` as well, specifically the > > > > "ASSIGNMENT" column to include the topic name? > > > > > > > > Thanks! > > > > > > > > On 2024/11/21 14:33:37 "Lianet M." wrote: > > > >> Thanks for the updates PoAn! > > > >> > > > >> LM6. Just a nit, under the --describe --state --verbose the header > is > > > fine > > > >> but the example is still missing the --state argument. > > > >> > > > >> LM7. Under the Proposed changes for kafka-share-groups.sh, should we > > > >> clearly mention that the KIP adds a new --verbose option? I think > it's > > > >> confusing because it's presented as if the option already exists and > > > we're > > > >> only changing the output (which is the case for > kafka-consumer-groups > > > but > > > >> not for kafka-share-groups) > > > >> > > > >> That's all on my side. Thanks! > > > >> > > > >> Lianet > > > >> > > > >> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang <yangp...@gmail.com> > > wrote: > > > >> > > > >>> Hi Lianet / Jeff, > > > >>> > > > >>> Thanks for review and suggestions. > > > >>> > > > >>> LM4: At group level, the protocol field is from > > > ModernGroup#protocolType. > > > >>> It’s a string. > > > >>> The value is defined in different places like > > ShareGroup#PROTOCOL_TYPE > > > and > > > >>> ConsumerProtocol#PROTOCOL_TYPE. > > > >>> In KIP-1043 < > > > >>> > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh > > > >, > > > >>> the kafka-groups.sh also shows protocol like “classic” / > “consumer” / > > > >>> “share”. > > > >>> I agree it’s better to use a type like GroupProtocol enum, but I > > would > > > >>> like to align the format with kafka-groups.sh. > > > >>> How about we use another Jira to add “SHARE” to GroupProtocol and > > > change > > > >>> return type of ModernGroup#protocolType? > > > >>> > > > >>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state in the > > > title. > > > >>> You’re correct. I would like to use `—describe —state —verbose` to > > show > > > >>> group level information. > > > >>> For `—describe —verbose`, it will show output as same as `—describe > > > >>> —offsets —verbose`, > > > >>> because `—describe` also shows output as same as `—describe > > —offsets`. > > > >>> > > > >>> JK2: Yes, I only use consumer group in the sample output, but the > > > classic > > > >>> group will use the same format as well. > > > >>> > > > >>> Thanks, > > > >>> PoAn > > > >>> > > > >>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim <jeffkb...@apache.org> > wrote: > > > >>>> > > > >>>> Hi PoAn, > > > >>>> > > > >>>> Thanks for the KIP. I have some questions: > > > >>>> > > > >>>> JK1: (Realized this is similar to LM5) On `--describe --verbose` > > > >>> proposed changes - doesn't `--describe $group` default to printing > > the > > > >>> offsets? Perhaps you're referring to the `--state` argument? Also, > > > would > > > >>> that mean the default `--describe $group --verbose` command would > not > > > print > > > >>> the added field to `--offsets verbose` (leader-epoch) or would it? > > > >>>> > > > >>>> JK2: Looking at ConsumerGroupCommand.java, the existing > "ASSIGNMENT" > > > >>> column under `--members --verbose` does group the topic partitions > by > > > topic > > > >>> but does not prefix the grouping with the topic name like you're > > > proposal: > > > >>> "my_topic:0,1;new_topic:0,1". Should we do apply the same format > for > > > the > > > >>> classic group as well? > > > >>>> > > > >>>> > > > >>>> > > > >>>> On 2024/11/20 21:32:41 "Lianet M." wrote: > > > >>>>> Hello PoAn, just of couple more minor comments: > > > >>>>> > > > >>>>> LM4. Regarding the new “protocol” field added to > MemberDescription. > > > >>> Should > > > >>>>> we consider reusing the existing GroupProtocol enum instead of > > > String? > > > >>>>> (It’s the one we use from the consumer side to refer to the > > protocol > > > in > > > >>>>> use, just missing Share I notice). > > > >>>>> > > > >>>>> LM5. Regarding the change to the output for > kafka-consumer-groups, > > > the > > > >>>>> command shown does not include the —state option, but the output > > > shows > > > >>>>> state info (state, #members, epochs). I would guess that we want > to > > > >>> modify > > > >>>>> the output only when we describe a group with the —state —verbose > > > >>> option, > > > >>>>> is my understanding right? If my understanding is right we’re > just > > > >>> missing > > > >>>>> adding the —state in the example, and the KIP does not introduce > > any > > > >>>>> changes to the —describe —verbose option. (If not, it would mean > a > > > >>> bigger > > > >>>>> change to the output of —describe —verbose which I expect is not > > the > > > >>>>> intention?) > > > >>>>> > > > >>>>> Thanks! > > > >>>>> > > > >>>>> Lianet > > > >>>>> > > > >>>>> > > > >>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang <yangp...@gmail.com> > > > wrote: > > > >>>>> > > > >>>>>> Hi Chia-Ping, > > > >>>>>> > > > >>>>>> Thanks for the review and suggestions. > > > >>>>>> > > > >>>>>> CI0: Thanks for the reminder. Update validVersions in > > > >>>>>> ConsumerGroupDescribeRequest to 0-1. > > > >>>>>> > > > >>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` to > check > > > >>> whether > > > >>>>>> a member in consumer group uses “classic” or “consumer” > protocol. > > > >>>>>> > > > >>>>>> CI2: Yes, a member in share group always uses “share” protocol. > > > >>>>>> > > > >>>>>> CI3: Add a table to show meaning of “classic”, “consumer”, and > > > “share” > > > >>>>>> protocol. > > > >>>>>> > > > >>>>>> BTW, the vote thread is in > > > >>>>>> > https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6. > > > >>>>>> > > > >>>>>> Thanks, > > > >>>>>> PoAn > > > >>>>>> > > > >>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai < > > chia7...@apache.org> > > > >>>>>> wrote: > > > >>>>>>> > > > >>>>>>> hi PoAn > > > >>>>>>> > > > >>>>>>> CI0: We have to bump the version of > ConsumerGroupDescribeRequest > > > as > > > >>>>>> well, so server can distinguish the new and old behavior. > > > >>>>>>> > > > >>>>>>> CI1: the type of new filed is string, so I guess you plan to > use > > > >>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to return > either > > > >>>>>> "classic" or "consumer", right? > > > >>>>>>> > > > >>>>>>> CI2: `MemberDescription` is used by `ShareGroupDescription` > too, > > so > > > >>> the > > > >>>>>> new filed protocol in shared group is always "shared", right? > > > >>>>>>> > > > >>>>>>> CI3: Could you consider adding a table to show the value of the > > > >>> protocol > > > >>>>>> field in each case? Andrew has a beautiful table in KIP-1043 > that > > > >>> lists all > > > >>>>>> possible protocol names. > > > >>>>>>> > > > >>>>>>> [0] > > > >>>>>> > > > >>> > > > > > > https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454 > > > >>>>>> > > > >>>>>>> > > > >>>>>>> Best, > > > >>>>>>> Chia-Ping > > > >>>>>>> > > > >>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote: > > > >>>>>>>> Hi David, > > > >>>>>>>> > > > >>>>>>>> Thanks for the reminder. > > > >>>>>>>> > > > >>>>>>>> DJ4: Add related content to the KIP. > > > >>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to “0-1”. > > > >>>>>>>> Add a new field “Protocol” to ConsumerGroupDescribeResponse. > > > >>>>>>>> > > > >>>>>>>> Thanks, > > > >>>>>>>> PoAn > > > >>>>>>>> > > > >>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot > > > >>> <dja...@confluent.io.INVALID> > > > >>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>> Hi PoAn, > > > >>>>>>>>> > > > >>>>>>>>> DJ4: Could you please also include the required changes to > the > > > >>>>>>>>> ConsumerGroupDescribed API in the public interface section? > We > > > >>>>>> basically > > > >>>>>>>>> need to bump the version, add the new field, etc. > > > >>>>>>>>> > > > >>>>>>>>> Thanks, > > > >>>>>>>>> David > > > >>>>>>>>> > > > >>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang < > yangp...@gmail.com> > > > >>> wrote: > > > >>>>>>>>> > > > >>>>>>>>>> Hi David, > > > >>>>>>>>>> > > > >>>>>>>>>> DJ4: Add a new field protocol to MemberDescription, > > > >>>>>>>>>> so the command line tool can show protocol information when > > > users > > > >>>>>> describe > > > >>>>>>>>>> members. > > > >>>>>>>>>> > > > >>>>>>>>>> If there is no further suggestion, I will start a vote > thread > > > >>> today. > > > >>>>>>>>>> > > > >>>>>>>>>> Thanks, > > > >>>>>>>>>> PoAn > > > >>>>>>>>>> > > > >>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot > > > >>>>>> <dja...@confluent.io.INVALID> > > > >>>>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>> Hi, > > > >>>>>>>>>>> > > > >>>>>>>>>>> DJ2: Using "-" sounds good to me. > > > >>>>>>>>>>> > > > >>>>>>>>>>> DJ3: That seems reasonable to me. > > > >>>>>>>>>>> > > > >>>>>>>>>>> DJ4: Why not add it right now? I don't want to change the > > > output > > > >>> of > > > >>>>>> the > > > >>>>>>>>>>> tool too many times. > > > >>>>>>>>>>> > > > >>>>>>>>>>> Best, > > > >>>>>>>>>>> David > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang < > > yangp...@gmail.com> > > > >>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>>> Hi David / Andrew, > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up > > > >>>>>> kafka-share-groups.sh > > > >>>>>>>>>>>> implementation. > > > >>>>>>>>>>>> I will handle kafka-consumer-groups.sh. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we think that > > > using > > > >>> new > > > >>>>>>>>>> format > > > >>>>>>>>>>>> is more clear. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> The new format will be like > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>> > > > >>> > > > > > > <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Using colon(:) to concat topic name and partition IDs. > > > >>>>>>>>>>>> Using comma(,) to concat partition IDs within same topic > > name. > > > >>>>>>>>>>>> Using semicolon(;) to concat topic strings. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> If there is no further suggestion, I will start a vote > > thread > > > >>> next > > > >>>>>>>>>> Monday. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>> PoAn > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield < > > > >>>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Hi PoAn, > > > >>>>>>>>>>>>> DJ2: I was just going to comment that "-" would be a more > > > >>>>>> appropriate > > > >>>>>>>>>>>> missing value, but > > > >>>>>>>>>>>>> you got there first. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh include > > > >>>>>>>>>>>> kafka-consumer-groups.sh in the > > > >>>>>>>>>>>>> command line. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up the > > > >>>>>> implementation of > > > >>>>>>>>>>>> the share groups > > > >>>>>>>>>>>>> part of this if it helps. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>> Andrew > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> ________________________________________ > > > >>>>>>>>>>>>> From: Frank Yang <yangp...@gmail.com> > > > >>>>>>>>>>>>> Sent: 14 November 2024 10:48 > > > >>>>>>>>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> > > > >>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend > > kafka-consumer-groups > > > >>>>>> command > > > >>>>>>>>>>>> line tool to support new consumer group > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Hi David, > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Thanks for the review and suggestion! I would like to get > > > this > > > >>> in > > > >>>>>> AK > > > >>>>>>>>>> 4.0 > > > >>>>>>>>>>>> as well. I will do my best. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and > > > TARGET-ASSIGNMENT-EPOCH > > > >>>>>> before > > > >>>>>>>>>>>> #MEMBERS. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> DJ2: I prefer to follow current missing column value “-“. > > > >>>>>> (reference < > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>> > > > >>> > > > > > > https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92 > > > >>>>>>>>>>>>> ) > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH CURRENT-ASSIGNMENT > > > >>>>>> TARGET-EPOCH > > > >>>>>>>>>>>> TARGET-ASSIGNMENT. > > > >>>>>>>>>>>>> Remove GROUP-EPOCH. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> For assignment value, it follows current output > (reference > > < > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>> > > > >>> > > > > > > https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418 > > > >>>>>>>>>>> ). > > > >>>>>>>>>>>> I think the form of `topicid-partitionid` is more clear. > > > >>>>>>>>>>>>> If we would like to use this form, I will update both > > output > > > in > > > >>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData only has > > > protocol > > > >>>>>> type at > > > >>>>>>>>>>>> group level. > > > >>>>>>>>>>>>> Both DescribeGroupsResponseData and > > > >>>>>> ConsumerGroupDescribeResponseData > > > >>>>>>>>>>>> don’t have protocol at member level. > > > >>>>>>>>>>>>> Could we use a followup to add it? > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before > CURRENT-OFFSET. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>> PoAn > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot > > > >>>>>> <dja...@confluent.io.INVALID > > > >>>>>>>>>>> > > > >>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Hi PoAn, > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Thanks for the KIP! I have a few minor > > comments/suggestions: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I would > > > suggest > > > >>>>>> putting > > > >>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` before > > > `#MEMBERS`. > > > >>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we will > print > > > out > > > >>> N/A > > > >>>>>> for > > > >>>>>>>>>>>> the > > > >>>>>>>>>>>>>> fields not supported by classic groups. Is this correct? > > > >>>>>>>>>>>>>> DJ3: In the output of `--describe --members --verbose`, > I > > > >>> wonder > > > >>>>>> if we > > > >>>>>>>>>>>>>> should use the following order and terms: CURRENT-EPOCH > > > >>>>>>>>>>>> CURRENT-ASSIGNMENT > > > >>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove the > > > GROUP-EPOCH > > > >>>>>>>>>> because > > > >>>>>>>>>>>> it > > > >>>>>>>>>>>>>> is already in the group description. The value `(0)` > used > > > for > > > >>> the > > > >>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we will > print > > > out > > > >>> the > > > >>>>>>>>>> list > > > >>>>>>>>>>>> of > > > >>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`. > > > >>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we should also > > add > > > >>> the > > > >>>>>>>>>>>> protocol > > > >>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is > possible > > to > > > >>> have > > > >>>>>>>>>>>> `classic` > > > >>>>>>>>>>>>>> members and `consumer` members in a `consumer` group > > during > > > an > > > >>>>>> online > > > >>>>>>>>>>>>>> upgrade from the classic protocol to the consumer > > protocol. > > > >>> Having > > > >>>>>>>>>> this > > > >>>>>>>>>>>>>> information may be handy for administrators. What do you > > > think? > > > >>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets --verbose`, > I > > > would > > > >>>>>>>>>> suggest > > > >>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to `CURRENT-OFFSET`. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> It would be great if we could get this one in AK 4.0 as > it > > > >>>>>> changes the > > > >>>>>>>>>>>>>> output of the command. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>>> DJ > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang < > > > yangp...@gmail.com> > > > >>>>>> wrote: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet, > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Thanks for all your review and suggestions. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information when > > users > > > >>> give > > > >>>>>>>>>>>> —verbose > > > >>>>>>>>>>>>>>> option. > > > >>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target > > assignment > > > >>>>>> epoch. > > > >>>>>>>>>>>>>>> —describe —members —verbose: shows above information, > > > member > > > >>>>>> epoch, > > > >>>>>>>>>> and > > > >>>>>>>>>>>>>>> target assignment. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with KIP-848 > > > >>> definition. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, so I use > > > >>> Optional > > > >>>>>>>>>>>> fields in > > > >>>>>>>>>>>>>>> ConsumerGroupDescription. > > > >>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must have > > epoch, > > > so > > > >>> I > > > >>>>>> use > > > >>>>>>>>>> int > > > >>>>>>>>>>>>>>> fields in ShareGroupDescription. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> LM3: Remove —state change. We can get group level > > > information > > > >>> by > > > >>>>>>>>>>>> —describe > > > >>>>>>>>>>>>>>> —verbose. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe > —offsets > > > >>>>>> —verbose. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Thanks. > > > >>>>>>>>>>>>>>> PoAn > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. < > > liane...@gmail.com > > > > > > > >>>>>> wrote: > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few comments: > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion of > moving > > > this > > > >>>>>> into a > > > >>>>>>>>>>>>>>>> --verbose option. I expect someone would only attempt > to > > > make > > > >>>>>> sense > > > >>>>>>>>>> of > > > >>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>> epochs while debugging an issue, not while trying to > > view > > > the > > > >>>>>> group > > > >>>>>>>>>> or > > > >>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more sense to > > me, > > > >>> even > > > >>>>>> more > > > >>>>>>>>>>>> if we > > > >>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to both > issues, > > > my > > > >>>>>> take is > > > >>>>>>>>>>>> that > > > >>>>>>>>>>>>>>>> whoever is close to the protocol will expect > > member-epoch, > > > >>>>>> whoever > > > >>>>>>>>>> is > > > >>>>>>>>>>>> not > > > >>>>>>>>>>>>>>>> will probably won't even need to see the epochs at > all. > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional (don't we > > > expect > > > >>> to > > > >>>>>>>>>> always > > > >>>>>>>>>>>>>>> have > > > >>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker side for > > the > > > >>> group > > > >>>>>>>>>> ones, > > > >>>>>>>>>>>>>>> and > > > >>>>>>>>>>>>>>>> client side for the member) > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state option in the > > > >>> proposed > > > >>>>>>>>>>>> changes? > > > >>>>>>>>>>>>>>> ( > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>> > > > >>> > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state > > > >>>>>>>>>>>>>>>> ). > > > >>>>>>>>>>>>>>>> I get that the output would change in that example, > but > > > it’s > > > >>> not > > > >>>>>>>>>>>> because > > > >>>>>>>>>>>>>>> of > > > >>>>>>>>>>>>>>>> any change to the –state option itself. It's because > of > > > the > > > >>>>>> change > > > >>>>>>>>>>>>>>> proposed > > > >>>>>>>>>>>>>>>> to the –described (required with the --state), and the > > > >>> changes > > > >>>>>> to > > > >>>>>>>>>>>>>>>> --describe are already explained above (seems > confusing, > > > got > > > >>> me > > > >>>>>>>>>>>> looking > > > >>>>>>>>>>>>>>> for > > > >>>>>>>>>>>>>>>> a change to the state filter which seemed unrelated). > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch are > > > conceptually > > > >>>>>> at the > > > >>>>>>>>>>>>>>> group > > > >>>>>>>>>>>>>>>> level. vs member epoch that lands at a member level. > So > > > >>> wonder > > > >>>>>> if we > > > >>>>>>>>>>>>>>> should > > > >>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose option) > > > >>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and target > > > assignment > > > >>>>>> epoch > > > >>>>>>>>>>>>>>>> –describe –members –verbose => shows member > > epoch > > > >>>>>> (along > > > >>>>>>>>>>>>>>>> with group epoch and target assignment epoch) > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Thanks! > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Lianet > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew Schofield < > > > >>>>>>>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote: > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Hi PoAn, > > > >>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions are most > > > >>> useful to > > > >>>>>>>>>>>> people > > > >>>>>>>>>>>>>>>>> trying to understand > > > >>>>>>>>>>>>>>>>> the progress of rebalancing in quite some detail. The > > > >>>>>> information > > > >>>>>>>>>> is > > > >>>>>>>>>>>>>>>>> really not understandable > > > >>>>>>>>>>>>>>>>> for most users who do not have deep knowledge of > > > >>> KIP-848/932. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> As a result, I suggest for kafka-share-groups.sh that > > you > > > >>> add a > > > >>>>>>>>>>>>>>> --members > > > >>>>>>>>>>>>>>>>> --verbose option > > > >>>>>>>>>>>>>>>>> and only include the new information in the output > for > > > that > > > >>>>>> option, > > > >>>>>>>>>>>>>>> rather > > > >>>>>>>>>>>>>>>>> than changing the > > > >>>>>>>>>>>>>>>>> non-verbose --members output. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> I also make a similar suggestion for > > > >>> kafka-consumer-groups.sh > > > >>>>>>>>>>>> --members > > > >>>>>>>>>>>>>>>>> and only add the > > > >>>>>>>>>>>>>>>>> new information for the --verbose output. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use MEMBER-EPOCH > > > instead of > > > >>>>>>>>>>>>>>>>> CONSUMER-EPOCH. > > > >>>>>>>>>>>>>>>>> I think it's more confusing not following the > > > terminology of > > > >>>>>> the > > > >>>>>>>>>>>> KIPs. > > > >>>>>>>>>>>>>>> For > > > >>>>>>>>>>>>>>>>> one thing, > > > >>>>>>>>>>>>>>>>> we already have "member" in the admin client such as > > > >>>>>>>>>>>> MemberDescription. > > > >>>>>>>>>>>>>>>>> It's not a > > > >>>>>>>>>>>>>>>>> new term. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>>>>>> Andrew > > > >>>>>>>>>>>>>>>>> ________________________________________ > > > >>>>>>>>>>>>>>>>> From: PoAn Yang <pay...@apache.org> > > > >>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55 > > > >>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> > > > >>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend > > > >>> kafka-consumer-groups > > > >>>>>>>>>> command > > > >>>>>>>>>>>>>>> line > > > >>>>>>>>>>>>>>>>> tool to support new consumer group > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Hi Lucas, > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Thanks for the review! > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> 1) Yes, I add related change for > kafka-share-groups.sh > > to > > > >>> the > > > >>>>>> KIP. > > > >>>>>>>>>>>> Could > > > >>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use > > > MEMBER-EPOCH > > > >>>>>> here, > > > >>>>>>>>>>>> users > > > >>>>>>>>>>>>>>> may > > > >>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and > MEMBER. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>>>>>> PoAn > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote: > > > >>>>>>>>>>>>>>>>>> Hi Frank, > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> thanks for the KIP! > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for > > > >>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, etc. ? > > > Even > > > >>> if > > > >>>>>> we do > > > >>>>>>>>>>>> not > > > >>>>>>>>>>>>>>>>>> implement it right now if the share group > > implementation > > > >>> may > > > >>>>>> still > > > >>>>>>>>>>>> be > > > >>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it in the > > KIP. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not MEMBER-EPOCH? > That > > > would > > > >>>>>> seem > > > >>>>>>>>>>>> more > > > >>>>>>>>>>>>>>>>>> consistent. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Cheers, > > > >>>>>>>>>>>>>>>>>> Lucas > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang < > > > >>>>>> yangp...@gmail.com> > > > >>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Hi all, > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of > KIP-1099. > > > This > > > >>> KIP > > > >>>>>>>>>>>> enhances > > > >>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state which > is > > > >>>>>> introduced > > > >>>>>>>>>> by > > > >>>>>>>>>>>>>>>>> KIP-848. > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups command line > > > tool > > > >>> to > > > >>>>>>>>>> support > > > >>>>>>>>>>>>>>> new > > > >>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache Software > > > Foundation > > > >>>>>>>>>>>>>>>>>>> < > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>> > > > >>> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> cwiki.apache.org > > > >>>>>>>>>>>>>>>>>>> < > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>> > > > >>> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> [image: favicon.ico] > > > >>>>>>>>>>>>>>>>>>> < > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>> > > > >>> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> < > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>> > > > >>> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>>> Thank you, > > > >>>>>>>>>>>>>>>>>>> PoAn > > > >>>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>> > > > >>> > > > >> > > > > > > > > >
