Thanks, LGTM. On Mon, Nov 25, 2024 at 4:26 PM PoAn Yang <yangp...@gmail.com> wrote:
> Hi David, > > Thanks for the feedback. > > Yes, I agree that the “UPGRADED” column name is better. I updated the KIP. > > Best, > PoAn > > > On Nov 25, 2024, at 11:11 PM, PoAn Yang <yangp...@gmail.com> wrote: > > > > Hi Andrew, > > > > Thanks for the feedback. > > > > AS4: Change TARGET-ASSIGNMENT-EPOCH to ASSIGNMENT-EPOCH for “—describe > —state —verbose" in kafka-share-groups.sh. > > > > AS5: Remove TARGET-EPOCH and TARGET-ASSIGNMENT. > > Change CURRENT-EPOCH to MEMBER-EPOCH and CURRENT-ASSIGNMENT to > ASSIGNMENT for “—describe —member —verbose” in kafka-share-groups.sh. > > > > Thanks, > > PoAn > > > >> On Nov 25, 2024, at 10:48 PM, Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > >> > >> Hi PoAn, > >> I've taken another pass over the KIP and have some more comments around > the share > >> group support. > >> > >> AS4: Share groups do not have the concept of a target assignment. As a > result, > >> the verbose state TARGET-ASSIGNMENT-EPOCH is a bit misleading from my > >> point of view. ASSIGNMENT-EPOCH is better. > >> > >> AS5: Similarly for the verbose members output, the TARGET-EPOCH and > >> TARGET-ASSIGNMENT are not really appropriate. Personally, I would prefer > >> just MEMBER-EPOCH and ASSIGNMENT, and remove the "target" fields. > >> > >> Thanks, > >> Andrew > >> > >> ________________________________________ > >> From: PoAn Yang <yangp...@gmail.com> > >> Sent: 24 November 2024 15:08 > >> To: dev@kafka.apache.org <dev@kafka.apache.org> > >> Subject: Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command > line tool to support new consumer group > >> > >> Hi Chia-Ping, > >> > >> Thanks for the suggestion. > >> > >> The “isClassic” field in ConsumerGroupDescribeResponse is a boolean > value, so we can’t use it as the protocol string. > >> I change the “PROTOCOL” column to “IS-CLASSIC” in > kafka-consumer-groups.sh. > >> It returns “true” if a member is in a classic group or it’s a classic > member in a consumer group. > >> In other cases, it returns “false”. > >> > >> I have a draft implementation in > https://github.com/apache/kafka/pull/17872. > >> > >> Thanks, > >> PoAn > >> > >>> On Nov 23, 2024, at 1:05 PM, Chia-Ping Tsai <chia7...@gmail.com> > wrote: > >>> > >>> hi PoAn > >>> > >>> `isClassic` is good to me :) > >>> > >>> Best, > >>> Chia-Ping > >>> > >>> PoAn Yang <yangp...@gmail.com> 於 2024年11月23日 週六 下午12:35寫道： > >>> > >>>> Hi Chia-Ping / David, > >>>> > >>>> Thanks for the great suggestion. > >>>> > >>>> How about using “isClassic”? > >>>> The term like “isLegacy” may be equivocal in the future. > >>>> For example, if we have a newer version of consumer / share consumer, > the > >>>> meaning will be different for different cases. > >>>> If we use “isClassic”, for MemberDescription in > ClassicGroupDescription, > >>>> the value can be always “true”. > >>>> In ConsumerGroupDescription, it’s “true” when it’s a classic member. > >>>> In ShareGroupDescription, it’s always “false”. > >>>> > >>>> Thanks, > >>>> PoAn > >>>> > >>>>> On Nov 23, 2024, at 1:02 AM, Chia-Ping Tsai <chia7...@gmail.com> > wrote: > >>>>> > >>>>> hi DJ > >>>>> > >>>>>> My goal was to have a way to tell the user that his (new) consumer > group > >>>>> may have non-upgraded members yet. I wonder if we could use a > boolean to > >>>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column > which > >>>>> is only displayed when they are non upgraded members. What do you > think? > >>>>> > >>>>> Yes, this is a good approach, but the downside is that the new > isLegacy > >>>>> field in MemberDescription would be irrelevant for classic and shared > >>>>> groups. This would require thorough documentation to clarify its > purpose. > >>>>> > >>>>> Additionally, we could use a boolean instead of a string for the new > >>>> field > >>>>> in the RPC to reduce the request size. > >>>>> Best, > >>>>> Chia-Ping > >>>>> > >>>>> > >>>>> > >>>>> David Jacot <dja...@confluent.io.invalid> 於 2024年11月23日 週六 > 上午12:41寫道： > >>>>> > >>>>>> Hi both, > >>>>>> > >>>>>> Ah, I understand what you mean now. You're saying that we have two > >>>> protocol > >>>>>> fields. One at the group level and one at the member level (the new > >>>> one). > >>>>>> The two fields are actually two different things. The former is the > >>>> name of > >>>>>> the embedded protocol used in the classic protocol whereas the > latter is > >>>>>> the name of the protocol/api used. This is indeed confusing. > >>>>>> > >>>>>> My goal was to have a way to tell the user that his (new) consumer > group > >>>>>> may have non-upgraded members yet. I wonder if we could use a > boolean to > >>>>>> signal this, e.g. isLegacy. Then, in the CLI we could have a column > >>>> which > >>>>>> is only displayed when they are non upgraded members. What do you > think? > >>>>>> > >>>>>> If we cannot find a good way to represent this, we could drop it > from > >>>> this > >>>>>> KIP too. > >>>>>> > >>>>>> Best, > >>>>>> David > >>>>>> > >>>>>> On Fri, Nov 22, 2024 at 5:12 PM PoAn Yang <yangp...@gmail.com> > wrote: > >>>>>> > >>>>>>> Hi Chia-Ping / David, > >>>>>>> > >>>>>>> Thanks for the review and suggestions. > >>>>>>> > >>>>>>>>> I don't fully follow your comment so please excuse me if I say > >>>>>> something > >>>>>>>>> wrong. > >>>>>>> > >>>>>>> I think Chia-Ping wants to say: originally, the default protocol in > >>>>>>> classic group is “consumer”. > >>>>>>> After we have a new consumer group, the protocol of a classic > member in > >>>>>>> consumer group is “classic”. > >>>>>>> The MemberDescription is used by both ClassicGroupDescription and > >>>>>>> ConsumerGroupDescription. > >>>>>>> If we add the protocol field to MemberDescription, the protocol of > a > >>>>>>> classic member in classic group is “consumer”, but it’s “classic” > in > >>>>>>> consumer group. > >>>>>>> To avoid misleading word, we can add a new collection of member > >>>> protocols > >>>>>>> to ConsumerGroupDescription only. > >>>>>>> The original goal is to show member protocol during consumer > migration, > >>>>>> so > >>>>>>> we don’t need to add this information to ClassicGroupDescription. > >>>>>>> > >>>>>>> Please correct me if I misunderstood anything. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> PoAn > >>>>>>> > >>>>>>>> On Nov 22, 2024, at 7:00 PM, Chia-Ping Tsai <chia7...@gmail.com> > >>>>>> wrote: > >>>>>>>> > >>>>>>>> hi DJ > >>>>>>>> > >>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and > >>>>>> change > >>>>>>>> return type of ModernGroup#protocolType? > >>>>>>>> > >>>>>>>> This is not my comment ... sorry for the incorrect quote :( > >>>>>>>> > >>>>>>>>> A member in a classic group should report "classic". > >>>>>>>> > >>>>>>>> I completely agree. However, in this case, > >>>>>>> ClassicGroupDescription#protocol > >>>>>>>> returns 'consumer', while > ClassicGroupDescription#members[x].protocol > >>>>>>>> returns 'classic'. This inconsistency feels a bit strange to me. > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Chia-Ping > >>>>>>>> > >>>>>>>> > >>>>>>>> David Jacot <dja...@confluent.io.invalid> 於 2024年11月22日 週五 > 下午6:51寫道： > >>>>>>>> > >>>>>>>>> Hi Chia, > >>>>>>>>> > >>>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol > enum, but > >>>>>> I > >>>>>>>>> would like to align the format with kafka-groups.sh. > >>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol and > >>>>>> change > >>>>>>>>> return type of ModernGroup#protocolType? > >>>>>>>>> > >>>>>>>>> My understanding is that `GroupProtocol` is used to drive the > config > >>>>>> of > >>>>>>> the > >>>>>>>>> `Consumer`. Hence adding `Share` seems wrong to me. Or are you > >>>>>>> referring to > >>>>>>>>> another GroupProtocol enum? Perhaps GroupType enum? > >>>>>>>>> > >>>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to > >>>>>>>>> MemberDescription, > >>>>>>>>> since the field has different values between classic and consumer > >>>>>>> groups—it > >>>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' in > >>>>>>> consumer > >>>>>>>>> groups. Yes, it seems like a typo, but it's true. This > inconsistency > >>>>>>> feels > >>>>>>>>> confusing to me. For example, users call > Admin#describeConsumerGroups > >>>>>>> and > >>>>>>>>> see that a member has the 'consumer' protocol when it's in a > classic > >>>>>>> group. > >>>>>>>>> Then, if the group is upgraded to a consumer group, calling > >>>>>>>>> Admin#describeConsumerGroups again shows the protocol as > 'classic' > >>>> for > >>>>>>> the > >>>>>>>>> same member. > >>>>>>>>> > >>>>>>>>> I don't fully follow your comment so please excuse me if I say > >>>>>> something > >>>>>>>>> wrong. > >>>>>>>>> > >>>>>>>>> I think that it should work as follows: > >>>>>>>>> * A member in a classic group should report "classic". > >>>>>>>>> * A member using the consumer protocol in a consumer group should > >>>>>> report > >>>>>>>>> "consumer". > >>>>>>>>> * A member using the classic protocol in a consumer group should > >>>>>> report > >>>>>>>>> "classic". > >>>>>>>>> * A member in a share group should report "share". > >>>>>>>>> > >>>>>>>>>> If all we want to do is distinguish classic members in the > >>>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common > >>>>>>>>> MemberDescription structure. Instead, we could add a collection > to > >>>>>>>>> ConsumerGroupDescription to trace members using the classic > protocol, > >>>>>>> which > >>>>>>>>> would allow us to enrich the output of command-line tools. > >>>>>>>>> > >>>>>>>>> Personally, I believe that using different collections per > protocol > >>>>>>> types > >>>>>>>>> is not necessary. A field in the member is more than enough. > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> DJ > >>>>>>>>> > >>>>>>>>> On Fri, Nov 22, 2024 at 12:01 PM Chia-Ping Tsai < > chia7...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> hi PoAn > >>>>>>>>>> > >>>>>>>>>>> LM4: I agree it’s better to use a type like GroupProtocol > enum, but > >>>>>> I > >>>>>>>>>> would like to align the format with kafka-groups.sh. > >>>>>>>>>> How about we use another Jira to add “SHARE” to GroupProtocol > and > >>>>>>> change > >>>>>>>>>> return type of ModernGroup#protocolType? > >>>>>>>>>> > >>>>>>>>>> MemberDescription is also used by ClassicGroupDescription, and > in > >>>>>>> classic > >>>>>>>>>> groups, the protocol can be an arbitrary string, so we can't > use an > >>>>>>> enum. > >>>>>>>>>> > >>>>>>>>>> In fact, I'm wondering if we should even add 'protocol' to > >>>>>>>>>> MemberDescription, > >>>>>>>>>> since the field has different values between classic and > consumer > >>>>>>>>> groups—it > >>>>>>>>>> shows 'consumer' in classic groups, whereas it shows 'classic' > in > >>>>>>>>> consumer > >>>>>>>>>> groups. Yes, it seems like a typo, but it's true. This > inconsistency > >>>>>>>>> feels > >>>>>>>>>> confusing to me. For example, users call > >>>> Admin#describeConsumerGroups > >>>>>>> and > >>>>>>>>>> see that a member has the 'consumer' protocol when it's in a > classic > >>>>>>>>> group. > >>>>>>>>>> Then, if the group is upgraded to a consumer group, calling > >>>>>>>>>> Admin#describeConsumerGroups again shows the protocol as > 'classic' > >>>>>> for > >>>>>>>>> the > >>>>>>>>>> same member. > >>>>>>>>>> > >>>>>>>>>> If all we want to do is distinguish classic members in the > >>>>>>>>>> ConsumerGroupDescription, then we shouldn't modify the common > >>>>>>>>>> MemberDescription structure. Instead, we could add a collection > to > >>>>>>>>>> ConsumerGroupDescription to trace members using the classic > >>>> protocol, > >>>>>>>>> which > >>>>>>>>>> would allow us to enrich the output of command-line tools. > >>>>>>>>>> Best, > >>>>>>>>>> Chia-Ping > >>>>>>>>>> > >>>>>>>>>> PoAn Yang <yangp...@gmail.com> 於 2024年11月22日 週五 下午3:46寫道： > >>>>>>>>>> > >>>>>>>>>>> Hi Lianet / Jeff, > >>>>>>>>>>> > >>>>>>>>>>> Thanks for the review. > >>>>>>>>>>> > >>>>>>>>>>> LM5: The kafka-share-groups.sh also uses “—describe” to show > >>>> offsets > >>>>>>>>>>> information. > >>>>>>>>>>> Change to use “—describe —state —verbose” to show group level > >>>>>>>>> information > >>>>>>>>>>> in kafka-share-groups.sh. > >>>>>>>>>>> > >>>>>>>>>>> LM6: Update the example. > >>>>>>>>>>> > >>>>>>>>>>> LM7: Add a description to mention “—verbose” is a new option in > >>>>>>>>>>> ShareGroupCommandOptions. > >>>>>>>>>>> > >>>>>>>>>>> JK2: Yes, add the statement about format change for ASSIGNMENT > >>>>>> value. > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> PoAn > >>>>>>>>>>> > >>>>>>>>>>>> On Nov 22, 2024, at 7:35 AM, Jeff Kim <jeffkb...@apache.org> > >>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Hi PoAn, > >>>>>>>>>>>> > >>>>>>>>>>>> JK2: Can we state in the KIP that we will reformat the > existing > >>>>>>>>>>>> classic group's `--members --verbose` as well, specifically > the > >>>>>>>>>>>> "ASSIGNMENT" column to include the topic name? > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks! > >>>>>>>>>>>> > >>>>>>>>>>>> On 2024/11/21 14:33:37 "Lianet M." wrote: > >>>>>>>>>>>>> Thanks for the updates PoAn! > >>>>>>>>>>>>> > >>>>>>>>>>>>> LM6. Just a nit, under the --describe --state --verbose the > >>>> header > >>>>>>>>> is > >>>>>>>>>>> fine > >>>>>>>>>>>>> but the example is still missing the --state argument. > >>>>>>>>>>>>> > >>>>>>>>>>>>> LM7. Under the Proposed changes for kafka-share-groups.sh, > should > >>>>>> we > >>>>>>>>>>>>> clearly mention that the KIP adds a new --verbose option? I > think > >>>>>>>>> it's > >>>>>>>>>>>>> confusing because it's presented as if the option already > exists > >>>>>> and > >>>>>>>>>>> we're > >>>>>>>>>>>>> only changing the output (which is the case for > >>>>>>>>> kafka-consumer-groups > >>>>>>>>>>> but > >>>>>>>>>>>>> not for kafka-share-groups) > >>>>>>>>>>>>> > >>>>>>>>>>>>> That's all on my side. Thanks! > >>>>>>>>>>>>> > >>>>>>>>>>>>> Lianet > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Wed, Nov 20, 2024 at 11:12 PM PoAn Yang < > yangp...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi Lianet / Jeff, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks for review and suggestions. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> LM4: At group level, the protocol field is from > >>>>>>>>>>> ModernGroup#protocolType. > >>>>>>>>>>>>>> It’s a string. > >>>>>>>>>>>>>> The value is defined in different places like > >>>>>>>>>> ShareGroup#PROTOCOL_TYPE > >>>>>>>>>>> and > >>>>>>>>>>>>>> ConsumerProtocol#PROTOCOL_TYPE. > >>>>>>>>>>>>>> In KIP-1043 < > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=305171038#KIP1043:Administrationofgroups-kafka-groups.sh > >>>>>>>>>>>> , > >>>>>>>>>>>>>> the kafka-groups.sh also shows protocol like “classic” / > >>>>>>>>> “consumer” / > >>>>>>>>>>>>>> “share”. > >>>>>>>>>>>>>> I agree it’s better to use a type like GroupProtocol enum, > but I > >>>>>>>>>> would > >>>>>>>>>>>>>> like to align the format with kafka-groups.sh. > >>>>>>>>>>>>>> How about we use another Jira to add “SHARE” to > GroupProtocol > >>>> and > >>>>>>>>>>> change > >>>>>>>>>>>>>> return type of ModernGroup#protocolType? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> LM5 / JK1: Sorry, I misunderstood LM3, so I deleted —state > in > >>>> the > >>>>>>>>>>> title. > >>>>>>>>>>>>>> You’re correct. I would like to use `—describe —state > —verbose` > >>>>>> to > >>>>>>>>>> show > >>>>>>>>>>>>>> group level information. > >>>>>>>>>>>>>> For `—describe —verbose`, it will show output as same as > >>>>>> `—describe > >>>>>>>>>>>>>> —offsets —verbose`, > >>>>>>>>>>>>>> because `—describe` also shows output as same as `—describe > >>>>>>>>>> —offsets`. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> JK2: Yes, I only use consumer group in the sample output, > but > >>>> the > >>>>>>>>>>> classic > >>>>>>>>>>>>>> group will use the same format as well. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Nov 21, 2024, at 8:28 AM, Jeff Kim < > jeffkb...@apache.org> > >>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi PoAn, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks for the KIP. I have some questions: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> JK1: (Realized this is similar to LM5) On `--describe > >>>> --verbose` > >>>>>>>>>>>>>> proposed changes - doesn't `--describe $group` default to > >>>>>> printing > >>>>>>>>>> the > >>>>>>>>>>>>>> offsets? Perhaps you're referring to the `--state` argument? > >>>>>> Also, > >>>>>>>>>>> would > >>>>>>>>>>>>>> that mean the default `--describe $group --verbose` command > >>>> would > >>>>>>>>> not > >>>>>>>>>>> print > >>>>>>>>>>>>>> the added field to `--offsets verbose` (leader-epoch) or > would > >>>>>> it? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> JK2: Looking at ConsumerGroupCommand.java, the existing > >>>>>>>>> "ASSIGNMENT" > >>>>>>>>>>>>>> column under `--members --verbose` does group the topic > >>>>>> partitions > >>>>>>>>> by > >>>>>>>>>>> topic > >>>>>>>>>>>>>> but does not prefix the grouping with the topic name like > you're > >>>>>>>>>>> proposal: > >>>>>>>>>>>>>> "my_topic:0,1;new_topic:0,1". Should we do apply the same > format > >>>>>>>>> for > >>>>>>>>>>> the > >>>>>>>>>>>>>> classic group as well? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 2024/11/20 21:32:41 "Lianet M." wrote: > >>>>>>>>>>>>>>>> Hello PoAn, just of couple more minor comments: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> LM4. Regarding the new “protocol” field added to > >>>>>>>>> MemberDescription. > >>>>>>>>>>>>>> Should > >>>>>>>>>>>>>>>> we consider reusing the existing GroupProtocol enum > instead of > >>>>>>>>>>> String? > >>>>>>>>>>>>>>>> (It’s the one we use from the consumer side to refer to > the > >>>>>>>>>> protocol > >>>>>>>>>>> in > >>>>>>>>>>>>>>>> use, just missing Share I notice). > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> LM5. Regarding the change to the output for > >>>>>>>>> kafka-consumer-groups, > >>>>>>>>>>> the > >>>>>>>>>>>>>>>> command shown does not include the —state option, but the > >>>>>> output > >>>>>>>>>>> shows > >>>>>>>>>>>>>>>> state info (state, #members, epochs). I would guess that > we > >>>>>> want > >>>>>>>>> to > >>>>>>>>>>>>>> modify > >>>>>>>>>>>>>>>> the output only when we describe a group with the —state > >>>>>> —verbose > >>>>>>>>>>>>>> option, > >>>>>>>>>>>>>>>> is my understanding right? If my understanding is right > we’re > >>>>>>>>> just > >>>>>>>>>>>>>> missing > >>>>>>>>>>>>>>>> adding the —state in the example, and the KIP does not > >>>>>> introduce > >>>>>>>>>> any > >>>>>>>>>>>>>>>> changes to the —describe —verbose option. (If not, it > would > >>>>>> mean > >>>>>>>>> a > >>>>>>>>>>>>>> bigger > >>>>>>>>>>>>>>>> change to the output of —describe —verbose which I expect > is > >>>>>> not > >>>>>>>>>> the > >>>>>>>>>>>>>>>> intention?) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks! > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Lianet > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Wed, Nov 20, 2024 at 2:12 AM PoAn Yang < > yangp...@gmail.com > >>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi Chia-Ping, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thanks for the review and suggestions. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> CI0: Thanks for the reminder. Update validVersions in > >>>>>>>>>>>>>>>>> ConsumerGroupDescribeRequest to 0-1. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> CI1: Yes, I use `ConsumerGroupMember#useClassicProtocol` > to > >>>>>>>>> check > >>>>>>>>>>>>>> whether > >>>>>>>>>>>>>>>>> a member in consumer group uses “classic” or “consumer” > >>>>>>>>> protocol. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> CI2: Yes, a member in share group always uses “share” > >>>>>> protocol. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> CI3: Add a table to show meaning of “classic”, > “consumer”, > >>>> and > >>>>>>>>>>> “share” > >>>>>>>>>>>>>>>>> protocol. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> BTW, the vote thread is in > >>>>>>>>>>>>>>>>> > >>>>>>>>> https://lists.apache.org/thread/rb25tf75tzf4c7jqqldlo5jh9w8chsq6 > . > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Nov 20, 2024, at 11:46 AM, Chia-Ping Tsai < > >>>>>>>>>> chia7...@apache.org> > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> hi PoAn > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> CI0: We have to bump the version of > >>>>>>>>> ConsumerGroupDescribeRequest > >>>>>>>>>>> as > >>>>>>>>>>>>>>>>> well, so server can distinguish the new and old behavior. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> CI1: the type of new filed is string, so I guess you > plan to > >>>>>>>>> use > >>>>>>>>>>>>>>>>> `ConsumerGroupMember#useClassicProtocol` [0] flag to > return > >>>>>>>>> either > >>>>>>>>>>>>>>>>> "classic" or "consumer", right? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> CI2: `MemberDescription` is used by > `ShareGroupDescription` > >>>>>>>>> too, > >>>>>>>>>> so > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> new filed protocol in shared group is always "shared", > right? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> CI3: Could you consider adding a table to show the > value of > >>>>>> the > >>>>>>>>>>>>>> protocol > >>>>>>>>>>>>>>>>> field in each case? Andrew has a beautiful table in > KIP-1043 > >>>>>>>>> that > >>>>>>>>>>>>>> lists all > >>>>>>>>>>>>>>>>> possible protocol names. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> [0] > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>> > https://github.com/apache/kafka/blob/441a6d0b790f4a17b454caeea7588a6b90fbd9db/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/consumer/ConsumerGroupMember.java#L454 > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>> Chia-Ping > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 2024/11/19 15:45:16 PoAn Yang wrote: > >>>>>>>>>>>>>>>>>>> Hi David, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Thanks for the reminder. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> DJ4: Add related content to the KIP. > >>>>>>>>>>>>>>>>>>> Bump validVersions in ConsumerGroupDescribeResponse to > >>>>>> “0-1”. > >>>>>>>>>>>>>>>>>>> Add a new field “Protocol” to > >>>> ConsumerGroupDescribeResponse. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Nov 19, 2024, at 3:57 PM, David Jacot > >>>>>>>>>>>>>> <dja...@confluent.io.INVALID> > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hi PoAn, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> DJ4: Could you please also include the required > changes to > >>>>>>>>> the > >>>>>>>>>>>>>>>>>>>> ConsumerGroupDescribed API in the public interface > >>>> section? > >>>>>>>>> We > >>>>>>>>>>>>>>>>> basically > >>>>>>>>>>>>>>>>>>>> need to bump the version, add the new field, etc. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>> David > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Mon, Nov 18, 2024 at 5:40 AM PoAn Yang < > >>>>>>>>> yangp...@gmail.com> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Hi David, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> DJ4: Add a new field protocol to MemberDescription, > >>>>>>>>>>>>>>>>>>>>> so the command line tool can show protocol > information > >>>>>> when > >>>>>>>>>>> users > >>>>>>>>>>>>>>>>> describe > >>>>>>>>>>>>>>>>>>>>> members. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a > vote > >>>>>>>>> thread > >>>>>>>>>>>>>> today. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On Nov 15, 2024, at 11:50 PM, David Jacot > >>>>>>>>>>>>>>>>> <dja...@confluent.io.INVALID> > >>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> DJ2: Using "-" sounds good to me. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> DJ3: That seems reasonable to me. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> DJ4: Why not add it right now? I don't want to > change > >>>> the > >>>>>>>>>>> output > >>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> tool too many times. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>>>> David > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 15, 2024 at 3:23 PM PoAn Yang < > >>>>>>>>>> yangp...@gmail.com> > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Hi David / Andrew, > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Thanks for review. Thanks Andrew for picking up > >>>>>>>>>>>>>>>>> kafka-share-groups.sh > >>>>>>>>>>>>>>>>>>>>>>> implementation. > >>>>>>>>>>>>>>>>>>>>>>> I will handle kafka-consumer-groups.sh. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> DJ3: After discussing with @Chia-Ping Tsai, we > think > >>>>>> that > >>>>>>>>>>> using > >>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>>>> format > >>>>>>>>>>>>>>>>>>>>>>> is more clear. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> The new format will be like > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>> > <topic-name1>:<partition-id1>,<partition-id2>;<topic-name2>:<partition-id1>,<partition-id2> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Using colon(:) to concat topic name and partition > IDs. > >>>>>>>>>>>>>>>>>>>>>>> Using comma(,) to concat partition IDs within same > >>>> topic > >>>>>>>>>> name. > >>>>>>>>>>>>>>>>>>>>>>> Using semicolon(;) to concat topic strings. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> AS3: Fix it with kafka-share-groups.sh. Thanks. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> If there is no further suggestion, I will start a > vote > >>>>>>>>>> thread > >>>>>>>>>>>>>> next > >>>>>>>>>>>>>>>>>>>>> Monday. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 9:05 PM, Andrew Schofield < > >>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn, > >>>>>>>>>>>>>>>>>>>>>>>> DJ2: I was just going to comment that "-" would > be a > >>>>>> more > >>>>>>>>>>>>>>>>> appropriate > >>>>>>>>>>>>>>>>>>>>>>> missing value, but > >>>>>>>>>>>>>>>>>>>>>>>> you got there first. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> AS3: The examples for kafka-share-groups.sh > include > >>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh in the > >>>>>>>>>>>>>>>>>>>>>>>> command line. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> If this is accepted in time, I'm happy to pick up > the > >>>>>>>>>>>>>>>>> implementation of > >>>>>>>>>>>>>>>>>>>>>>> the share groups > >>>>>>>>>>>>>>>>>>>>>>>> part of this if it helps. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>> Andrew > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> ________________________________________ > >>>>>>>>>>>>>>>>>>>>>>>> From: Frank Yang <yangp...@gmail.com> > >>>>>>>>>>>>>>>>>>>>>>>> Sent: 14 November 2024 10:48 > >>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org> > >>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend > >>>>>>>>>> kafka-consumer-groups > >>>>>>>>>>>>>>>>> command > >>>>>>>>>>>>>>>>>>>>>>> line tool to support new consumer group > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Hi David, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review and suggestion! I would > like to > >>>>>> get > >>>>>>>>>>> this > >>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>> AK > >>>>>>>>>>>>>>>>>>>>> 4.0 > >>>>>>>>>>>>>>>>>>>>>>> as well. I will do my best. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> DJ1: Update KIP to put GROUP-EPOCH and > >>>>>>>>>>> TARGET-ASSIGNMENT-EPOCH > >>>>>>>>>>>>>>>>> before > >>>>>>>>>>>>>>>>>>>>>>> #MEMBERS. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> DJ2: I prefer to follow current missing column > value > >>>>>> “-“. > >>>>>>>>>>>>>>>>> (reference < > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>> > https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L92 > >>>>>>>>>>>>>>>>>>>>>>>> ) > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> DJ3: Update KIP to use CURRENT-EPOCH > >>>> CURRENT-ASSIGNMENT > >>>>>>>>>>>>>>>>> TARGET-EPOCH > >>>>>>>>>>>>>>>>>>>>>>> TARGET-ASSIGNMENT. > >>>>>>>>>>>>>>>>>>>>>>>> Remove GROUP-EPOCH. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> For assignment value, it follows current output > >>>>>>>>> (reference > >>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>> > https://github.com/apache/kafka/blob/18199028a672fd973ac37bf26316994babc2a6da/tools/src/main/java/org/apache/kafka/tools/consumer/group/ConsumerGroupCommand.java#L413-L418 > >>>>>>>>>>>>>>>>>>>>>> ). > >>>>>>>>>>>>>>>>>>>>>>> I think the form of `topicid-partitionid` is more > >>>> clear. > >>>>>>>>>>>>>>>>>>>>>>>> If we would like to use this form, I will update > both > >>>>>>>>>> output > >>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups.sh and kafka-share-groups.sh. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> DJ4: It looks like DescribeGroupsResponseData > only has > >>>>>>>>>>> protocol > >>>>>>>>>>>>>>>>> type at > >>>>>>>>>>>>>>>>>>>>>>> group level. > >>>>>>>>>>>>>>>>>>>>>>>> Both DescribeGroupsResponseData and > >>>>>>>>>>>>>>>>> ConsumerGroupDescribeResponseData > >>>>>>>>>>>>>>>>>>>>>>> don’t have protocol at member level. > >>>>>>>>>>>>>>>>>>>>>>>> Could we use a followup to add it? > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> DJ5: Update KIP to put LEADER-EPOCH before > >>>>>>>>> CURRENT-OFFSET. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> On Nov 14, 2024, at 3:43 PM, David Jacot > >>>>>>>>>>>>>>>>> <dja...@confluent.io.INVALID > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! I have a few minor > >>>>>>>>>> comments/suggestions: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> DJ1: In the output of `--describe --verbose`, I > would > >>>>>>>>>>> suggest > >>>>>>>>>>>>>>>>> putting > >>>>>>>>>>>>>>>>>>>>>>>>> `GROUP-EPOCH` and `TARGET-ASSIGNMENT-EPOCH` > before > >>>>>>>>>>> `#MEMBERS`. > >>>>>>>>>>>>>>>>>>>>>>>>> DJ2: Continuing on the above, I assume that we > will > >>>>>>>>> print > >>>>>>>>>>> out > >>>>>>>>>>>>>> N/A > >>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>> fields not supported by classic groups. Is this > >>>>>> correct? > >>>>>>>>>>>>>>>>>>>>>>>>> DJ3: In the output of `--describe --members > >>>>>> --verbose`, > >>>>>>>>> I > >>>>>>>>>>>>>> wonder > >>>>>>>>>>>>>>>>> if we > >>>>>>>>>>>>>>>>>>>>>>>>> should use the following order and terms: > >>>>>> CURRENT-EPOCH > >>>>>>>>>>>>>>>>>>>>>>> CURRENT-ASSIGNMENT > >>>>>>>>>>>>>>>>>>>>>>>>> TARGET-EPOCH TARGET-ASSIGNMENT . I would remove > the > >>>>>>>>>>> GROUP-EPOCH > >>>>>>>>>>>>>>>>>>>>> because > >>>>>>>>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>>>>>> is already in the group description. The value > `(0)` > >>>>>>>>> used > >>>>>>>>>>> for > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>> assignment is incorrect. Here I suppose that we > will > >>>>>>>>> print > >>>>>>>>>>> out > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>> list > >>>>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>> partitions in the form of `topicid-partitionid`. > >>>>>>>>>>>>>>>>>>>>>>>>> DJ4: Continuing on the above, I wonder if we > should > >>>>>> also > >>>>>>>>>> add > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>> protocol > >>>>>>>>>>>>>>>>>>>>>>>>> used `classic` or `consumer`. For context, it is > >>>>>>>>> possible > >>>>>>>>>> to > >>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>>>>>>>>> `classic` > >>>>>>>>>>>>>>>>>>>>>>>>> members and `consumer` members in a `consumer` > group > >>>>>>>>>> during > >>>>>>>>>>> an > >>>>>>>>>>>>>>>>> online > >>>>>>>>>>>>>>>>>>>>>>>>> upgrade from the classic protocol to the consumer > >>>>>>>>>> protocol. > >>>>>>>>>>>>>> Having > >>>>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>>>>>>> information may be handy for administrators. > What do > >>>>>> you > >>>>>>>>>>> think? > >>>>>>>>>>>>>>>>>>>>>>>>> DJ5: In the output of `--describe --offsets > >>>>>> --verbose`, > >>>>>>>>> I > >>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>> suggest > >>>>>>>>>>>>>>>>>>>>>>>>> putting `LEADER-EPOCH` closer to > `CURRENT-OFFSET`. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> It would be great if we could get this one in AK > 4.0 > >>>>>> as > >>>>>>>>> it > >>>>>>>>>>>>>>>>> changes the > >>>>>>>>>>>>>>>>>>>>>>>>> output of the command. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>> DJ > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 1, 2024 at 7:40 AM Frank Yang < > >>>>>>>>>>> yangp...@gmail.com> > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Sean / Andrew / Lianet, > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all your review and suggestions. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> AS1, LM1, LM4: Change to add KIP-848 information > >>>> when > >>>>>>>>>> users > >>>>>>>>>>>>>> give > >>>>>>>>>>>>>>>>>>>>>>> —verbose > >>>>>>>>>>>>>>>>>>>>>>>>>> option. > >>>>>>>>>>>>>>>>>>>>>>>>>> —describe —verbose: shows group epoch and target > >>>>>>>>>> assignment > >>>>>>>>>>>>>>>>> epoch. > >>>>>>>>>>>>>>>>>>>>>>>>>> —describe —members —verbose: shows above > >>>> information, > >>>>>>>>>>> member > >>>>>>>>>>>>>>>>> epoch, > >>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>> target assignment. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> AS2: Change to use MEMBER-EPOCH to align with > >>>> KIP-848 > >>>>>>>>>>>>>> definition. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> LM2: For classic group, it doesn’t have epoch, > so I > >>>>>> use > >>>>>>>>>>>>>> Optional > >>>>>>>>>>>>>>>>>>>>>>> fields in > >>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupDescription. > >>>>>>>>>>>>>>>>>>>>>>>>>> For share group, it relies on KIP-848. It must > have > >>>>>>>>>> epoch, > >>>>>>>>>>> so > >>>>>>>>>>>>>> I > >>>>>>>>>>>>>>>>> use > >>>>>>>>>>>>>>>>>>>>> int > >>>>>>>>>>>>>>>>>>>>>>>>>> fields in ShareGroupDescription. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> LM3: Remove —state change. We can get group > level > >>>>>>>>>>> information > >>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>>>>>>>> —describe > >>>>>>>>>>>>>>>>>>>>>>>>>> —verbose. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> SQ1: Add LEADER-EPOCH when users give —describe > >>>>>>>>> —offsets > >>>>>>>>>>>>>>>>> —verbose. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks. > >>>>>>>>>>>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Nov 1, 2024, at 5:08 AM, Lianet M. < > >>>>>>>>>> liane...@gmail.com > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Frank, thanks for the KIP! A few > comments: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> LM1. I strongly agree with Andrew's suggestion > of > >>>>>>>>> moving > >>>>>>>>>>> this > >>>>>>>>>>>>>>>>> into a > >>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option. I expect someone would only > >>>>>> attempt > >>>>>>>>> to > >>>>>>>>>>> make > >>>>>>>>>>>>>>>>> sense > >>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>> epochs while debugging an issue, not while > trying > >>>> to > >>>>>>>>>> view > >>>>>>>>>>> the > >>>>>>>>>>>>>>>>> group > >>>>>>>>>>>>>>>>>>>>> or > >>>>>>>>>>>>>>>>>>>>>>>>>>> member's info. (Also member-epoch makes more > sense > >>>>>> to > >>>>>>>>>> me, > >>>>>>>>>>>>>> even > >>>>>>>>>>>>>>>>> more > >>>>>>>>>>>>>>>>>>>>>>> if we > >>>>>>>>>>>>>>>>>>>>>>>>>>> end up agreeing in a --verbose). Related to > both > >>>>>>>>> issues, > >>>>>>>>>>> my > >>>>>>>>>>>>>>>>> take is > >>>>>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>>>>> whoever is close to the protocol will expect > >>>>>>>>>> member-epoch, > >>>>>>>>>>>>>>>>> whoever > >>>>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>>>>>>>> will probably won't even need to see the > epochs at > >>>>>>>>> all. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> LM2. Why are the epochs defined as Optional > (don't > >>>>>> we > >>>>>>>>>>> expect > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>> always > >>>>>>>>>>>>>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>>>>>>>>>>>>> them? with 0 initially, defined on the broker > side > >>>>>> for > >>>>>>>>>> the > >>>>>>>>>>>>>> group > >>>>>>>>>>>>>>>>>>>>> ones, > >>>>>>>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>> client side for the member) > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> LM3. Why is the KIP including the –state > option in > >>>>>> the > >>>>>>>>>>>>>> proposed > >>>>>>>>>>>>>>>>>>>>>>> changes? > >>>>>>>>>>>>>>>>>>>>>>>>>> ( > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=327977411#KIP1099:Extendkafkaconsumergroupscommandlinetooltosupportnewconsumergroup---state > >>>>>>>>>>>>>>>>>>>>>>>>>>> ). > >>>>>>>>>>>>>>>>>>>>>>>>>>> I get that the output would change in that > example, > >>>>>>>>> but > >>>>>>>>>>> it’s > >>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>>>> because > >>>>>>>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>>>> any change to the –state option itself. It's > >>>> because > >>>>>>>>> of > >>>>>>>>>>> the > >>>>>>>>>>>>>>>>> change > >>>>>>>>>>>>>>>>>>>>>>>>>> proposed > >>>>>>>>>>>>>>>>>>>>>>>>>>> to the –described (required with the --state), > and > >>>>>> the > >>>>>>>>>>>>>> changes > >>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>>>> --describe are already explained above (seems > >>>>>>>>> confusing, > >>>>>>>>>>> got > >>>>>>>>>>>>>> me > >>>>>>>>>>>>>>>>>>>>>>> looking > >>>>>>>>>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>>>>>>> a change to the state filter which seemed > >>>>>> unrelated). > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> LM4. Group epoch and target assignment epoch > are > >>>>>>>>>>> conceptually > >>>>>>>>>>>>>>>>> at the > >>>>>>>>>>>>>>>>>>>>>>>>>> group > >>>>>>>>>>>>>>>>>>>>>>>>>>> level. vs member epoch that lands at a member > >>>> level. > >>>>>>>>> So > >>>>>>>>>>>>>> wonder > >>>>>>>>>>>>>>>>> if we > >>>>>>>>>>>>>>>>>>>>>>>>>> should > >>>>>>>>>>>>>>>>>>>>>>>>>>> show them accordingly (ex. using the --verbose > >>>>>> option) > >>>>>>>>>>>>>>>>>>>>>>>>>>> –describe –verbose => shows group epoch and > target > >>>>>>>>>>> assignment > >>>>>>>>>>>>>>>>> epoch > >>>>>>>>>>>>>>>>>>>>>>>>>>> –describe –members –verbose => shows member > >>>>>>>>>> epoch > >>>>>>>>>>>>>>>>> (along > >>>>>>>>>>>>>>>>>>>>>>>>>>> with group epoch and target assignment epoch) > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks! > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Lianet > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 31, 2024 at 7:30 AM Andrew > Schofield < > >>>>>>>>>>>>>>>>>>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi PoAn, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I have a few comments. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AS1: It seems to me that these new additions > are > >>>>>> most > >>>>>>>>>>>>>> useful to > >>>>>>>>>>>>>>>>>>>>>>> people > >>>>>>>>>>>>>>>>>>>>>>>>>>>> trying to understand > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the progress of rebalancing in quite some > detail. > >>>>>> The > >>>>>>>>>>>>>>>>> information > >>>>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>>>>>>> really not understandable > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for most users who do not have deep knowledge > of > >>>>>>>>>>>>>> KIP-848/932. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> As a result, I suggest for > kafka-share-groups.sh > >>>>>> that > >>>>>>>>>> you > >>>>>>>>>>>>>> add a > >>>>>>>>>>>>>>>>>>>>>>>>>> --members > >>>>>>>>>>>>>>>>>>>>>>>>>>>> --verbose option > >>>>>>>>>>>>>>>>>>>>>>>>>>>> and only include the new information in the > output > >>>>>>>>> for > >>>>>>>>>>> that > >>>>>>>>>>>>>>>>> option, > >>>>>>>>>>>>>>>>>>>>>>>>>> rather > >>>>>>>>>>>>>>>>>>>>>>>>>>>> than changing the > >>>>>>>>>>>>>>>>>>>>>>>>>>>> non-verbose --members output. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I also make a similar suggestion for > >>>>>>>>>>>>>> kafka-consumer-groups.sh > >>>>>>>>>>>>>>>>>>>>>>> --members > >>>>>>>>>>>>>>>>>>>>>>>>>>>> and only add the > >>>>>>>>>>>>>>>>>>>>>>>>>>>> new information for the --verbose output. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> AS2: I strongly suggest that you use > MEMBER-EPOCH > >>>>>>>>>>> instead of > >>>>>>>>>>>>>>>>>>>>>>>>>>>> CONSUMER-EPOCH. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think it's more confusing not following the > >>>>>>>>>>> terminology of > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>> KIPs. > >>>>>>>>>>>>>>>>>>>>>>>>>> For > >>>>>>>>>>>>>>>>>>>>>>>>>>>> one thing, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we already have "member" in the admin client > such > >>>>>> as > >>>>>>>>>>>>>>>>>>>>>>> MemberDescription. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> It's not a > >>>>>>>>>>>>>>>>>>>>>>>>>>>> new term. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Andrew > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ________________________________________ > >>>>>>>>>>>>>>>>>>>>>>>>>>>> From: PoAn Yang <pay...@apache.org> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: 25 October 2024 13:55 > >>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org < > dev@kafka.apache.org> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-1099: Extend > >>>>>>>>>>>>>> kafka-consumer-groups > >>>>>>>>>>>>>>>>>>>>> command > >>>>>>>>>>>>>>>>>>>>>>>>>> line > >>>>>>>>>>>>>>>>>>>>>>>>>>>> tool to support new consumer group > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lucas, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review! > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Yes, I add related change for > >>>>>>>>> kafka-share-groups.sh > >>>>>>>>>> to > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> KIP. > >>>>>>>>>>>>>>>>>>>>>>> Could > >>>>>>>>>>>>>>>>>>>>>>>>>>>> you take a look? Thanks for the suggestion. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) We use CONSUMER-ID as member ID. If we use > >>>>>>>>>>> MEMBER-EPOCH > >>>>>>>>>>>>>>>>> here, > >>>>>>>>>>>>>>>>>>>>>>> users > >>>>>>>>>>>>>>>>>>>>>>>>>> may > >>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse what is different between CONSUMER and > >>>>>>>>> MEMBER. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2024/10/23 13:28:17 Lucas Brutschy wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Frank, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP! > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) For consistency, should we do the same for > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-share-groups.sh, ShareGroupDescription, > >>>>>> etc. ? > >>>>>>>>>>> Even > >>>>>>>>>>>>>> if > >>>>>>>>>>>>>>>>> we do > >>>>>>>>>>>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> implement it right now if the share group > >>>>>>>>>> implementation > >>>>>>>>>>>>>> may > >>>>>>>>>>>>>>>>> still > >>>>>>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> incomplete, it may make sense to include it > in > >>>> the > >>>>>>>>>> KIP. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Why call it CONSUMER-EPOCH, not > MEMBER-EPOCH? > >>>>>>>>> That > >>>>>>>>>>> would > >>>>>>>>>>>>>>>>> seem > >>>>>>>>>>>>>>>>>>>>>>> more > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consistent. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lucas > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2024 at 2:41 PM Frank Yang < > >>>>>>>>>>>>>>>>> yangp...@gmail.com> > >>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to kick off the discussion of > >>>>>>>>> KIP-1099. > >>>>>>>>>>> This > >>>>>>>>>>>>>> KIP > >>>>>>>>>>>>>>>>>>>>>>> enhances > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka-consumer-groups tools to include state > >>>>>> which > >>>>>>>>> is > >>>>>>>>>>>>>>>>> introduced > >>>>>>>>>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-848. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1099: Extend kafka-consumer-groups > command > >>>>>> line > >>>>>>>>>>> tool > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>> support > >>>>>>>>>>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer group - Apache Kafka - Apache > Software > >>>>>>>>>>> Foundation > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cwiki.apache.org > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [image: favicon.ico] > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1099%3A+Extend+kafka-consumer-groups+command+line+tool+to+support+new+consumer+group > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PoAn > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>> > >>>> > >>>> > > > > >
