Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Hi, If there is no further feedback on this KIP, I'll start the vote tomorrow. Thanks. --Vahid From: Vahid S Hashemian/Silicon Valley/IBM To: dev <dev@kafka.apache.org>, "Kafka User" <us...@kafka.apache.org> Date: 07/03/2017 04:06 PM Subject: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand Hi, I created KIP-175 to make some improvements to the ConsumerGroupCommand tool. The KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand Your review and feedback is welcome! Thanks. --Vahid
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Hi Jason, Yes, I meant as a separate KIP. I can start a KIP for that sometime soon. Thanks. --Vahid From: Jason Gustafson <ja...@confluent.io> To: dev@kafka.apache.org Cc: Kafka Users <us...@kafka.apache.org> Date: 07/21/2017 11:37 AM Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand > > Regarding your comment about the current limitation on the information > returned for a consumer group, do you think it's worth expanding the API > to return some additional info (e.g. generation id, group leader, ...)? Seems outside the scope of this KIP. Up to you, but I'd probably leave it for future work. -Jason On Thu, Jul 20, 2017 at 4:21 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Jason, > > Regarding your comment about the current limitation on the information > returned for a consumer group, do you think it's worth expanding the API > to return some additional info (e.g. generation id, group leader, ...)? > > Thanks. > --Vahid > > > > > From: Jason Gustafson <ja...@confluent.io> > To: Kafka Users <us...@kafka.apache.org> > Cc: dev@kafka.apache.org > Date: 07/19/2017 01:46 PM > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > Hey Vahid, > > Thanks for the updates. Looks pretty good. A couple comments: > > 1. For the --state option, should we use the same column-oriented format > as > we use for the other options? I realize there would only be one row, but > the inconsistency is a little vexing. Also, since this tool is working > only > with consumer groups, perhaps we can leave out "protocol type" and use > "assignment strategy" in place of "protocol"? It would be nice to also > include the group generation, but it seems we didn't add that to the > DescribeGroup response. Perhaps we could also include a count of the > number > of members? > 2. It's a little annoying that --subscription and --members share so much > in common. Maybe we could drop --subscription and use a --verbose flag to > control whether or not to include the subscription and perhaps the > assignment as well? Not sure if that's more annoying or less, but maybe a > generic --verbose will be useful in other contexts. > > As for your question on whether we need the --offsets option at all, I > don't have a strong opinion, but it seems to make the command semantics a > little more consistent. > > -Jason > > On Tue, Jul 18, 2017 at 12:56 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > Hi Jason, > > > > I updated the KIP based on your earlier suggestions: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand > > The only thing I am wondering at this point is whether it's worth to > have > > a `--describe --offsets` option that behaves exactly like `--describe`. > > > > Thanks. > > --Vahid > > > > > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > > To: dev@kafka.apache.org > > Cc: Kafka Users <us...@kafka.apache.org> > > Date: 07/17/2017 03:24 PM > > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > > ConsumerGroupCommand > > > > > > > > Hi Jason, > > > > Thanks for your quick feedback. Your suggestions seem reasonable. > > I'll start updating the KIP accordingly and will send out another note > > when it's ready. > > > > Regards. > > --Vahid > > > > > > > > > > From: Jason Gustafson <ja...@confluent.io> > > To: dev@kafka.apache.org > > Cc: Kafka Users <us...@kafka.apache.org> > > Date: 07/17/2017 02:11 PM > > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > > ConsumerGroupCommand > > > > > > > > Hey Vahid, > > > > Hmm... If possible, it would be nice to avoid cluttering the default > > option > > too much, especially if it is information which is going to be the same > > for > > all members (such as the generation). My preference would be to use the > > --state option that you've suggested for that info so that we can > > represent > > it more concisely. > > > > The reason I prefer the current output is that it is clear every entry > > corresponds to a partition for which we have committed offset. Entries > > like > > this look strange: > > > > TOPIC PARTITION CURRENT-
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
> > Regarding your comment about the current limitation on the information > returned for a consumer group, do you think it's worth expanding the API > to return some additional info (e.g. generation id, group leader, ...)? Seems outside the scope of this KIP. Up to you, but I'd probably leave it for future work. -Jason On Thu, Jul 20, 2017 at 4:21 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Jason, > > Regarding your comment about the current limitation on the information > returned for a consumer group, do you think it's worth expanding the API > to return some additional info (e.g. generation id, group leader, ...)? > > Thanks. > --Vahid > > > > > From: Jason Gustafson <ja...@confluent.io> > To: Kafka Users <us...@kafka.apache.org> > Cc: dev@kafka.apache.org > Date: 07/19/2017 01:46 PM > Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > Hey Vahid, > > Thanks for the updates. Looks pretty good. A couple comments: > > 1. For the --state option, should we use the same column-oriented format > as > we use for the other options? I realize there would only be one row, but > the inconsistency is a little vexing. Also, since this tool is working > only > with consumer groups, perhaps we can leave out "protocol type" and use > "assignment strategy" in place of "protocol"? It would be nice to also > include the group generation, but it seems we didn't add that to the > DescribeGroup response. Perhaps we could also include a count of the > number > of members? > 2. It's a little annoying that --subscription and --members share so much > in common. Maybe we could drop --subscription and use a --verbose flag to > control whether or not to include the subscription and perhaps the > assignment as well? Not sure if that's more annoying or less, but maybe a > generic --verbose will be useful in other contexts. > > As for your question on whether we need the --offsets option at all, I > don't have a strong opinion, but it seems to make the command semantics a > little more consistent. > > -Jason > > On Tue, Jul 18, 2017 at 12:56 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > Hi Jason, > > > > I updated the KIP based on your earlier suggestions: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand > > The only thing I am wondering at this point is whether it's worth to > have > > a `--describe --offsets` option that behaves exactly like `--describe`. > > > > Thanks. > > --Vahid > > > > > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > > To: dev@kafka.apache.org > > Cc: Kafka Users <us...@kafka.apache.org> > > Date: 07/17/2017 03:24 PM > > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > > ConsumerGroupCommand > > > > > > > > Hi Jason, > > > > Thanks for your quick feedback. Your suggestions seem reasonable. > > I'll start updating the KIP accordingly and will send out another note > > when it's ready. > > > > Regards. > > --Vahid > > > > > > > > > > From: Jason Gustafson <ja...@confluent.io> > > To: dev@kafka.apache.org > > Cc: Kafka Users <us...@kafka.apache.org> > > Date: 07/17/2017 02:11 PM > > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > > ConsumerGroupCommand > > > > > > > > Hey Vahid, > > > > Hmm... If possible, it would be nice to avoid cluttering the default > > option > > too much, especially if it is information which is going to be the same > > for > > all members (such as the generation). My preference would be to use the > > --state option that you've suggested for that info so that we can > > represent > > it more concisely. > > > > The reason I prefer the current output is that it is clear every entry > > corresponds to a partition for which we have committed offset. Entries > > like > > this look strange: > > > > TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET > > LAGCONSUMER-ID > > HOST CLIENT-ID > > - - - - > > - consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff > > /127.0.0.1 > > consumer4 > > - - - - > > -
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Hi Jason, Regarding your comment about the current limitation on the information returned for a consumer group, do you think it's worth expanding the API to return some additional info (e.g. generation id, group leader, ...)? Thanks. --Vahid From: Jason Gustafson <ja...@confluent.io> To: Kafka Users <us...@kafka.apache.org> Cc: dev@kafka.apache.org Date: 07/19/2017 01:46 PM Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand Hey Vahid, Thanks for the updates. Looks pretty good. A couple comments: 1. For the --state option, should we use the same column-oriented format as we use for the other options? I realize there would only be one row, but the inconsistency is a little vexing. Also, since this tool is working only with consumer groups, perhaps we can leave out "protocol type" and use "assignment strategy" in place of "protocol"? It would be nice to also include the group generation, but it seems we didn't add that to the DescribeGroup response. Perhaps we could also include a count of the number of members? 2. It's a little annoying that --subscription and --members share so much in common. Maybe we could drop --subscription and use a --verbose flag to control whether or not to include the subscription and perhaps the assignment as well? Not sure if that's more annoying or less, but maybe a generic --verbose will be useful in other contexts. As for your question on whether we need the --offsets option at all, I don't have a strong opinion, but it seems to make the command semantics a little more consistent. -Jason On Tue, Jul 18, 2017 at 12:56 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Jason, > > I updated the KIP based on your earlier suggestions: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand > The only thing I am wondering at this point is whether it's worth to have > a `--describe --offsets` option that behaves exactly like `--describe`. > > Thanks. > --Vahid > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > To: dev@kafka.apache.org > Cc: Kafka Users <us...@kafka.apache.org> > Date: 07/17/2017 03:24 PM > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > Hi Jason, > > Thanks for your quick feedback. Your suggestions seem reasonable. > I'll start updating the KIP accordingly and will send out another note > when it's ready. > > Regards. > --Vahid > > > > > From: Jason Gustafson <ja...@confluent.io> > To: dev@kafka.apache.org > Cc: Kafka Users <us...@kafka.apache.org> > Date: 07/17/2017 02:11 PM > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > Hey Vahid, > > Hmm... If possible, it would be nice to avoid cluttering the default > option > too much, especially if it is information which is going to be the same > for > all members (such as the generation). My preference would be to use the > --state option that you've suggested for that info so that we can > represent > it more concisely. > > The reason I prefer the current output is that it is clear every entry > corresponds to a partition for which we have committed offset. Entries > like > this look strange: > > TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET > LAGCONSUMER-ID > HOST CLIENT-ID > - - - - > - consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff > /127.0.0.1 > consumer4 > - - - - > - consumer5-7b80e428-f8ff-43f3-8360-afd1c8ba43ea > /127.0.0.1 > consumer5 > > It makes me think that the consumers have committed offsets for an unknown > partition. The --members option seems like a clearer way to communicate > the > fact that there are some members with no assigned partitions. > > A few additional suggestions: > > 1. Maybe we can rename --partitions to --offsets or --committed-offsets > and > the output could match the default output (in other words, --offsets is > treated as the default switch). Seems no harm including the assignment > information if we have it. > 2. Along the lines of Onur's comment, it would be nice if the --members > option included the list of assignment strategies that the consumer joined > with (round-robin, range, etc). This list should always be small. > 3. Thinking a little more, I'm not sure how necessary a --topics option > is. > The --partitions (or --offsets) option already shows
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
It makes sense. Thanks for clarifying. The KIP is updated based on your feedback. Thanks again. --Vahid From: Jason Gustafson <ja...@confluent.io> To: dev@kafka.apache.org Cc: Kafka Users <us...@kafka.apache.org> Date: 07/19/2017 05:06 PM Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand I was just thinking that the subscription/assignment information can be quite large (especially in MM use cases), so it would be nice to keep the default output concise. I'm also not thrilled about adding more options, but --verbose is sort of a standard one. What do you think? -Jason On Wed, Jul 19, 2017 at 4:39 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Jason, > > Thanks for sharing your feedback on the updated KIP. > Your suggestions look good to me. > Do you see a problem with having the `--members` provide member > subscription and assignment too, so we can avoid an additional `--verbose` > option? > > Thanks. > --Vahid > > > > > From: Jason Gustafson <ja...@confluent.io> > To: Kafka Users <us...@kafka.apache.org> > Cc: dev@kafka.apache.org > Date: 07/19/2017 01:46 PM > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > Hey Vahid, > > Thanks for the updates. Looks pretty good. A couple comments: > > 1. For the --state option, should we use the same column-oriented format > as > we use for the other options? I realize there would only be one row, but > the inconsistency is a little vexing. Also, since this tool is working > only > with consumer groups, perhaps we can leave out "protocol type" and use > "assignment strategy" in place of "protocol"? It would be nice to also > include the group generation, but it seems we didn't add that to the > DescribeGroup response. Perhaps we could also include a count of the > number > of members? > 2. It's a little annoying that --subscription and --members share so much > in common. Maybe we could drop --subscription and use a --verbose flag to > control whether or not to include the subscription and perhaps the > assignment as well? Not sure if that's more annoying or less, but maybe a > generic --verbose will be useful in other contexts. > > As for your question on whether we need the --offsets option at all, I > don't have a strong opinion, but it seems to make the command semantics a > little more consistent. > > -Jason > > On Tue, Jul 18, 2017 at 12:56 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > Hi Jason, > > > > I updated the KIP based on your earlier suggestions: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand > > The only thing I am wondering at this point is whether it's worth to > have > > a `--describe --offsets` option that behaves exactly like `--describe`. > > > > Thanks. > > --Vahid > > > > > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > > To: dev@kafka.apache.org > > Cc: Kafka Users <us...@kafka.apache.org> > > Date: 07/17/2017 03:24 PM > > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > > ConsumerGroupCommand > > > > > > > > Hi Jason, > > > > Thanks for your quick feedback. Your suggestions seem reasonable. > > I'll start updating the KIP accordingly and will send out another note > > when it's ready. > > > > Regards. > > --Vahid > > > > > > > > > > From: Jason Gustafson <ja...@confluent.io> > > To: dev@kafka.apache.org > > Cc: Kafka Users <us...@kafka.apache.org> > > Date: 07/17/2017 02:11 PM > > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > > ConsumerGroupCommand > > > > > > > > Hey Vahid, > > > > Hmm... If possible, it would be nice to avoid cluttering the default > > option > > too much, especially if it is information which is going to be the same > > for > > all members (such as the generation). My preference would be to use the > > --state option that you've suggested for that info so that we can > > represent > > it more concisely. > > > > The reason I prefer the current output is that it is clear every entry > > corresponds to a partition for which we have committed offset. Entries > > like > > this look strange: > > > > TOPIC PARTITION CURRENT-OFFSET LOG-END-O
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
I was just thinking that the subscription/assignment information can be quite large (especially in MM use cases), so it would be nice to keep the default output concise. I'm also not thrilled about adding more options, but --verbose is sort of a standard one. What do you think? -Jason On Wed, Jul 19, 2017 at 4:39 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Jason, > > Thanks for sharing your feedback on the updated KIP. > Your suggestions look good to me. > Do you see a problem with having the `--members` provide member > subscription and assignment too, so we can avoid an additional `--verbose` > option? > > Thanks. > --Vahid > > > > > From: Jason Gustafson <ja...@confluent.io> > To: Kafka Users <us...@kafka.apache.org> > Cc: dev@kafka.apache.org > Date: 07/19/2017 01:46 PM > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > Hey Vahid, > > Thanks for the updates. Looks pretty good. A couple comments: > > 1. For the --state option, should we use the same column-oriented format > as > we use for the other options? I realize there would only be one row, but > the inconsistency is a little vexing. Also, since this tool is working > only > with consumer groups, perhaps we can leave out "protocol type" and use > "assignment strategy" in place of "protocol"? It would be nice to also > include the group generation, but it seems we didn't add that to the > DescribeGroup response. Perhaps we could also include a count of the > number > of members? > 2. It's a little annoying that --subscription and --members share so much > in common. Maybe we could drop --subscription and use a --verbose flag to > control whether or not to include the subscription and perhaps the > assignment as well? Not sure if that's more annoying or less, but maybe a > generic --verbose will be useful in other contexts. > > As for your question on whether we need the --offsets option at all, I > don't have a strong opinion, but it seems to make the command semantics a > little more consistent. > > -Jason > > On Tue, Jul 18, 2017 at 12:56 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > Hi Jason, > > > > I updated the KIP based on your earlier suggestions: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand > > The only thing I am wondering at this point is whether it's worth to > have > > a `--describe --offsets` option that behaves exactly like `--describe`. > > > > Thanks. > > --Vahid > > > > > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > > To: dev@kafka.apache.org > > Cc: Kafka Users <us...@kafka.apache.org> > > Date: 07/17/2017 03:24 PM > > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > > ConsumerGroupCommand > > > > > > > > Hi Jason, > > > > Thanks for your quick feedback. Your suggestions seem reasonable. > > I'll start updating the KIP accordingly and will send out another note > > when it's ready. > > > > Regards. > > --Vahid > > > > > > > > > > From: Jason Gustafson <ja...@confluent.io> > > To: dev@kafka.apache.org > > Cc: Kafka Users <us...@kafka.apache.org> > > Date: 07/17/2017 02:11 PM > > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > > ConsumerGroupCommand > > > > > > > > Hey Vahid, > > > > Hmm... If possible, it would be nice to avoid cluttering the default > > option > > too much, especially if it is information which is going to be the same > > for > > all members (such as the generation). My preference would be to use the > > --state option that you've suggested for that info so that we can > > represent > > it more concisely. > > > > The reason I prefer the current output is that it is clear every entry > > corresponds to a partition for which we have committed offset. Entries > > like > > this look strange: > > > > TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET > > LAGCONSUMER-ID > > HOST CLIENT-ID > > - - - - > > - consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff > > /127.0.0.1 > > consumer4 > > - - - - > > - consumer5-7
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Hi Jason, Thanks for sharing your feedback on the updated KIP. Your suggestions look good to me. Do you see a problem with having the `--members` provide member subscription and assignment too, so we can avoid an additional `--verbose` option? Thanks. --Vahid From: Jason Gustafson <ja...@confluent.io> To: Kafka Users <us...@kafka.apache.org> Cc: dev@kafka.apache.org Date: 07/19/2017 01:46 PM Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand Hey Vahid, Thanks for the updates. Looks pretty good. A couple comments: 1. For the --state option, should we use the same column-oriented format as we use for the other options? I realize there would only be one row, but the inconsistency is a little vexing. Also, since this tool is working only with consumer groups, perhaps we can leave out "protocol type" and use "assignment strategy" in place of "protocol"? It would be nice to also include the group generation, but it seems we didn't add that to the DescribeGroup response. Perhaps we could also include a count of the number of members? 2. It's a little annoying that --subscription and --members share so much in common. Maybe we could drop --subscription and use a --verbose flag to control whether or not to include the subscription and perhaps the assignment as well? Not sure if that's more annoying or less, but maybe a generic --verbose will be useful in other contexts. As for your question on whether we need the --offsets option at all, I don't have a strong opinion, but it seems to make the command semantics a little more consistent. -Jason On Tue, Jul 18, 2017 at 12:56 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Jason, > > I updated the KIP based on your earlier suggestions: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand > The only thing I am wondering at this point is whether it's worth to have > a `--describe --offsets` option that behaves exactly like `--describe`. > > Thanks. > --Vahid > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > To: dev@kafka.apache.org > Cc: Kafka Users <us...@kafka.apache.org> > Date: 07/17/2017 03:24 PM > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > Hi Jason, > > Thanks for your quick feedback. Your suggestions seem reasonable. > I'll start updating the KIP accordingly and will send out another note > when it's ready. > > Regards. > --Vahid > > > > > From: Jason Gustafson <ja...@confluent.io> > To: dev@kafka.apache.org > Cc: Kafka Users <us...@kafka.apache.org> > Date: 07/17/2017 02:11 PM > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > Hey Vahid, > > Hmm... If possible, it would be nice to avoid cluttering the default > option > too much, especially if it is information which is going to be the same > for > all members (such as the generation). My preference would be to use the > --state option that you've suggested for that info so that we can > represent > it more concisely. > > The reason I prefer the current output is that it is clear every entry > corresponds to a partition for which we have committed offset. Entries > like > this look strange: > > TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET > LAGCONSUMER-ID > HOST CLIENT-ID > - - - - > - consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff > /127.0.0.1 > consumer4 > - - - - > - consumer5-7b80e428-f8ff-43f3-8360-afd1c8ba43ea > /127.0.0.1 > consumer5 > > It makes me think that the consumers have committed offsets for an unknown > partition. The --members option seems like a clearer way to communicate > the > fact that there are some members with no assigned partitions. > > A few additional suggestions: > > 1. Maybe we can rename --partitions to --offsets or --committed-offsets > and > the output could match the default output (in other words, --offsets is > treated as the default switch). Seems no harm including the assignment > information if we have it. > 2. Along the lines of Onur's comment, it would be nice if the --members > option included the list of assignment strategies that the consumer joined > with (round-robin, range, etc). This list should always be small. > 3. Thinking a little more, I'm not sure how necessary a --topics option > is. > The --partitions (or --of
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Hey Vahid, Thanks for the updates. Looks pretty good. A couple comments: 1. For the --state option, should we use the same column-oriented format as we use for the other options? I realize there would only be one row, but the inconsistency is a little vexing. Also, since this tool is working only with consumer groups, perhaps we can leave out "protocol type" and use "assignment strategy" in place of "protocol"? It would be nice to also include the group generation, but it seems we didn't add that to the DescribeGroup response. Perhaps we could also include a count of the number of members? 2. It's a little annoying that --subscription and --members share so much in common. Maybe we could drop --subscription and use a --verbose flag to control whether or not to include the subscription and perhaps the assignment as well? Not sure if that's more annoying or less, but maybe a generic --verbose will be useful in other contexts. As for your question on whether we need the --offsets option at all, I don't have a strong opinion, but it seems to make the command semantics a little more consistent. -Jason On Tue, Jul 18, 2017 at 12:56 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Jason, > > I updated the KIP based on your earlier suggestions: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand > The only thing I am wondering at this point is whether it's worth to have > a `--describe --offsets` option that behaves exactly like `--describe`. > > Thanks. > --Vahid > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > To: dev@kafka.apache.org > Cc: Kafka Users <us...@kafka.apache.org> > Date: 07/17/2017 03:24 PM > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > Hi Jason, > > Thanks for your quick feedback. Your suggestions seem reasonable. > I'll start updating the KIP accordingly and will send out another note > when it's ready. > > Regards. > --Vahid > > > > > From: Jason Gustafson <ja...@confluent.io> > To: dev@kafka.apache.org > Cc: Kafka Users <us...@kafka.apache.org> > Date: 07/17/2017 02:11 PM > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > Hey Vahid, > > Hmm... If possible, it would be nice to avoid cluttering the default > option > too much, especially if it is information which is going to be the same > for > all members (such as the generation). My preference would be to use the > --state option that you've suggested for that info so that we can > represent > it more concisely. > > The reason I prefer the current output is that it is clear every entry > corresponds to a partition for which we have committed offset. Entries > like > this look strange: > > TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET > LAGCONSUMER-ID > HOST CLIENT-ID > - - - - > - consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff > /127.0.0.1 > consumer4 > - - - - > - consumer5-7b80e428-f8ff-43f3-8360-afd1c8ba43ea > /127.0.0.1 > consumer5 > > It makes me think that the consumers have committed offsets for an unknown > partition. The --members option seems like a clearer way to communicate > the > fact that there are some members with no assigned partitions. > > A few additional suggestions: > > 1. Maybe we can rename --partitions to --offsets or --committed-offsets > and > the output could match the default output (in other words, --offsets is > treated as the default switch). Seems no harm including the assignment > information if we have it. > 2. Along the lines of Onur's comment, it would be nice if the --members > option included the list of assignment strategies that the consumer joined > with (round-robin, range, etc). This list should always be small. > 3. Thinking a little more, I'm not sure how necessary a --topics option > is. > The --partitions (or --offsets) option already shows the current > assignment. Maybe --topics could be --subscription and just list the > topics > that the members subscribed to? > > Thanks, > Jason > > On Mon, Jul 17, 2017 at 11:04 AM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > Jason, Onur, thank you for reviewing the KIP. > > > > Regarding the default `--describe` option, so far there have been a few > > suggestions that conflict a bit. Here are the suggestions: > > - Keep
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Hi Jason, I updated the KIP based on your earlier suggestions: https://cwiki.apache.org/confluence/display/KAFKA/KIP-175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand The only thing I am wondering at this point is whether it's worth to have a `--describe --offsets` option that behaves exactly like `--describe`. Thanks. --Vahid From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> To: dev@kafka.apache.org Cc: Kafka Users <us...@kafka.apache.org> Date: 07/17/2017 03:24 PM Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand Hi Jason, Thanks for your quick feedback. Your suggestions seem reasonable. I'll start updating the KIP accordingly and will send out another note when it's ready. Regards. --Vahid From: Jason Gustafson <ja...@confluent.io> To: dev@kafka.apache.org Cc: Kafka Users <us...@kafka.apache.org> Date: 07/17/2017 02:11 PM Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand Hey Vahid, Hmm... If possible, it would be nice to avoid cluttering the default option too much, especially if it is information which is going to be the same for all members (such as the generation). My preference would be to use the --state option that you've suggested for that info so that we can represent it more concisely. The reason I prefer the current output is that it is clear every entry corresponds to a partition for which we have committed offset. Entries like this look strange: TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAGCONSUMER-ID HOST CLIENT-ID - - - - - consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff /127.0.0.1 consumer4 - - - - - consumer5-7b80e428-f8ff-43f3-8360-afd1c8ba43ea /127.0.0.1 consumer5 It makes me think that the consumers have committed offsets for an unknown partition. The --members option seems like a clearer way to communicate the fact that there are some members with no assigned partitions. A few additional suggestions: 1. Maybe we can rename --partitions to --offsets or --committed-offsets and the output could match the default output (in other words, --offsets is treated as the default switch). Seems no harm including the assignment information if we have it. 2. Along the lines of Onur's comment, it would be nice if the --members option included the list of assignment strategies that the consumer joined with (round-robin, range, etc). This list should always be small. 3. Thinking a little more, I'm not sure how necessary a --topics option is. The --partitions (or --offsets) option already shows the current assignment. Maybe --topics could be --subscription and just list the topics that the members subscribed to? Thanks, Jason On Mon, Jul 17, 2017 at 11:04 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Jason, Onur, thank you for reviewing the KIP. > > Regarding the default `--describe` option, so far there have been a few > suggestions that conflict a bit. Here are the suggestions: > - Keep the current behavior exactly as is (Edo, Jeff) > - Remove members with no assignments from the current result set (Jason) > - Add additional status info to the result set (Onur) -- I assume the > additional status (which are group related info, rather than group member > related) will appear in the result separate from the member table (e.g., > before the table) > > One thing we could do to remain as close as possible to these suggestions > is trim the resulting rows as per Jason's suggestion, and add the > additional details that Onur suggested. Would this work for everyone? Edo, > Jeff, what do you think? > If so, I'll update the KIP accordingly. > > Some of the other updates based on the feedback received: > * "--describe --members" will not include a topic(partitions) column. > Instead there will be a #Partitions (number of partitions assigned to this > member) column > * "--describe --topics" will be added to list topic partitions in the > group and the relevant info > * "--describe --state" will be added to report group related info, such as > state, protocol, ... > > Thanks. > --Vahid >
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Hi Jason, Thanks for your quick feedback. Your suggestions seem reasonable. I'll start updating the KIP accordingly and will send out another note when it's ready. Regards. --Vahid From: Jason Gustafson <ja...@confluent.io> To: dev@kafka.apache.org Cc: Kafka Users <us...@kafka.apache.org> Date: 07/17/2017 02:11 PM Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand Hey Vahid, Hmm... If possible, it would be nice to avoid cluttering the default option too much, especially if it is information which is going to be the same for all members (such as the generation). My preference would be to use the --state option that you've suggested for that info so that we can represent it more concisely. The reason I prefer the current output is that it is clear every entry corresponds to a partition for which we have committed offset. Entries like this look strange: TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAGCONSUMER-ID HOST CLIENT-ID - - - - - consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff /127.0.0.1 consumer4 - - - - - consumer5-7b80e428-f8ff-43f3-8360-afd1c8ba43ea /127.0.0.1 consumer5 It makes me think that the consumers have committed offsets for an unknown partition. The --members option seems like a clearer way to communicate the fact that there are some members with no assigned partitions. A few additional suggestions: 1. Maybe we can rename --partitions to --offsets or --committed-offsets and the output could match the default output (in other words, --offsets is treated as the default switch). Seems no harm including the assignment information if we have it. 2. Along the lines of Onur's comment, it would be nice if the --members option included the list of assignment strategies that the consumer joined with (round-robin, range, etc). This list should always be small. 3. Thinking a little more, I'm not sure how necessary a --topics option is. The --partitions (or --offsets) option already shows the current assignment. Maybe --topics could be --subscription and just list the topics that the members subscribed to? Thanks, Jason On Mon, Jul 17, 2017 at 11:04 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Jason, Onur, thank you for reviewing the KIP. > > Regarding the default `--describe` option, so far there have been a few > suggestions that conflict a bit. Here are the suggestions: > - Keep the current behavior exactly as is (Edo, Jeff) > - Remove members with no assignments from the current result set (Jason) > - Add additional status info to the result set (Onur) -- I assume the > additional status (which are group related info, rather than group member > related) will appear in the result separate from the member table (e.g., > before the table) > > One thing we could do to remain as close as possible to these suggestions > is trim the resulting rows as per Jason's suggestion, and add the > additional details that Onur suggested. Would this work for everyone? Edo, > Jeff, what do you think? > If so, I'll update the KIP accordingly. > > Some of the other updates based on the feedback received: > * "--describe --members" will not include a topic(partitions) column. > Instead there will be a #Partitions (number of partitions assigned to this > member) column > * "--describe --topics" will be added to list topic partitions in the > group and the relevant info > * "--describe --state" will be added to report group related info, such as > state, protocol, ... > > Thanks. > --Vahid >
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Hey Vahid, Hmm... If possible, it would be nice to avoid cluttering the default option too much, especially if it is information which is going to be the same for all members (such as the generation). My preference would be to use the --state option that you've suggested for that info so that we can represent it more concisely. The reason I prefer the current output is that it is clear every entry corresponds to a partition for which we have committed offset. Entries like this look strange: TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAGCONSUMER-ID HOST CLIENT-ID - - - - - consumer4-e173f09d-c761-4f4e-95c7-6fb73bb8fbff /127.0.0.1 consumer4 - - - - - consumer5-7b80e428-f8ff-43f3-8360-afd1c8ba43ea /127.0.0.1 consumer5 It makes me think that the consumers have committed offsets for an unknown partition. The --members option seems like a clearer way to communicate the fact that there are some members with no assigned partitions. A few additional suggestions: 1. Maybe we can rename --partitions to --offsets or --committed-offsets and the output could match the default output (in other words, --offsets is treated as the default switch). Seems no harm including the assignment information if we have it. 2. Along the lines of Onur's comment, it would be nice if the --members option included the list of assignment strategies that the consumer joined with (round-robin, range, etc). This list should always be small. 3. Thinking a little more, I'm not sure how necessary a --topics option is. The --partitions (or --offsets) option already shows the current assignment. Maybe --topics could be --subscription and just list the topics that the members subscribed to? Thanks, Jason On Mon, Jul 17, 2017 at 11:04 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Jason, Onur, thank you for reviewing the KIP. > > Regarding the default `--describe` option, so far there have been a few > suggestions that conflict a bit. Here are the suggestions: > - Keep the current behavior exactly as is (Edo, Jeff) > - Remove members with no assignments from the current result set (Jason) > - Add additional status info to the result set (Onur) -- I assume the > additional status (which are group related info, rather than group member > related) will appear in the result separate from the member table (e.g., > before the table) > > One thing we could do to remain as close as possible to these suggestions > is trim the resulting rows as per Jason's suggestion, and add the > additional details that Onur suggested. Would this work for everyone? Edo, > Jeff, what do you think? > If so, I'll update the KIP accordingly. > > Some of the other updates based on the feedback received: > * "--describe --members" will not include a topic(partitions) column. > Instead there will be a #Partitions (number of partitions assigned to this > member) column > * "--describe --topics" will be added to list topic partitions in the > group and the relevant info > * "--describe --state" will be added to report group related info, such as > state, protocol, ... > > Thanks. > --Vahid > > > > From: Onur Karaman <onurkaraman.apa...@gmail.com> > To: dev@kafka.apache.org > Cc: Kafka Users <us...@kafka.apache.org> > Date: 07/14/2017 11:40 AM > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > In other words, I think the default should be the exact behavior we have > today plus the remaining group information from DescribeGroupResponse. > > On Fri, Jul 14, 2017 at 11:36 AM, Onur Karaman > <onurkaraman.apa...@gmail.com > > wrote: > > > I think if we had the opportunity to start from scratch, --describe > would > > have been the following: > > --describe --offsets: shows all offsets committed for the group as well > as > > lag > > --describe --state (or maybe --members): shows the full > > DescribeGroupResponse output (including things like generation id, > state, > > protocol type, etc) > > --describe: shows the merged version of the above two. > > > > On Fri, Jul 14, 2017 at 10:56 AM, Jason Gustafson <ja...@confluent.io> > > wrote: > > > >> Hey Vahid, > >> > >> Thanks for the KIP. Looks like a nice improvement. One minor > suggestion: > >> Since consumers can be subscribed to a large number of topics, I'm > >> wondering if it might be better to leave out the topic list from the > >> "describe members" option so that the output remains concise? Perhaps > we &g
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Jason, Onur, thank you for reviewing the KIP. Regarding the default `--describe` option, so far there have been a few suggestions that conflict a bit. Here are the suggestions: - Keep the current behavior exactly as is (Edo, Jeff) - Remove members with no assignments from the current result set (Jason) - Add additional status info to the result set (Onur) -- I assume the additional status (which are group related info, rather than group member related) will appear in the result separate from the member table (e.g., before the table) One thing we could do to remain as close as possible to these suggestions is trim the resulting rows as per Jason's suggestion, and add the additional details that Onur suggested. Would this work for everyone? Edo, Jeff, what do you think? If so, I'll update the KIP accordingly. Some of the other updates based on the feedback received: * "--describe --members" will not include a topic(partitions) column. Instead there will be a #Partitions (number of partitions assigned to this member) column * "--describe --topics" will be added to list topic partitions in the group and the relevant info * "--describe --state" will be added to report group related info, such as state, protocol, ... Thanks. --Vahid From: Onur Karaman <onurkaraman.apa...@gmail.com> To: dev@kafka.apache.org Cc: Kafka Users <us...@kafka.apache.org> Date: 07/14/2017 11:40 AM Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand In other words, I think the default should be the exact behavior we have today plus the remaining group information from DescribeGroupResponse. On Fri, Jul 14, 2017 at 11:36 AM, Onur Karaman <onurkaraman.apa...@gmail.com > wrote: > I think if we had the opportunity to start from scratch, --describe would > have been the following: > --describe --offsets: shows all offsets committed for the group as well as > lag > --describe --state (or maybe --members): shows the full > DescribeGroupResponse output (including things like generation id, state, > protocol type, etc) > --describe: shows the merged version of the above two. > > On Fri, Jul 14, 2017 at 10:56 AM, Jason Gustafson <ja...@confluent.io> > wrote: > >> Hey Vahid, >> >> Thanks for the KIP. Looks like a nice improvement. One minor suggestion: >> Since consumers can be subscribed to a large number of topics, I'm >> wondering if it might be better to leave out the topic list from the >> "describe members" option so that the output remains concise? Perhaps we >> could list only the number of assigned partitions so that users have an >> easy way to check the overall balance and we can add a separate "describe >> topics" switch to see the topic breakdown? >> >> As for the default --describe, it seems safest to keep its current >> behavior. In other words, we should list all partitions which have >> committed offsets for the group even if the partition is not currently >> assigned. However, I don't think we need to try and fit members without >> any >> assigned partitions into that view. >> >> Thanks, >> Jason >> >> On Fri, Jul 7, 2017 at 10:49 AM, Vahid S Hashemian < >> vahidhashem...@us.ibm.com> wrote: >> >> > Thanks Jeff for your feedback on the usefulness of the current tool. >> > >> > --Vahid >> > >> > >> > >> > >> > From: Jeff Widman <j...@netskope.com> >> > To: dev@kafka.apache.org >> > Cc: Kafka User <us...@kafka.apache.org> >> > Date: 07/06/2017 02:25 PM >> > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for >> > ConsumerGroupCommand >> > >> > >> > >> > Thanks for the KIP Vahid. I think it'd be useful to have these filters. >> > >> > That said, I also agree with Edo. >> > >> > We don't currently rely on the output, but there's been more than one >> time >> > when debugging an issue that I notice something amiss when I see all the >> > data at once but if it wasn't present in the default view I probably >> would >> > have missed it as I wouldn't have thought to look at that particular >> > filter. >> > >> > This would also be more consistent with the API of the kafka-topics.sh >> > where "--describe" gives everything and then can be filtered down. >> > >> > >> > >> > On Tue, Jul 4, 2017 at 10:42 AM, Edoardo Comar <eco...@uk.ibm.com> >> wrote: >> > >> > > Hi Vahid, >> > > no we are not relying on parsing th
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
In other words, I think the default should be the exact behavior we have today plus the remaining group information from DescribeGroupResponse. On Fri, Jul 14, 2017 at 11:36 AM, Onur Karaman <onurkaraman.apa...@gmail.com > wrote: > I think if we had the opportunity to start from scratch, --describe would > have been the following: > --describe --offsets: shows all offsets committed for the group as well as > lag > --describe --state (or maybe --members): shows the full > DescribeGroupResponse output (including things like generation id, state, > protocol type, etc) > --describe: shows the merged version of the above two. > > On Fri, Jul 14, 2017 at 10:56 AM, Jason Gustafson <ja...@confluent.io> > wrote: > >> Hey Vahid, >> >> Thanks for the KIP. Looks like a nice improvement. One minor suggestion: >> Since consumers can be subscribed to a large number of topics, I'm >> wondering if it might be better to leave out the topic list from the >> "describe members" option so that the output remains concise? Perhaps we >> could list only the number of assigned partitions so that users have an >> easy way to check the overall balance and we can add a separate "describe >> topics" switch to see the topic breakdown? >> >> As for the default --describe, it seems safest to keep its current >> behavior. In other words, we should list all partitions which have >> committed offsets for the group even if the partition is not currently >> assigned. However, I don't think we need to try and fit members without >> any >> assigned partitions into that view. >> >> Thanks, >> Jason >> >> On Fri, Jul 7, 2017 at 10:49 AM, Vahid S Hashemian < >> vahidhashem...@us.ibm.com> wrote: >> >> > Thanks Jeff for your feedback on the usefulness of the current tool. >> > >> > --Vahid >> > >> > >> > >> > >> > From: Jeff Widman <j...@netskope.com> >> > To: dev@kafka.apache.org >> > Cc: Kafka User <us...@kafka.apache.org> >> > Date: 07/06/2017 02:25 PM >> > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for >> > ConsumerGroupCommand >> > >> > >> > >> > Thanks for the KIP Vahid. I think it'd be useful to have these filters. >> > >> > That said, I also agree with Edo. >> > >> > We don't currently rely on the output, but there's been more than one >> time >> > when debugging an issue that I notice something amiss when I see all the >> > data at once but if it wasn't present in the default view I probably >> would >> > have missed it as I wouldn't have thought to look at that particular >> > filter. >> > >> > This would also be more consistent with the API of the kafka-topics.sh >> > where "--describe" gives everything and then can be filtered down. >> > >> > >> > >> > On Tue, Jul 4, 2017 at 10:42 AM, Edoardo Comar <eco...@uk.ibm.com> >> wrote: >> > >> > > Hi Vahid, >> > > no we are not relying on parsing the current output. >> > > >> > > I just thought that keeping the full output isn't necessarily that bad >> > as >> > > it shows some sort of history of how a group was used. >> > > >> > > ciao >> > > Edo >> > > -- >> > > >> > > Edoardo Comar >> > > >> > > IBM Message Hub >> > > >> > > IBM UK Ltd, Hursley Park, SO21 2JN >> > > >> > > "Vahid S Hashemian" <vahidhashem...@us.ibm.com> wrote on 04/07/2017 >> > > 17:11:43: >> > > >> > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> >> > > > To: dev@kafka.apache.org >> > > > Cc: "Kafka User" <us...@kafka.apache.org> >> > > > Date: 04/07/2017 17:12 >> > > > Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for >> > > > ConsumerGroupCommand >> > > > >> > > > Hi Edo, >> > > > >> > > > Thanks for reviewing the KIP. >> > > > >> > > > Modifying the default behavior of `--describe` was suggested in the >> > > > related JIRA. >> > > > We could poll the community to see whether they go for that option, >> > or, >> > > as >> > >
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
I think if we had the opportunity to start from scratch, --describe would have been the following: --describe --offsets: shows all offsets committed for the group as well as lag --describe --state (or maybe --members): shows the full DescribeGroupResponse output (including things like generation id, state, protocol type, etc) --describe: shows the merged version of the above two. On Fri, Jul 14, 2017 at 10:56 AM, Jason Gustafson <ja...@confluent.io> wrote: > Hey Vahid, > > Thanks for the KIP. Looks like a nice improvement. One minor suggestion: > Since consumers can be subscribed to a large number of topics, I'm > wondering if it might be better to leave out the topic list from the > "describe members" option so that the output remains concise? Perhaps we > could list only the number of assigned partitions so that users have an > easy way to check the overall balance and we can add a separate "describe > topics" switch to see the topic breakdown? > > As for the default --describe, it seems safest to keep its current > behavior. In other words, we should list all partitions which have > committed offsets for the group even if the partition is not currently > assigned. However, I don't think we need to try and fit members without any > assigned partitions into that view. > > Thanks, > Jason > > On Fri, Jul 7, 2017 at 10:49 AM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > Thanks Jeff for your feedback on the usefulness of the current tool. > > > > --Vahid > > > > > > > > > > From: Jeff Widman <j...@netskope.com> > > To: dev@kafka.apache.org > > Cc: Kafka User <us...@kafka.apache.org> > > Date: 07/06/2017 02:25 PM > > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > > ConsumerGroupCommand > > > > > > > > Thanks for the KIP Vahid. I think it'd be useful to have these filters. > > > > That said, I also agree with Edo. > > > > We don't currently rely on the output, but there's been more than one > time > > when debugging an issue that I notice something amiss when I see all the > > data at once but if it wasn't present in the default view I probably > would > > have missed it as I wouldn't have thought to look at that particular > > filter. > > > > This would also be more consistent with the API of the kafka-topics.sh > > where "--describe" gives everything and then can be filtered down. > > > > > > > > On Tue, Jul 4, 2017 at 10:42 AM, Edoardo Comar <eco...@uk.ibm.com> > wrote: > > > > > Hi Vahid, > > > no we are not relying on parsing the current output. > > > > > > I just thought that keeping the full output isn't necessarily that bad > > as > > > it shows some sort of history of how a group was used. > > > > > > ciao > > > Edo > > > -------------- > > > > > > Edoardo Comar > > > > > > IBM Message Hub > > > > > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > > > "Vahid S Hashemian" <vahidhashem...@us.ibm.com> wrote on 04/07/2017 > > > 17:11:43: > > > > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > > > > To: dev@kafka.apache.org > > > > Cc: "Kafka User" <us...@kafka.apache.org> > > > > Date: 04/07/2017 17:12 > > > > Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for > > > > ConsumerGroupCommand > > > > > > > > Hi Edo, > > > > > > > > Thanks for reviewing the KIP. > > > > > > > > Modifying the default behavior of `--describe` was suggested in the > > > > related JIRA. > > > > We could poll the community to see whether they go for that option, > > or, > > > as > > > > you suggested, introducing a new `--only-xxx` ( can't also think of a > > > > proper name right now :) ) option instead. > > > > > > > > Are you making use of the current `--describe` output and relying on > > the > > > > > > > full data set? > > > > > > > > Thanks. > > > > --Vahid > > > > > > > > > > > > > > > > > > > > From: Edoardo Comar <eco...@uk.ibm.com> > > > > To: dev@kafka.apache.org > > > > Cc: "Kafka User" <us...@kafka.apache.org> > > > > Date
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Hey Vahid, Thanks for the KIP. Looks like a nice improvement. One minor suggestion: Since consumers can be subscribed to a large number of topics, I'm wondering if it might be better to leave out the topic list from the "describe members" option so that the output remains concise? Perhaps we could list only the number of assigned partitions so that users have an easy way to check the overall balance and we can add a separate "describe topics" switch to see the topic breakdown? As for the default --describe, it seems safest to keep its current behavior. In other words, we should list all partitions which have committed offsets for the group even if the partition is not currently assigned. However, I don't think we need to try and fit members without any assigned partitions into that view. Thanks, Jason On Fri, Jul 7, 2017 at 10:49 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Thanks Jeff for your feedback on the usefulness of the current tool. > > --Vahid > > > > > From: Jeff Widman <j...@netskope.com> > To: dev@kafka.apache.org > Cc: Kafka User <us...@kafka.apache.org> > Date: 07/06/2017 02:25 PM > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > Thanks for the KIP Vahid. I think it'd be useful to have these filters. > > That said, I also agree with Edo. > > We don't currently rely on the output, but there's been more than one time > when debugging an issue that I notice something amiss when I see all the > data at once but if it wasn't present in the default view I probably would > have missed it as I wouldn't have thought to look at that particular > filter. > > This would also be more consistent with the API of the kafka-topics.sh > where "--describe" gives everything and then can be filtered down. > > > > On Tue, Jul 4, 2017 at 10:42 AM, Edoardo Comar <eco...@uk.ibm.com> wrote: > > > Hi Vahid, > > no we are not relying on parsing the current output. > > > > I just thought that keeping the full output isn't necessarily that bad > as > > it shows some sort of history of how a group was used. > > > > ciao > > Edo > > -- > > > > Edoardo Comar > > > > IBM Message Hub > > > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > "Vahid S Hashemian" <vahidhashem...@us.ibm.com> wrote on 04/07/2017 > > 17:11:43: > > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > > > To: dev@kafka.apache.org > > > Cc: "Kafka User" <us...@kafka.apache.org> > > > Date: 04/07/2017 17:12 > > > Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for > > > ConsumerGroupCommand > > > > > > Hi Edo, > > > > > > Thanks for reviewing the KIP. > > > > > > Modifying the default behavior of `--describe` was suggested in the > > > related JIRA. > > > We could poll the community to see whether they go for that option, > or, > > as > > > you suggested, introducing a new `--only-xxx` ( can't also think of a > > > proper name right now :) ) option instead. > > > > > > Are you making use of the current `--describe` output and relying on > the > > > > > full data set? > > > > > > Thanks. > > > --Vahid > > > > > > > > > > > > > > > From: Edoardo Comar <eco...@uk.ibm.com> > > > To: dev@kafka.apache.org > > > Cc: "Kafka User" <us...@kafka.apache.org> > > > Date: 07/04/2017 03:17 AM > > > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views > for > > > > > ConsumerGroupCommand > > > > > > > > > > > > Thanks Vahid, I like the KIP. > > > > > > One question - could we keep the current "--describe" behavior > unchanged > > > > > and introduce "--only-xxx" options to filter down the full output as > you > > > > > proposed ? > > > > > > ciao, > > > Edo > > > -- > > > > > > Edoardo Comar > > > > > > IBM Message Hub > > > > > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > > > > > > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > > > To: dev <dev@kafka.apache.org>, "Kafka User" > > <us..
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Thanks Jeff for your feedback on the usefulness of the current tool. --Vahid From: Jeff Widman <j...@netskope.com> To: dev@kafka.apache.org Cc: Kafka User <us...@kafka.apache.org> Date: 07/06/2017 02:25 PM Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand Thanks for the KIP Vahid. I think it'd be useful to have these filters. That said, I also agree with Edo. We don't currently rely on the output, but there's been more than one time when debugging an issue that I notice something amiss when I see all the data at once but if it wasn't present in the default view I probably would have missed it as I wouldn't have thought to look at that particular filter. This would also be more consistent with the API of the kafka-topics.sh where "--describe" gives everything and then can be filtered down. On Tue, Jul 4, 2017 at 10:42 AM, Edoardo Comar <eco...@uk.ibm.com> wrote: > Hi Vahid, > no we are not relying on parsing the current output. > > I just thought that keeping the full output isn't necessarily that bad as > it shows some sort of history of how a group was used. > > ciao > Edo > -- > > Edoardo Comar > > IBM Message Hub > > IBM UK Ltd, Hursley Park, SO21 2JN > > "Vahid S Hashemian" <vahidhashem...@us.ibm.com> wrote on 04/07/2017 > 17:11:43: > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > > To: dev@kafka.apache.org > > Cc: "Kafka User" <us...@kafka.apache.org> > > Date: 04/07/2017 17:12 > > Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for > > ConsumerGroupCommand > > > > Hi Edo, > > > > Thanks for reviewing the KIP. > > > > Modifying the default behavior of `--describe` was suggested in the > > related JIRA. > > We could poll the community to see whether they go for that option, or, > as > > you suggested, introducing a new `--only-xxx` ( can't also think of a > > proper name right now :) ) option instead. > > > > Are you making use of the current `--describe` output and relying on the > > > full data set? > > > > Thanks. > > --Vahid > > > > > > > > > > From: Edoardo Comar <eco...@uk.ibm.com> > > To: dev@kafka.apache.org > > Cc: "Kafka User" <us...@kafka.apache.org> > > Date: 07/04/2017 03:17 AM > > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > > > ConsumerGroupCommand > > > > > > > > Thanks Vahid, I like the KIP. > > > > One question - could we keep the current "--describe" behavior unchanged > > > and introduce "--only-xxx" options to filter down the full output as you > > > proposed ? > > > > ciao, > > Edo > > -- > > > > Edoardo Comar > > > > IBM Message Hub > > > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > > To: dev <dev@kafka.apache.org>, "Kafka User" > <us...@kafka.apache.org> > > Date: 04/07/2017 00:06 > > Subject:[DISCUSS] KIP-175: Additional '--describe' views for > > ConsumerGroupCommand > > > > > > > > Hi, > > > > I created KIP-175 to make some improvements to the ConsumerGroupCommand > > tool. > > The KIP can be found here: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-175%3A > > +Additional+%27--describe%27+views+for+ConsumerGroupCommand > > > > > > > > Your review and feedback is welcome! > > > > Thanks. > > --Vahid > > > > > > > > > > > > Unless stated otherwise above: > > IBM United Kingdom Limited - Registered in England and Wales with number > > > 741598. > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 > 3AU > > > > > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Thanks for the KIP Vahid. I think it'd be useful to have these filters. That said, I also agree with Edo. We don't currently rely on the output, but there's been more than one time when debugging an issue that I notice something amiss when I see all the data at once but if it wasn't present in the default view I probably would have missed it as I wouldn't have thought to look at that particular filter. This would also be more consistent with the API of the kafka-topics.sh where "--describe" gives everything and then can be filtered down. On Tue, Jul 4, 2017 at 10:42 AM, Edoardo Comar <eco...@uk.ibm.com> wrote: > Hi Vahid, > no we are not relying on parsing the current output. > > I just thought that keeping the full output isn't necessarily that bad as > it shows some sort of history of how a group was used. > > ciao > Edo > -- > > Edoardo Comar > > IBM Message Hub > > IBM UK Ltd, Hursley Park, SO21 2JN > > "Vahid S Hashemian" <vahidhashem...@us.ibm.com> wrote on 04/07/2017 > 17:11:43: > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > > To: dev@kafka.apache.org > > Cc: "Kafka User" <us...@kafka.apache.org> > > Date: 04/07/2017 17:12 > > Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for > > ConsumerGroupCommand > > > > Hi Edo, > > > > Thanks for reviewing the KIP. > > > > Modifying the default behavior of `--describe` was suggested in the > > related JIRA. > > We could poll the community to see whether they go for that option, or, > as > > you suggested, introducing a new `--only-xxx` ( can't also think of a > > proper name right now :) ) option instead. > > > > Are you making use of the current `--describe` output and relying on the > > > full data set? > > > > Thanks. > > --Vahid > > > > > > > > > > From: Edoardo Comar <eco...@uk.ibm.com> > > To: dev@kafka.apache.org > > Cc: "Kafka User" <us...@kafka.apache.org> > > Date: 07/04/2017 03:17 AM > > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > > > ConsumerGroupCommand > > > > > > > > Thanks Vahid, I like the KIP. > > > > One question - could we keep the current "--describe" behavior unchanged > > > and introduce "--only-xxx" options to filter down the full output as you > > > proposed ? > > > > ciao, > > Edo > > -- > > > > Edoardo Comar > > > > IBM Message Hub > > > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > > To: dev <dev@kafka.apache.org>, "Kafka User" > <us...@kafka.apache.org> > > Date: 04/07/2017 00:06 > > Subject:[DISCUSS] KIP-175: Additional '--describe' views for > > ConsumerGroupCommand > > > > > > > > Hi, > > > > I created KIP-175 to make some improvements to the ConsumerGroupCommand > > tool. > > The KIP can be found here: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-175%3A > > +Additional+%27--describe%27+views+for+ConsumerGroupCommand > > > > > > > > Your review and feedback is welcome! > > > > Thanks. > > --Vahid > > > > > > > > > > > > Unless stated otherwise above: > > IBM United Kingdom Limited - Registered in England and Wales with number > > > 741598. > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 > 3AU > > > > > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Hi Vahid, no we are not relying on parsing the current output. I just thought that keeping the full output isn't necessarily that bad as it shows some sort of history of how a group was used. ciao Edo -- Edoardo Comar IBM Message Hub IBM UK Ltd, Hursley Park, SO21 2JN "Vahid S Hashemian" <vahidhashem...@us.ibm.com> wrote on 04/07/2017 17:11:43: > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > To: dev@kafka.apache.org > Cc: "Kafka User" <us...@kafka.apache.org> > Date: 04/07/2017 17:12 > Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > Hi Edo, > > Thanks for reviewing the KIP. > > Modifying the default behavior of `--describe` was suggested in the > related JIRA. > We could poll the community to see whether they go for that option, or, as > you suggested, introducing a new `--only-xxx` ( can't also think of a > proper name right now :) ) option instead. > > Are you making use of the current `--describe` output and relying on the > full data set? > > Thanks. > --Vahid > > > > > From: Edoardo Comar <eco...@uk.ibm.com> > To: dev@kafka.apache.org > Cc: "Kafka User" <us...@kafka.apache.org> > Date: 07/04/2017 03:17 AM > Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > Thanks Vahid, I like the KIP. > > One question - could we keep the current "--describe" behavior unchanged > and introduce "--only-xxx" options to filter down the full output as you > proposed ? > > ciao, > Edo > -- > > Edoardo Comar > > IBM Message Hub > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> > To: dev <dev@kafka.apache.org>, "Kafka User" <us...@kafka.apache.org> > Date: 04/07/2017 00:06 > Subject:[DISCUSS] KIP-175: Additional '--describe' views for > ConsumerGroupCommand > > > > Hi, > > I created KIP-175 to make some improvements to the ConsumerGroupCommand > tool. > The KIP can be found here: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-175%3A > +Additional+%27--describe%27+views+for+ConsumerGroupCommand > > > > Your review and feedback is welcome! > > Thanks. > --Vahid > > > > > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > > > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Hi Edo, Thanks for reviewing the KIP. Modifying the default behavior of `--describe` was suggested in the related JIRA. We could poll the community to see whether they go for that option, or, as you suggested, introducing a new `--only-xxx` ( can't also think of a proper name right now :) ) option instead. Are you making use of the current `--describe` output and relying on the full data set? Thanks. --Vahid From: Edoardo Comar <eco...@uk.ibm.com> To: dev@kafka.apache.org Cc: "Kafka User" <us...@kafka.apache.org> Date: 07/04/2017 03:17 AM Subject: Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand Thanks Vahid, I like the KIP. One question - could we keep the current "--describe" behavior unchanged and introduce "--only-xxx" options to filter down the full output as you proposed ? ciao, Edo -- Edoardo Comar IBM Message Hub IBM UK Ltd, Hursley Park, SO21 2JN From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> To: dev <dev@kafka.apache.org>, "Kafka User" <us...@kafka.apache.org> Date: 04/07/2017 00:06 Subject: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand Hi, I created KIP-175 to make some improvements to the ConsumerGroupCommand tool. The KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand Your review and feedback is welcome! Thanks. --Vahid Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Thanks Vahid, I like the KIP. One question - could we keep the current "--describe" behavior unchanged and introduce "--only-xxx" options to filter down the full output as you proposed ? ciao, Edo -- Edoardo Comar IBM Message Hub IBM UK Ltd, Hursley Park, SO21 2JN From: "Vahid S Hashemian" <vahidhashem...@us.ibm.com> To: dev <dev@kafka.apache.org>, "Kafka User" <us...@kafka.apache.org> Date: 04/07/2017 00:06 Subject:[DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand Hi, I created KIP-175 to make some improvements to the ConsumerGroupCommand tool. The KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand Your review and feedback is welcome! Thanks. --Vahid Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
[DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand
Hi, I created KIP-175 to make some improvements to the ConsumerGroupCommand tool. The KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand Your review and feedback is welcome! Thanks. --Vahid
