Re: [DISCUSS] KIP-308: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimize the number of requests to server

[Dániel Urbán]

That's a good question. In the PR I submitted, it would result in a list of
partitions contained by a topic for which the user has DESCRIBE privilege.
The tool utilizes Consumer.listTopics, so unauthorized topics are not
present in the response at all. The current version in trunk simply reports
that the topic does not exist if the user has no DESCRIBE privilege for it.

It would be hard (impossible?) to support detailed information about
unauthorized topics when using a pattern as a filter. It could be
manageable if the tool only supported a list of topics.

Maybe the only improvement needed is to explicitly document that the tool
only scans the authorized topics?

Hu Xi  ezt írta (időpont: 2020. jún. 30., K, 17:04):

> That's a great KIP for GetOffsetShell tool. I have a question about the
> multiple-topic lookup situation.
>
> In a secured environment, what does the tool output if it has DESCRIBE
> privileges for some topics but hasn't for others?
>
> 
> 发件人: Dániel Urbán 
> 发送时间: 2020年6月30日 22:15
> 收件人: dev@kafka.apache.org 
> 主题: Re: [DISCUSS] KIP-308: GetOffsetShell: new KafkaConsumer API, support
> for multiple topics, minimize the number of requests to server
>
> Hi Manikumar,
> Thanks, went ahead and assigned a new ID, it is KIP-635 now:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-635%3A+GetOffsetShell%3A+support+for+multiple+topics+and+consumer+configuration+override
> Daniel
>
> Manikumar  ezt írta (időpont: 2020. jún. 30.,
> K,
> 16:03):
>
> > Hi,
> >
> > Yes, we can assign new id to this KIP.
> >
> > Thanks.
> >
> > On Tue, Jun 30, 2020 at 6:59 PM Dániel Urbán 
> > wrote:
> >
> > > Hi,
> > >
> > > To help with the discussion, I also have a PR for this KIP now.
> > reflecting
> > > the current state of the KIP:
> https://github.com/apache/kafka/pull/8957.
> > > I would like to ask a committer to start the test job on it.
> > >
> > > One thing I realised though is that there is a KIP id collision, there
> is
> > > another KIP with the same id:
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=85474993
> > > What is the protocol in this case? Should I acquire a new id for the
> > > GetOffsetShell KIP, and update it?
> > >
> > > Thanks in advance,
> > > Daniel
> > >
> > > Dániel Urbán  ezt írta (időpont: 2020.
> jún.
> > > 30., K, 9:23):
> > >
> > > > Hi Manikumar,
> > > >
> > > > Thanks for the comments.
> > > > 1. Will change this - thought that "command-config" is used for admin
> > > > clients.
> > > > 2. It's not necessary, just felt like a nice quality-of-life feature
> -
> > > will
> > > > remove it.
> > > >
> > > > Thanks,
> > > > Daniel
> > > >
> > > > On Tue, Jun 30, 2020 at 4:16 AM Manikumar  >
> > > > wrote:
> > > >
> > > > > Hi Daniel,
> > > > >
> > > > > Thanks for working on this KIP.  Proposed changes looks good to me,
> > > > >
> > > > > minor comments:
> > > > > 1. We use "command-config" option name in most of the cmdline tools
> > to
> > > > pass
> > > > > config
> > > > > properties file. We can use the same name here.
> > > > >
> > > > > 2. Not sure, if we need a separate option to pass an consumer
> > property.
> > > > > fewer options are better.
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > > > On Wed, Jun 24, 2020 at 8:53 PM Dániel Urbán <
> urb.dani...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I see that this KIP turned somewhat inactive - I'd like to pick
> it
> > up
> > > > and
> > > > > > work on it if it is okay.
> > > > > > Part of the work is done, as switching to the Consumer API is
> > already
> > > > in
> > > > > > trunk, but some functionality is still missing.
> > > > > >
> > > > > > I've seen the current PR and the discussion so far, only have a
> few
> > > > > things
> > > > > > to add:
> > > > > > - I like the idea of the topic-partition argument, it would be
> > useful
> > > > to
> > > > > > filter down to specific partitions.
> > > > > > - Instead of a topic list arg, a pattern would be more powerful,
> > and
> > > > also
> > > > > > fit better with the other tools (e.g. how the kafka-topics tool
> > > works).
> > > > > >
> > > > > > Regards,
> > > > > > Daniel
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-308: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimize the number of requests to server

[Dániel Urbán]

Hi Manikumar,
Thanks, went ahead and assigned a new ID, it is KIP-635 now:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-635%3A+GetOffsetShell%3A+support+for+multiple+topics+and+consumer+configuration+override
Daniel

Manikumar  ezt írta (időpont: 2020. jún. 30., K,
16:03):

> Hi,
>
> Yes, we can assign new id to this KIP.
>
> Thanks.
>
> On Tue, Jun 30, 2020 at 6:59 PM Dániel Urbán 
> wrote:
>
> > Hi,
> >
> > To help with the discussion, I also have a PR for this KIP now.
> reflecting
> > the current state of the KIP: https://github.com/apache/kafka/pull/8957.
> > I would like to ask a committer to start the test job on it.
> >
> > One thing I realised though is that there is a KIP id collision, there is
> > another KIP with the same id:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=85474993
> > What is the protocol in this case? Should I acquire a new id for the
> > GetOffsetShell KIP, and update it?
> >
> > Thanks in advance,
> > Daniel
> >
> > Dániel Urbán  ezt írta (időpont: 2020. jún.
> > 30., K, 9:23):
> >
> > > Hi Manikumar,
> > >
> > > Thanks for the comments.
> > > 1. Will change this - thought that "command-config" is used for admin
> > > clients.
> > > 2. It's not necessary, just felt like a nice quality-of-life feature -
> > will
> > > remove it.
> > >
> > > Thanks,
> > > Daniel
> > >
> > > On Tue, Jun 30, 2020 at 4:16 AM Manikumar 
> > > wrote:
> > >
> > > > Hi Daniel,
> > > >
> > > > Thanks for working on this KIP.  Proposed changes looks good to me,
> > > >
> > > > minor comments:
> > > > 1. We use "command-config" option name in most of the cmdline tools
> to
> > > pass
> > > > config
> > > > properties file. We can use the same name here.
> > > >
> > > > 2. Not sure, if we need a separate option to pass an consumer
> property.
> > > > fewer options are better.
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > > > On Wed, Jun 24, 2020 at 8:53 PM Dániel Urbán 
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I see that this KIP turned somewhat inactive - I'd like to pick it
> up
> > > and
> > > > > work on it if it is okay.
> > > > > Part of the work is done, as switching to the Consumer API is
> already
> > > in
> > > > > trunk, but some functionality is still missing.
> > > > >
> > > > > I've seen the current PR and the discussion so far, only have a few
> > > > things
> > > > > to add:
> > > > > - I like the idea of the topic-partition argument, it would be
> useful
> > > to
> > > > > filter down to specific partitions.
> > > > > - Instead of a topic list arg, a pattern would be more powerful,
> and
> > > also
> > > > > fit better with the other tools (e.g. how the kafka-topics tool
> > works).
> > > > >
> > > > > Regards,
> > > > > Daniel
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-308: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimize the number of requests to server

[Manikumar]

Hi,

Yes, we can assign new id to this KIP.

Thanks.

On Tue, Jun 30, 2020 at 6:59 PM Dániel Urbán  wrote:

> Hi,
>
> To help with the discussion, I also have a PR for this KIP now. reflecting
> the current state of the KIP: https://github.com/apache/kafka/pull/8957.
> I would like to ask a committer to start the test job on it.
>
> One thing I realised though is that there is a KIP id collision, there is
> another KIP with the same id:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=85474993
> What is the protocol in this case? Should I acquire a new id for the
> GetOffsetShell KIP, and update it?
>
> Thanks in advance,
> Daniel
>
> Dániel Urbán  ezt írta (időpont: 2020. jún.
> 30., K, 9:23):
>
> > Hi Manikumar,
> >
> > Thanks for the comments.
> > 1. Will change this - thought that "command-config" is used for admin
> > clients.
> > 2. It's not necessary, just felt like a nice quality-of-life feature -
> will
> > remove it.
> >
> > Thanks,
> > Daniel
> >
> > On Tue, Jun 30, 2020 at 4:16 AM Manikumar 
> > wrote:
> >
> > > Hi Daniel,
> > >
> > > Thanks for working on this KIP.  Proposed changes looks good to me,
> > >
> > > minor comments:
> > > 1. We use "command-config" option name in most of the cmdline tools to
> > pass
> > > config
> > > properties file. We can use the same name here.
> > >
> > > 2. Not sure, if we need a separate option to pass an consumer property.
> > > fewer options are better.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > > On Wed, Jun 24, 2020 at 8:53 PM Dániel Urbán 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I see that this KIP turned somewhat inactive - I'd like to pick it up
> > and
> > > > work on it if it is okay.
> > > > Part of the work is done, as switching to the Consumer API is already
> > in
> > > > trunk, but some functionality is still missing.
> > > >
> > > > I've seen the current PR and the discussion so far, only have a few
> > > things
> > > > to add:
> > > > - I like the idea of the topic-partition argument, it would be useful
> > to
> > > > filter down to specific partitions.
> > > > - Instead of a topic list arg, a pattern would be more powerful, and
> > also
> > > > fit better with the other tools (e.g. how the kafka-topics tool
> works).
> > > >
> > > > Regards,
> > > > Daniel
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-308: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimize the number of requests to server

[Dániel Urbán]

Hi,

To help with the discussion, I also have a PR for this KIP now. reflecting
the current state of the KIP: https://github.com/apache/kafka/pull/8957.
I would like to ask a committer to start the test job on it.

One thing I realised though is that there is a KIP id collision, there is
another KIP with the same id:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=85474993
What is the protocol in this case? Should I acquire a new id for the
GetOffsetShell KIP, and update it?

Thanks in advance,
Daniel

Dániel Urbán  ezt írta (időpont: 2020. jún.
30., K, 9:23):

> Hi Manikumar,
>
> Thanks for the comments.
> 1. Will change this - thought that "command-config" is used for admin
> clients.
> 2. It's not necessary, just felt like a nice quality-of-life feature - will
> remove it.
>
> Thanks,
> Daniel
>
> On Tue, Jun 30, 2020 at 4:16 AM Manikumar 
> wrote:
>
> > Hi Daniel,
> >
> > Thanks for working on this KIP.  Proposed changes looks good to me,
> >
> > minor comments:
> > 1. We use "command-config" option name in most of the cmdline tools to
> pass
> > config
> > properties file. We can use the same name here.
> >
> > 2. Not sure, if we need a separate option to pass an consumer property.
> > fewer options are better.
> >
> > Thanks,
> > Manikumar
> >
> > On Wed, Jun 24, 2020 at 8:53 PM Dániel Urbán 
> > wrote:
> >
> > > Hi,
> > >
> > > I see that this KIP turned somewhat inactive - I'd like to pick it up
> and
> > > work on it if it is okay.
> > > Part of the work is done, as switching to the Consumer API is already
> in
> > > trunk, but some functionality is still missing.
> > >
> > > I've seen the current PR and the discussion so far, only have a few
> > things
> > > to add:
> > > - I like the idea of the topic-partition argument, it would be useful
> to
> > > filter down to specific partitions.
> > > - Instead of a topic list arg, a pattern would be more powerful, and
> also
> > > fit better with the other tools (e.g. how the kafka-topics tool works).
> > >
> > > Regards,
> > > Daniel
> > >
> >
>

Re: [DISCUSS] KIP-308: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimize the number of requests to server

[Dániel Urbán]

Hi Manikumar,

Thanks for the comments.
1. Will change this - thought that "command-config" is used for admin
clients.
2. It's not necessary, just felt like a nice quality-of-life feature - will
remove it.

Thanks,
Daniel

On Tue, Jun 30, 2020 at 4:16 AM Manikumar  wrote:

> Hi Daniel,
>
> Thanks for working on this KIP.  Proposed changes looks good to me,
>
> minor comments:
> 1. We use "command-config" option name in most of the cmdline tools to pass
> config
> properties file. We can use the same name here.
>
> 2. Not sure, if we need a separate option to pass an consumer property.
> fewer options are better.
>
> Thanks,
> Manikumar
>
> On Wed, Jun 24, 2020 at 8:53 PM Dániel Urbán 
> wrote:
>
> > Hi,
> >
> > I see that this KIP turned somewhat inactive - I'd like to pick it up and
> > work on it if it is okay.
> > Part of the work is done, as switching to the Consumer API is already in
> > trunk, but some functionality is still missing.
> >
> > I've seen the current PR and the discussion so far, only have a few
> things
> > to add:
> > - I like the idea of the topic-partition argument, it would be useful to
> > filter down to specific partitions.
> > - Instead of a topic list arg, a pattern would be more powerful, and also
> > fit better with the other tools (e.g. how the kafka-topics tool works).
> >
> > Regards,
> > Daniel
> >
>

Re: [DISCUSS] KIP-308: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimize the number of requests to server

[Manikumar]

Hi Daniel,

Thanks for working on this KIP.  Proposed changes looks good to me,

minor comments:
1. We use "command-config" option name in most of the cmdline tools to pass
config
properties file. We can use the same name here.

2. Not sure, if we need a separate option to pass an consumer property.
fewer options are better.

Thanks,
Manikumar

On Wed, Jun 24, 2020 at 8:53 PM Dániel Urbán  wrote:

> Hi,
>
> I see that this KIP turned somewhat inactive - I'd like to pick it up and
> work on it if it is okay.
> Part of the work is done, as switching to the Consumer API is already in
> trunk, but some functionality is still missing.
>
> I've seen the current PR and the discussion so far, only have a few things
> to add:
> - I like the idea of the topic-partition argument, it would be useful to
> filter down to specific partitions.
> - Instead of a topic list arg, a pattern would be more powerful, and also
> fit better with the other tools (e.g. how the kafka-topics tool works).
>
> Regards,
> Daniel
>

Re: [DISCUSS] KIP-308: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimize the number of requests to server

[Dániel Urbán]

Hi,

I see that this KIP turned somewhat inactive - I'd like to pick it up and
work on it if it is okay.
Part of the work is done, as switching to the Consumer API is already in
trunk, but some functionality is still missing.

I've seen the current PR and the discussion so far, only have a few things
to add:
- I like the idea of the topic-partition argument, it would be useful to
filter down to specific partitions.
- Instead of a topic list arg, a pattern would be more powerful, and also
fit better with the other tools (e.g. how the kafka-topics tool works).

Regards,
Daniel

Re: [DISCUSS] KIP-308: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimize the number of requests to server

[Stanislav Kozlovski]

Hey Arseniy,

Do you plan on continuing the work for this KIP?
I would gladly take over. I find myself needing the GetOffsetShell tool to
work on secured cluster time and time again, I had even implemented a
duplicate JIRA (KAFKA-7373
) about it once.

Thanks,
Stanislav

On Tue, Sep 25, 2018 at 3:09 PM Manikumar  wrote:

> Hi Arseniy,
>
> It is good have GetOffsetShell tool to work on secure clusters.  Passing
> additional configuration to the
> underlying KafkaConsumer is sufficient for the requirement. Can you revive
> this discussion thread?
> Let us know if you need any help.
>
> Thanks,
>
>
> On Mon, Jun 4, 2018 at 3:27 PM Viktor Somogyi 
> wrote:
>
> > Hi Arseniy,
> >
> > A few comments from me collected in points:
> >
> > 1. Usually the commands use "bootstrap-server" as in singular and not the
> > plural "bootstrap-servers", so I suggest to keep this functionality.
> > Otherwise their behavior is still the same.
> >
> > 2. kafka-get-offsets.sh --bootstrap-servers vm:9092 --topics AAA,ZZZ
> > --partitions 0,1
> > I think that we could try out one of the "--topics AAA:0-3,ZZZ:1" or
> > "--topic-partitions AAA:0-3,ZZZ:1" or "--tp AAA:0-3,ZZZ:1" notations so
> > that asking for specific topic-partitions (or intervals as in AAA) is
> more
> > concise. Perhaps we could enable using your originally proposed and one
> of
> > these. What do you think?
> > Also I would be interested in what others think about this notation. I'd
> > find it useful as we often refer to partitions as "topic-partition" and
> > this might strengthen this kind of thinking.
> >
> > Viktor
> >
> >
> > On Fri, Jun 1, 2018 at 11:32 PM, Arseniy Tashoyan 
> > wrote:
> >
> > > Hi,
> > >
> > > I have just created KIP that proposes enhancements to GetOffsetShell
> > > command line tool.
> > > KIP:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 308%3A+GetOffsetShell%3A+new+KafkaConsumer+API%2C+support+
> > > for+multiple+topics%2C+minimize+the+number+of+requests+to+server
> > >
> > > PR: https://github.com/apache/kafka/pull/3051
> > >
> > > Suggestions are welcome.
> > >
> > > Regards,
> > > Arseniy
> > >
> >
>


-- 
Best,
Stanislav

Re: [DISCUSS] KIP-308: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimize the number of requests to server

[Manikumar]

Hi Arseniy,

It is good have GetOffsetShell tool to work on secure clusters.  Passing
additional configuration to the
underlying KafkaConsumer is sufficient for the requirement. Can you revive
this discussion thread?
Let us know if you need any help.

Thanks,


On Mon, Jun 4, 2018 at 3:27 PM Viktor Somogyi 
wrote:

> Hi Arseniy,
>
> A few comments from me collected in points:
>
> 1. Usually the commands use "bootstrap-server" as in singular and not the
> plural "bootstrap-servers", so I suggest to keep this functionality.
> Otherwise their behavior is still the same.
>
> 2. kafka-get-offsets.sh --bootstrap-servers vm:9092 --topics AAA,ZZZ
> --partitions 0,1
> I think that we could try out one of the "--topics AAA:0-3,ZZZ:1" or
> "--topic-partitions AAA:0-3,ZZZ:1" or "--tp AAA:0-3,ZZZ:1" notations so
> that asking for specific topic-partitions (or intervals as in AAA) is more
> concise. Perhaps we could enable using your originally proposed and one of
> these. What do you think?
> Also I would be interested in what others think about this notation. I'd
> find it useful as we often refer to partitions as "topic-partition" and
> this might strengthen this kind of thinking.
>
> Viktor
>
>
> On Fri, Jun 1, 2018 at 11:32 PM, Arseniy Tashoyan 
> wrote:
>
> > Hi,
> >
> > I have just created KIP that proposes enhancements to GetOffsetShell
> > command line tool.
> > KIP:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 308%3A+GetOffsetShell%3A+new+KafkaConsumer+API%2C+support+
> > for+multiple+topics%2C+minimize+the+number+of+requests+to+server
> >
> > PR: https://github.com/apache/kafka/pull/3051
> >
> > Suggestions are welcome.
> >
> > Regards,
> > Arseniy
> >
>

Re: [DISCUSS] KIP-308: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimize the number of requests to server

[Viktor Somogyi]

Hi Arseniy,

A few comments from me collected in points:

1. Usually the commands use "bootstrap-server" as in singular and not the
plural "bootstrap-servers", so I suggest to keep this functionality.
Otherwise their behavior is still the same.

2. kafka-get-offsets.sh --bootstrap-servers vm:9092 --topics AAA,ZZZ
--partitions 0,1
I think that we could try out one of the "--topics AAA:0-3,ZZZ:1" or
"--topic-partitions AAA:0-3,ZZZ:1" or "--tp AAA:0-3,ZZZ:1" notations so
that asking for specific topic-partitions (or intervals as in AAA) is more
concise. Perhaps we could enable using your originally proposed and one of
these. What do you think?
Also I would be interested in what others think about this notation. I'd
find it useful as we often refer to partitions as "topic-partition" and
this might strengthen this kind of thinking.

Viktor


On Fri, Jun 1, 2018 at 11:32 PM, Arseniy Tashoyan 
wrote:

> Hi,
>
> I have just created KIP that proposes enhancements to GetOffsetShell
> command line tool.
> KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 308%3A+GetOffsetShell%3A+new+KafkaConsumer+API%2C+support+
> for+multiple+topics%2C+minimize+the+number+of+requests+to+server
>
> PR: https://github.com/apache/kafka/pull/3051
>
> Suggestions are welcome.
>
> Regards,
> Arseniy
>