Good point. We should probably have a maximum number of results like
1000 or something. That can go in the request RPC as well...
Cheers,
Colin
On Fri, Jul 13, 2018, at 18:15, Ted Yu wrote:
> bq. describe topics by a regular expression on the server side
>
> Should caution be taken if the regex doesn't filter ("*") ?
>
> Cheers
>
> On Fri, Jul 13, 2018 at 6:02 PM Colin McCabe
> <cmcc...@apache.org> wrote:>
> > As Jason wrote, this won't scale as the number of partitions
> > increases.> > We already have users who have tens of thousands of topics, or
> > more. If> > you multiply that by 100x over the next few years, you end up
> > with
> > this API> > returning full information about millions of topics, which
> > clearly
> > doesn't> > work.
> >
> > We discussed this a lot in the original KIP-117 DISCUSS thread
> > which added> > the Java AdminClient. ListTopics and DescribeTopics were
> > deliberately kept> > separate because we understood that eventually a
> > single RPC would
> > not be> > able to return information about all the topics in the cluster.
> > So
> > I have> > to vote -1 for this proposal as it stands.
> >
> > I do agree that adding a way to describe topics by a regular
> > expression on> > the server side would be very useful. This would also fix
> > a major
> > scalability problem we have now, which is that when
> > subscribing via a> > regular expression, clients need to fetch the full
> > list of all
> > topics in> > the cluster and filter locally.
> >
> > I think a regular expression library like re2 would be ideal
> > for this> > purpose. re2 is standardized and language-agnostic (it's not
> > tied
> > only to> > Java). In contrast, Java regular expression change with
> > different
> > releases> > of the JDK (there were some changes in java 8, for example).
> > Also, re2> > regular expressions are linear time, never exponential time.
> > See
> > https://github.com/google/re2j
> >
> > regards,
> > Colin
> >
> >
> > On Fri, Jul 13, 2018, at 05:00, Andras Beni wrote:
> > > The KIP looks good to me.
> > > However, if there is willingness in the community to work on
> > > metadata> > > request with patterns, the feature proposed here and
> > > filtering by
> > > '*' or> > > '.*' would be redundant.
> > >
> > > Andras
> > >
> > >
> > >
> > > On Fri, Jul 13, 2018 at 12:38 AM Jason Gustafson
> > > <ja...@confluent.io>> > wrote:
> > >
> > > > Hey Manikumar,
> > > >
> > > > As Kafka begins to scale to larger and larger numbers of
> > topics/partitions,
> > > > I'm a little concerned about the scalability of APIs such as
> > > > this. The> > API
> > > > looks benign, but imagine you have have a few million
> > > > partitions. We> > > > already expose similar APIs in the producer and
> > > > consumer, so
> > > > probably> > not
> > > > much additional harm to expose it in the AdminClient, but it
> > > > would be> > nice
> > > > to put a little thought into some longer term options. We should
> > > > be> > giving
> > > > users an efficient way to select a smaller set of the topics
> > > > they are> > > > interested in. We have always discussed adding some
> > > > filtering
> > > > support> > to
> > > > the Metadata API. Perhaps now is a good time to reconsider this?
> > > > We now> > > > have a convention for wildcard ACLs, so perhaps we can do
> > > > something> > > > similar. Full regex support might be ideal given the
> > > > consumer's> > > > subscription API, but that is more challenging. What
> > > > do you
> > > > think?> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Thu, Jul 12, 2018 at 2:35 PM, Harsha <ka...@harsha.io> wrote:> > > >
> > > > > Very useful. LGTM.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Thu, Jul 12, 2018, at 9:56 AM, Manikumar wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I have created a KIP to add describe all topics API to
> > > > > > AdminClient> > .
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 327%3A+Add+describe+all+topics+API+to+AdminClient
> > > > > >
> > > > > > Please take a look.
> > > > > >
> > > > > > Thanks,
> > > > >
> > > >
> >
