Hey Dong, I have observed two usecases: a) monitoring (if infrastructure does not have any control over clients) - if I only provide Kafka brokers, but I am not aware of clients, I cannot say if anyone would even consume the data - this can be somehow compared to listing existing consumers/connections in RabbitMQ queues (theoretically, someone use some kind of network interface monitoring for an estimate) b) cleanup of unused topics - when Kafka is used as a messaging hub, with dynamically created topics, we can reach a very large number of them. Removing the ones that still have active consumers causes odd side effects (e.g. the consumers are again reading from offset 0) and bugs ( https://issues.apache.org/jira/browse/KAFKA-6438 ).
[fwiw, b) also can be solved by having topic TTL (in a fashion similar to e.g. RabbitMQ) - I will be submitting a relevant KIP soon] Yours sincerely, Adam Kotwasinski 2018-05-18 19:49 GMT+01:00 Dong Lin <lindon...@gmail.com>: > Hey Adam, > > Got it. Thanks for the explanation. Could you also explain the use-case for > knowing whether a topic with no new messages has active consumers? Just > want to make sure that the use-case is reasonable and can not be > accomplished with the existing available approach. > > Thanks, > Dong > > > On Fri, May 18, 2018 at 11:10 AM, Adam Kotwasinski < > adam.kotwasin...@gmail.com> wrote: > >> Hi Dong, >> >> Unfortunately this does not work when there are no messages, but >> consumers are still running. >> BytesOutPerSec increases only when the messages are being delivered to >> the consumer. >> But you could have a situation when consumer is already at the end of >> log & noone is going to produce any more messages. >> In such a case you would be getting fetch requests, and nothing would >> be sent out. >> >> Trivial to replicate with `kafka-console-consumer.sh >> --bootstrap-server localhost:9092 --topic idontexist` >> >> Best regards, >> Adam >> >> 2018-05-18 18:49 GMT+01:00 Dong Lin <lindon...@gmail.com>: >> > Hey Adam, >> > >> > Thanks for the KIP. We currently already have the per-topic byte-out-rate >> > (not including replication traffic) with MBean >> > path kafka.server:name=BytesOutPerSec,type=BrokerTopicMetrics,topic=*. >> > Though this is not the FetchRequest rate, it seems to address the >> > motivation of the KIP by telling Kafka cluster operator whether consumers >> > are actively reading from a topic. Does that address the problem? >> > >> > Thanks, >> > Dong >> > >> > >> > >> > On Fri, May 18, 2018 at 8:55 AM, Adam Kotwasinski < >> > adam.kotwasin...@gmail.com> wrote: >> > >> >> Hello, >> >> >> >> There haven't been many comments regarding this KIP (not surprising as >> >> the code change is ~10 lines long). Would it be okay to start the >> >> vote? >> >> >> >> https://cwiki.apache.org/confluence/pages/viewpage.action? >> pageId=80452537 >> >> >> >> Best regards, >> >> Adam Kotwasinski >> >> >> >> On Wed, May 2, 2018 at 6:49 AM, Adam Kotwasinski < >> >> adam.kotwasin...@gmail.com >> >> > wrote: >> >> >> >> > Hello, >> >> > >> >> > I have created a KIP to add some additional metrics re number of fetch >> >> > requests made to brokers - >> >> > https://cwiki.apache.org/confluence/pages/viewpage. >> >> action?pageId=80452537 >> >> > >> >> > Feedback and suggestions are welcome. >> >> > >> >> > Best regards, >> >> > Adam Kotwasinski >> >> > >> >> >>
