Re: How to know the begin offset of a partition

2018-07-31 Thread Gabriele Paggi
. It doesn't seem possible to provide the consumer.config like we do > for kafka-console-consumer.sh. > > On Tue, Jul 31, 2018 at 12:19 PM Gabriele Paggi > wrote: > > > Hi Pierre, > > > > You can use kafka.tools.GetOffsetShell, which will return the earliest &

Re: How to know the begin offset of a partition

2018-07-31 Thread Gabriele Paggi
Hi Pierre, You can use kafka.tools.GetOffsetShell, which will return the earliest offset for each partition in a given topic. If you want to see the latest available ones, change "--time -2" with "--time -1" gpaggi@kafkalog001:~$ kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list

Re: Total Volume metrics of Kafka

2019-01-17 Thread Gabriele Paggi
On Thu, 17 Jan 2019 at 00:44, Peter Bukowinski wrote: > On each broker, we have a process (scheduled with cron) that polls the > kafka jmx api every 60 seconds. It sends the metrics data to graphite ( > https://graphiteapp.org). We have graphite configured as a data source > for grafana

Re: Metrics for Topic/Partition size

2020-02-24 Thread Gabriele Paggi
Hi Richard, If you are running Kafka > 1.0.0 the information you are looking for is exposed by the Admin API, describeLogDirs method, or via CLI with kafka-log-dirs.sh. The metrics are also exposed by the dropwizard metrics reporter, e.g. for Graphite:

Re: Metrics for Topic/Partition size

2020-02-26 Thread Gabriele Paggi
partition: "$3" > > what do you think? > > On Mon, Feb 24, 2020 at 4:05 PM Gabriele Paggi > wrote: > > > > Hi Richard, > > > > If you are running Kafka > 1.0.0 the information you are looking for > > is exposed by the Admin API, descri

Re: Metrics for Topic/Partition size

2020-02-26 Thread Gabriele Paggi
Hi Richard, Yes, it's the size in bytes for all log segments for a given topic/partition on a given broker, without the index files: [gpaggi@kafkalog001 ~]$ kafka-log-dirs.sh --bootstrap-server $(hostname -f):9092 --describe --broker-list 1 --topic-list access_logs | tail -n+3 | jq

Re: Initializing a consumer groups committed offset with -1

2020-02-04 Thread Gabriele Paggi
Hi Sean, Unless this has changed, this is the expected behavior when there's no offset associated with a topic-partition under a given consumer group, in which case the Kafka broker will return -1: