Re: KafkaIo Metrics
KafkaIO#commitOffsetsInFinalize[1] is likely what you want if you want to see Kafka's view of how the pipeline is consuming from it since the pipeline will ensure that offsets are committed as the pipeline has guaranteed to ingest the data. I would suggested to use pipeline level concepts and metrics so have you considered looking at pipeline level metrics like: * PCollection elements processed/size instead of bytes-consumed-rate * watermark lag / processing time lag instead of records-lag Obviously if your trying to dig down into an existing problem then it sure does make sense to look at Kafka level metrics if pipeline level metrics are tell you that there is a problem in the part of the pipeline containing Kafka. 1: https://beam.apache.org/releases/javadoc/2.44.0/org/apache/beam/sdk/io/kafka/KafkaIO.Read.html#commitOffsetsInFinalize-- On Fri, Jan 20, 2023 at 8:28 AM Alexey Romanenko wrote: > IIRC, we don’t expose any Kafka Consumer metrics, so I’m afraid, that > there is no easy way to get them in a Beam pipeline. > > — > Alexey > > On 18 Jan 2023, at 21:43, Lydian wrote: > > Hi, > I know that Beam KafkaIO doesn't use the native kafka offset, and > therefore I cannot use kafka metrics directly. > > Wondering what would be the right way to expose those metrics of my > KafkaIO pipeline? > Things I am interested includes: > >- bytes-consumed-rate >- fetch-latency-avg >- records-lag >- commit-rate > > consumer lagWondering how people get these metrics or instead of doing > this? or we should just enable `commit_offset_in_finalize` and then use the > Kafka metrics directly? > > also wondering if there's anything to notice when enabling the > commit_offset_in_finalize? Thanks! > > Sincerely, > Lydian Lee > > >
Re: KafkaIo Metrics
IIRC, we don’t expose any Kafka Consumer metrics, so I’m afraid, that there is no easy way to get them in a Beam pipeline. — Alexey > On 18 Jan 2023, at 21:43, Lydian wrote: > > Hi, > I know that Beam KafkaIO doesn't use the native kafka offset, and therefore I > cannot use kafka metrics directly. > > Wondering what would be the right way to expose those metrics of my KafkaIO > pipeline? > Things I am interested includes: > bytes-consumed-rate > fetch-latency-avg > records-lag > commit-rate > consumer lagWondering how people get these metrics or instead of doing this? > or we should just enable `commit_offset_in_finalize` and then use the Kafka > metrics directly? > > also wondering if there's anything to notice when enabling the > commit_offset_in_finalize? Thanks! > > Sincerely, > Lydian Lee >
Re: KafkaIo Metrics
/cc @John Casey - who might be able to help. On Wed, Jan 18, 2023 at 12:43 PM Lydian wrote: > Hi, > I know that Beam KafkaIO doesn't use the native kafka offset, and > therefore I cannot use kafka metrics directly. > > Wondering what would be the right way to expose those metrics of my > KafkaIO pipeline? > Things I am interested includes: > >- bytes-consumed-rate >- fetch-latency-avg >- records-lag >- commit-rate > > consumer lagWondering how people get these metrics or instead of doing > this? or we should just enable `commit_offset_in_finalize` and then use the > Kafka metrics directly? > > also wondering if there's anything to notice when enabling the > commit_offset_in_finalize? Thanks! > > Sincerely, > Lydian Lee > >
KafkaIo Metrics
Hi, I know that Beam KafkaIO doesn't use the native kafka offset, and therefore I cannot use kafka metrics directly. Wondering what would be the right way to expose those metrics of my KafkaIO pipeline? Things I am interested includes: - bytes-consumed-rate - fetch-latency-avg - records-lag - commit-rate consumer lagWondering how people get these metrics or instead of doing this? or we should just enable `commit_offset_in_finalize` and then use the Kafka metrics directly? also wondering if there's anything to notice when enabling the commit_offset_in_finalize? Thanks! Sincerely, Lydian Lee
