KafkaIO#commitOffsetsInFinalize[1] is likely what you want if you want to
see Kafka's view of how the pipeline is consuming from it since the
pipeline will ensure that offsets are committed as the pipeline has
guaranteed to ingest the data.
I would suggested to use pipeline level concepts and
IIRC, we don’t expose any Kafka Consumer metrics, so I’m afraid, that there is
no easy way to get them in a Beam pipeline.
—
Alexey
> On 18 Jan 2023, at 21:43, Lydian wrote:
>
> Hi,
> I know that Beam KafkaIO doesn't use the native kafka offset, and therefore I
> cannot use kafka metrics
/cc @John Casey - who might be able to help.
On Wed, Jan 18, 2023 at 12:43 PM Lydian wrote:
> Hi,
> I know that Beam KafkaIO doesn't use the native kafka offset, and
> therefore I cannot use kafka metrics directly.
>
> Wondering what would be the right way to expose those metrics of my
>
Hi,
I know that Beam KafkaIO doesn't use the native kafka offset, and therefore
I cannot use kafka metrics directly.
Wondering what would be the right way to expose those metrics of my KafkaIO
pipeline?
Things I am interested includes:
- bytes-consumed-rate
- fetch-latency-avg
-
