Hi Xiuming, +1 on your idea. BTW, Flink also provides a debug tool to track the latency of records travelling through the system[1]. But you should note the following issue if enable the latency tracking. (1) It's a tool for debugging purposes because enabling latency metrics can significantly impact the performance ( worse if enable subtask granularity). (2) latency tracking metrics could only partially reflect latency, please see details explanation in document [1] (3) This feature is disabled by default.
So your proposal to add metric in source and sink sounds better. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/metrics/#latency-tracking xm lian <lian...@gmail.com> 于2021年7月1日周四 下午2:06写道： > Hello community, > > I would like to know how long it takes for an event to flow through the > whole Flink pipeline, that consumes from Kafka and sinks to Redis. > > My current idea is, for each event: > > 1. calculate a start_time in source (timestamp field of [metadata]( > https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/connectors/kafka.html#available-metadata) > or flink PROCTIME) > 2. calculate end_time in the sink (System.currentTimeMillis or flink > PROCTIME) > 3. push the (end_time - start_time) to Prometheus > > I wonder if flink provides a better and more native way of to calculate > the time spent in Flink? > > Thanks! > > Best, > Xiuming >
