GitHub user rmetzger opened a pull request:
https://github.com/apache/flink/pull/2386
[FLINK-3660] Measure latency and exposes them via a metric
This commit adds the initial runtime support for measuring latency of
records going through the system.
I therefore introduced a new StreamElement, called a LatencyMarker.
Similar to Watermarks, LatencyMarkers are emitted from the sources at an
configured interval. The default value for the interval is 2000 ms. The
emission of markers can be disabled by setting the interval to 0.
LatencyMarkers can not "overtake" regular elements. This ensures that the
measured latency approximates the end-to-end latency of regular stream elements.
Regular operators (excluding those participating in iterations) forward
latency markers if they are not a sink.
Operators with many outputs randomly select one to forward the maker to.
This ensures that every marker exists only once in the system, and that
repartition steps are not causing an explosion in the number of transferred
markers.
If an operator is a sink, it will maintain the last 512 latencies from each
known source instance.
The min/max/mean/p50/p95/p99 of each known source is reported using a
special LatencyGauge from the sink (every operator can be a sink, if it doesn't
have any outputs).
This commit does not visualize the latency in the web interface.
Also, there is currently no mechanism to ensure that the system clocks are
in-sync, so the latency measurements will be inaccurate if the hardware clocks
are not correct.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmetzger/flink flink3660-pr
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2386.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2386
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
