[
https://issues.apache.org/jira/browse/FLINK-32957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Xia updated FLINK-32957:
----------------------------
Description:
Timer trigger lag denotes the gap between the actual trigger timestamp and the
expected trigger timestamp (registered timestamp to `TimeService`). This metric
can aid users to find out whether there is a backlog of timers.
The backlog of timers may affect downstream data processing. Users customize
the trigger logic, which may interact with downstream data processing. For
example, a trigger logic can inject some records to downstream operators. The
backlog of timers blocks the record injection.
On the other side, The backlog of timers makes jobs unstable. Timers are used
by window operators, which leverage a timer to remove the window state of a
triggered window. The backlog of timers blocks data removal, and the state size
may grow unexpectedly large. The large state size affects the performance of
state-backend. In cloud-native environment, a k8s pod is prone to reach local
disk limit due to large state files (RocksDB SST).
Currently, users are hard to observe the backlog of timers. As far as I known,
heap dump is the only way to learn the backlog of timers. Thus, users cannot
notice the backlog of timers in time. FLINK-32954
(https://issues.apache.org/jira/browse/FLINK-32954) exposes number of heap
timers, but is not suitable for RocksDB timer due to performance loss.
Compare with FLINK-32954, timer trigger lag is much more lightweight for
RocksDB timer.
* Reason 1: Timer trigger lag does not affect timer registering.
* Reason 2: The effect on timer triggering is limited. Timer registering is a
hot code-path, while timer triggering is much colder. In general, the trigger
interval is tens of second, and the timer trigger code-path is invoked every
tens of second. Thus, the addition of timer trigger lag calculation has little
performance overhead.
> Add current timer trigger lag to metrics
> ----------------------------------------
>
> Key: FLINK-32957
> URL: https://issues.apache.org/jira/browse/FLINK-32957
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics, Runtime / State Backends
> Reporter: Rui Xia
> Priority: Minor
>
> Timer trigger lag denotes the gap between the actual trigger timestamp and
> the expected trigger timestamp (registered timestamp to `TimeService`). This
> metric can aid users to find out whether there is a backlog of timers.
> The backlog of timers may affect downstream data processing. Users customize
> the trigger logic, which may interact with downstream data processing. For
> example, a trigger logic can inject some records to downstream operators. The
> backlog of timers blocks the record injection.
> On the other side, The backlog of timers makes jobs unstable. Timers are used
> by window operators, which leverage a timer to remove the window state of a
> triggered window. The backlog of timers blocks data removal, and the state
> size may grow unexpectedly large. The large state size affects the
> performance of state-backend. In cloud-native environment, a k8s pod is prone
> to reach local disk limit due to large state files (RocksDB SST).
> Currently, users are hard to observe the backlog of timers. As far as I
> known, heap dump is the only way to learn the backlog of timers. Thus, users
> cannot notice the backlog of timers in time. FLINK-32954
> (https://issues.apache.org/jira/browse/FLINK-32954) exposes number of heap
> timers, but is not suitable for RocksDB timer due to performance loss.
> Compare with FLINK-32954, timer trigger lag is much more lightweight for
> RocksDB timer.
> * Reason 1: Timer trigger lag does not affect timer registering.
> * Reason 2: The effect on timer triggering is limited. Timer registering is
> a hot code-path, while timer triggering is much colder. In general, the
> trigger interval is tens of second, and the timer trigger code-path is
> invoked every tens of second. Thus, the addition of timer trigger lag
> calculation has little performance overhead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
