[
https://issues.apache.org/jira/browse/RATIS-2685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz-wo Sze updated RATIS-2685:
------------------------------
Component/s: Leader
> Follower gauges are stale after LogAppender restart
> ---------------------------------------------------
>
> Key: RATIS-2685
> URL: https://issues.apache.org/jira/browse/RATIS-2685
> Project: Ratis
> Issue Type: Bug
> Components: Leader
> Reporter: Janus Chow
> Assignee: Janus Chow
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {{LeaderStateImpl.restart(LogAppender)}} creates a new {{FollowerInfo}} for
> the same follower and calls {{LogAppenderMetrics.addFollowerGauges}} again.
> The old follower gauges are never removed:
> * {{follower_<id>_next_index}}
> * {{follower_<id>_match_index}}
> * {{follower_<id>_rpc_response_time}}
> {{RatisMetricRegistry.gauge(..)}} delegates to Dropwizard
> {{{}MetricRegistry.gauge(..){}}}, which returns the existing gauge instead of
> replacing it. So after a restart these gauges keep reading the stopped
> {{FollowerInfo}} and never change again.
> In Apache Ozone, this made a pipeline look like it was lagging more and more:
> {{appliedIndex - follower_next_index}} kept growing, even though all
> followers were in sync.
> Proposed fix: remove the follower gauges when senders are stopped and before
> re-registering them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
