[
https://issues.apache.org/jira/browse/KUDU-3048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17040629#comment-17040629
]
ASF subversion and git services commented on KUDU-3048:
-------------------------------------------------------
Commit 7d9d7009ac288327a54ea694d171f8c2ad23165d in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=7d9d700 ]
[clock] KUDU-3048 add builtin-NTP-specific metrics
Added metrics specific to the built-in NTP client:
* info : difference between tracked true time and local wallclock
* debug: most recently computed true time
* debug: histogram on the maximum error of the computed true time
For testing, I manually ran kudu-master and verified the newly
introduced metrics are present and their values make sense.
I also re-visited the level of recently introduced metrics in the
HybridClock class.
Change-Id: Ia0a13d58c6e5da50ef7a98ae73de267ec4eab491
Reviewed-on: http://gerrit.cloudera.org:8080/15249
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <[email protected]>
> Add time/clock synchronization metrics
> --------------------------------------
>
> Key: KUDU-3048
> URL: https://issues.apache.org/jira/browse/KUDU-3048
> Project: Kudu
> Issue Type: Improvement
> Components: clock, master, tserver
> Reporter: Alexey Serbin
> Assignee: Alexey Serbin
> Priority: Major
> Labels: clock
>
> For better visibility, it would be great to add metrics reflecting time/clock
> synchronization parameters:
> * the stats on the max_error sampled while reading the underlying clock
> * the stats on time intervals when the underlying clock was extrapolated
> instead of using the actual readings: number of such intervals and stats on
> the interval duration
> * whether hybrid clock timestamps are generated using interpolated clock
> readings instead of real ones
> * if using the {{built-in}} time source:
> ** difference between tracked true time and local wallclock
> ** most recently computed true time
> ** the stats on the maximum error of the computed true time
> As for the rationale behind the new metrics:
> * max_error shows how far the clock is from the true time, and maybe it's
> time to use other set of NTP servers or instead increase the
> {{\-\-max_clock_sync_error_usec}} flag value
> * presence of the extrapolation intervals for the hybrid clock signals about
> periods of non-availability for NTP servers, and possible action would be
> re-visiting the set of NTP servers
> * if hybrid timestamps are being extrapolated for some time, Kudu masters and
> tablet servers might crash if the clock errors eventually goes beyond the
> configured threshold: it's time to start troubleshooting the issue to avoid
> possible non-availability of the cluster
> The new metrics can be used for monitoring and alerting, allowing for
> pro-active maintenance of a Kudu cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
