This is an automated email from the ASF dual-hosted git repository.
alexey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 96a6a8b [docs] add information on clock and time source
96a6a8b is described below
commit 96a6a8b91bd069ed5ebe3dfd4eb8179708eacda9
Author: Alexey Serbin <[email protected]>
AuthorDate: Wed Jan 19 23:26:54 2022 -0800
[docs] add information on clock and time source
Change-Id: Iaf887caa8ce1b0dcd2f0ff81762ee55094a8c773
Reviewed-on: http://gerrit.cloudera.org:8080/18163
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <[email protected]>
---
docs/configuration.adoc | 109 ++++++++++++++++++++++++++++++++++++++++++++++
docs/troubleshooting.adoc | 2 +-
2 files changed, 110 insertions(+), 1 deletion(-)
diff --git a/docs/configuration.adoc b/docs/configuration.adoc
index 02ccabe..7e39f0c 100644
--- a/docs/configuration.adoc
+++ b/docs/configuration.adoc
@@ -54,6 +54,115 @@ Take care when configuring undocumented flags, as not every
possible
configuration has been tested, and undocumented options are not guaranteed to
be
maintained in future releases.
+[[clock_and_time_source]]
+=== Configuring Clock and Time Source
+Kudu relies on timestamps generated by its clock implementation for the MVCC
+and for providing consistency guarantees when processing write and read
+requests. Aside from the test-only mock clock, Kudu has two different clock
+implementations: one is based on logical time and the other is based on
+so-called hybrid time. The former is a plain Lamport clock, the latter
+is a combination of the node's system clock and a Lamport clock. Below,
+the former is referred to as `LogicalClock` and the latter as `HybridClock`.
+
+Using the `HybridClock` implementation is a must for any production-grade, POC,
+and other regular Kudu deployments: that's why `--use_hybrid_clock` is set
+`true` by default. Setting the flag to `false` makes Kudu servers use the
+`LogicalClock` implementation: running with such a clock implementation is
+acceptable only in the context of running specifically crafted test scenarios
+in Kudu development environment.
+
+WARNING: Setting `--use_hybrid_clock=false` is strongly discouraged in any
+production-grade deployment since that could introduce out-of-control latency
+and not-quite-expected behavior, especially when working with multiple tables
+in a multi-node Kudu cluster.
+
+To provide better accuracy for multi-node cluster deployments where each node
+maintains its own system clock, the `HybridClock` implementation requires each
+node's system clock to be synchronized by NTP.
+
+NOTE: Setting `--time_source=system_unsync` removes the requirement for the
+node's system clock to be synchronized by NTP -- this allows users to run test
+clusters on a single node where there is only one clock used by all Kudu
+servers. Setting `--time_source=system_unsync` is strongly discouraged in any
+multi-node Kudu cluster, unless system clocks of all Kudu nodes are guaranteed
+to always be synchronized with each other.
+
+For Kudu masters and tablet servers, there are two options to make the
+`HybridClock` implementation use a clock synchronized by NTP:
+
+- Ensure that the system clock of the Kudu node is synchronized with reference
+ servers using an NTP daemon running on the node. Usually, the NTP daemon is
+ a part of the node's OS distribution. As of Kudu 1.12.0 and newer, both
+ `ntpd` and `chronyd` are supported. Prior Kudu versions were tested only
+ with `ntpd`, but might work just fine with `chronyd` as well if `chronyd` is
+ configured as recommended by the
+ link:troubleshooting.html#chronyd[chronyd configuration tips for Kudu].
+- Make Kudu servers maintain their own local clock, synchronizing it with
+ reference NTP servers. For that, Kudu servers use their built-in NTP client.
+ This option is available in Kudu 1.11.0 and newer versions.
+
+The latter option is provided as a last resort for deployments where properly
+configuring NTP daemons at every node of a Kudu cluster is not feasible for
+some reason and to simplify Kudu deployments in public cloud environments such
+as EC2 and GCP. For on-prem deployments, it's still recommended to use the
+former option since the current implementation of the Kudu built-in NTP client
+might not be as robust as the battle-tested `ntpd` and `chronyd` system
+NTP daemons.
+
+To switch between these two options above, use the `--time_source` flag:
+
+- Setting `--time_source=system` makes the `HybridClock` rely on the node's
+ system clock.
+- Setting `--time_source=builtin` turns on the built-in NTP client in
+ Kudu masters and tablet servers. Use the `--builtin_ntp_servers` flag to
+ customize the set of reference NTP servers for the built-in NTP client: the
+ value is expected to be a comma-separated list.
+
+NOTE: The default setting for the `--builtin_ntp_servers` flag might require
+access to the NTP servers hosted by the
+link:https://www.ntppool.org/[NTP Pool Project].
+
+If deploying a Kudu cluster in AWS/EC2 or GCE/GCP public clouds, it might make
+sense to set `--time_source=auto` for all Kudu masters and tablet servers in
+the cluster. In this context, setting `--time_source=auto` leads to the
+following:
+
+- Upon every start, a Kudu server runs the auto-detection procedure to
+ determine the type of the cloud environment it runs at.
+- If the procedure of the cloud type auto-detection completes successfully,
+ the Kudu server starts using its built-in NTP client to synchronize with the
+ NTP server provided by the cloud environment (see the appropriate
+ documentation for
+ link:https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html[EC2]
+ and link:https://cloud.google.com/compute/docs/instances/configure-ntp[GCP]
+ correspondingly).
+
+NOTE: Running a Kudu server with `--time_source=auto` in cloud environments
+other than EC2 and GCP, or when the cloud type auto-detection fails, makes
+the Kudu server fall back to using the built-in NTP client with the list
+of NTP servers as specified by the `--builtin_ntp_servers` flag, unless it's
+empty or otherwise unparsable. When `--builtin_ntp_servers` is set to an empty
+list and the cloud type auto-detection fails, the Kudu server runs as if it
+were configured with the `system` time source if the OS/platform supports the
+`get_ntptime()` API. Finally, the catch-all case is `system_unsync` for the
+time source. As already mentioned, the `system_unsync` time source is targeted
+for development-only platforms or single-node-runs-it-all proof-of-concept
+Kudu clusters.
+
+The `kudu cluster ksck` CLI utility reports the configured and the effective
+time source for every Kudu master and tablet server in a cluster. The list of
+the NTP servers for the built-in client is reported as well when the effective
+time source is `builtin`. The utility is also able to show the difference in
+settings of the related time source flags and warn operators if a discrepancy
+is detected. In addition, the information on the configured and effective time
+source is reported by the embedded Web server in the `Time Source` panel at
+the `/config` page.
+
+NOTE: Changing the value of the `--time_source` flag implies restarting a Kudu
+server. Keep the time source the same for all master and tablet servers in
+a Kudu cluster. If using the built-in NTP Kudu client, make sure to use
+the same list of reference NTP servers for every Kudu server in a cluster.
+
[[directory_configuration]]
=== Directory Configurations
Every Kudu node requires the specification of directory flags. The
diff --git a/docs/troubleshooting.adoc b/docs/troubleshooting.adoc
index 8433a16..df42be7 100644
--- a/docs/troubleshooting.adoc
+++ b/docs/troubleshooting.adoc
@@ -348,7 +348,7 @@ $ ntpq -nc opeers
TIP: Both `lpeers` and `opeers` may be helpful as `lpeers` lists refid and
jitter, while `opeers` lists clock dispersion.
-
+[[chronyd]]
===== Installing And Running `chronyd`
Kudu has been tested and is supported on machines whose local clock is
synchronized with NTP using `chronyd` version 3.2 and newer.
