Todd Lipcon has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8531
Change subject: KUDU-2209. HybridClock doesn't handle changes in STA_NANO flag ...................................................................... KUDU-2209. HybridClock doesn't handle changes in STA_NANO flag Users have occasionally reported spurious crashes due to Kudu thinking that another node has a time stamp from the future. After some debugging I realized that the issue is that we currently capture the flag 'STA_NANO' from the kernel only at startup. This flag indicates whether the kernel's sub-second timestamp is in nanoseconds or microseconds. We initially assumed this was a static property of the kernel. However it turns out that this flag can get toggled at runtime by ntp in certain circumstances. Given this, it was possible for us to interpret a number of nanoseconds as if it were microseconds, resulting in a timestamp up to 1000 seconds in the future. This patch changes the SystemNtp time source to always use the 'adjtime' call to fetch the clock, and looks at the STA_NANO flag on every such call rather than only at startup. I checked the source for the ntp_gettime call that we used to use, and it turns out it was implemented in terms of the same adjtime() call, so there should be no performance penalty in this change. This patch doesn't include tests since it's based on some kernel functionality. However, I was able to test it as follows: - wrote a simple program to print the time once a second - stopped ntp, ran chrony, stopped chrony, and started ntpd -- this has the side effect of clearing STA_NANO - waited 10 minutes or so, and eventually ntp reset back to STA_NANO -- this caused my test program to start printing incorrect timestamps as it was interpreting nanosecond values from the kernel as if they were microseconds ------------------ Backport notes ------------------- The clock module was somewhat refactored for Kudu 1.5 and later, so the cherry-pick was a fairly manual process. Essentially ported over the same fix rather than trying to cherry-pick and resolve conflicts. Reviewed-on: http://gerrit.cloudera.org:8080/8450 Reviewed-by: Alexey Serbin <aser...@cloudera.com> Tested-by: Alexey Serbin <aser...@cloudera.com> (cherry-picked from commit 10f6164b1217e0299bcfedc061d2c57581c389bd) Change-Id: I4583a4d5ef26da791c9f6cba561a9aa2f22d3dd6 --- M src/kudu/server/hybrid_clock.cc M src/kudu/server/hybrid_clock.h 2 files changed, 23 insertions(+), 51 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/31/8531/1 -- To view, visit http://gerrit.cloudera.org:8080/8531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.3.x Gerrit-MessageType: newchange Gerrit-Change-Id: I4583a4d5ef26da791c9f6cba561a9aa2f22d3dd6 Gerrit-Change-Number: 8531 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon <t...@apache.org>
