[kudu-CR](branch-1.4.x) KUDU-2209. HybridClock doesn't handle changes in STA NANO flag
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8530 ) Change subject: KUDU-2209. HybridClock doesn't handle changes in STA_NANO flag .. KUDU-2209. HybridClock doesn't handle changes in STA_NANO flag Users have occasionally reported spurious crashes due to Kudu thinking that another node has a time stamp from the future. After some debugging I realized that the issue is that we currently capture the flag 'STA_NANO' from the kernel only at startup. This flag indicates whether the kernel's sub-second timestamp is in nanoseconds or microseconds. We initially assumed this was a static property of the kernel. However it turns out that this flag can get toggled at runtime by ntp in certain circumstances. Given this, it was possible for us to interpret a number of nanoseconds as if it were microseconds, resulting in a timestamp up to 1000 seconds in the future. This patch changes the SystemNtp time source to always use the 'adjtime' call to fetch the clock, and looks at the STA_NANO flag on every such call rather than only at startup. I checked the source for the ntp_gettime call that we used to use, and it turns out it was implemented in terms of the same adjtime() call, so there should be no performance penalty in this change. This patch doesn't include tests since it's based on some kernel functionality. However, I was able to test it as follows: - wrote a simple program to print the time once a second - stopped ntp, ran chrony, stopped chrony, and started ntpd -- this has the side effect of clearing STA_NANO - waited 10 minutes or so, and eventually ntp reset back to STA_NANO -- this caused my test program to start printing incorrect timestamps as it was interpreting nanosecond values from the kernel as if they were microseconds -- Backport notes --- The clock module was somewhat refactored for Kudu 1.5 and later, so the cherry-pick was a fairly manual process. Essentially ported over the same fix rather than trying to cherry-pick and resolve conflicts. Reviewed-on: http://gerrit.cloudera.org:8080/8450 Reviewed-by: Alexey SerbinTested-by: Alexey Serbin (cherry-picked from commit 10f6164b1217e0299bcfedc061d2c57581c389bd) Change-Id: I4583a4d5ef26da791c9f6cba561a9aa2f22d3dd6 Reviewed-on: http://gerrit.cloudera.org:8080/8530 Reviewed-by: Alexey Serbin Tested-by: Kudu Jenkins --- M src/kudu/server/hybrid_clock.cc M src/kudu/server/hybrid_clock.h 2 files changed, 23 insertions(+), 51 deletions(-) Approvals: Alexey Serbin: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.4.x Gerrit-MessageType: merged Gerrit-Change-Id: I4583a4d5ef26da791c9f6cba561a9aa2f22d3dd6 Gerrit-Change-Number: 8530 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon
[kudu-CR](branch-1.4.x) KUDU-2209. HybridClock doesn't handle changes in STA NANO flag
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/8530 ) Change subject: KUDU-2209. HybridClock doesn't handle changes in STA_NANO flag .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.4.x Gerrit-MessageType: comment Gerrit-Change-Id: I4583a4d5ef26da791c9f6cba561a9aa2f22d3dd6 Gerrit-Change-Number: 8530 Gerrit-PatchSet: 1 Gerrit-Owner: Todd LipconGerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins Gerrit-Comment-Date: Mon, 13 Nov 2017 20:15:58 + Gerrit-HasComments: No
[kudu-CR](branch-1.4.x) KUDU-2209. HybridClock doesn't handle changes in STA NANO flag
Hello Alexey Serbin, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/8530 to review the following change. Change subject: KUDU-2209. HybridClock doesn't handle changes in STA_NANO flag .. KUDU-2209. HybridClock doesn't handle changes in STA_NANO flag Users have occasionally reported spurious crashes due to Kudu thinking that another node has a time stamp from the future. After some debugging I realized that the issue is that we currently capture the flag 'STA_NANO' from the kernel only at startup. This flag indicates whether the kernel's sub-second timestamp is in nanoseconds or microseconds. We initially assumed this was a static property of the kernel. However it turns out that this flag can get toggled at runtime by ntp in certain circumstances. Given this, it was possible for us to interpret a number of nanoseconds as if it were microseconds, resulting in a timestamp up to 1000 seconds in the future. This patch changes the SystemNtp time source to always use the 'adjtime' call to fetch the clock, and looks at the STA_NANO flag on every such call rather than only at startup. I checked the source for the ntp_gettime call that we used to use, and it turns out it was implemented in terms of the same adjtime() call, so there should be no performance penalty in this change. This patch doesn't include tests since it's based on some kernel functionality. However, I was able to test it as follows: - wrote a simple program to print the time once a second - stopped ntp, ran chrony, stopped chrony, and started ntpd -- this has the side effect of clearing STA_NANO - waited 10 minutes or so, and eventually ntp reset back to STA_NANO -- this caused my test program to start printing incorrect timestamps as it was interpreting nanosecond values from the kernel as if they were microseconds -- Backport notes --- The clock module was somewhat refactored for Kudu 1.5 and later, so the cherry-pick was a fairly manual process. Essentially ported over the same fix rather than trying to cherry-pick and resolve conflicts. Reviewed-on: http://gerrit.cloudera.org:8080/8450 Reviewed-by: Alexey SerbinTested-by: Alexey Serbin (cherry-picked from commit 10f6164b1217e0299bcfedc061d2c57581c389bd) Change-Id: I4583a4d5ef26da791c9f6cba561a9aa2f22d3dd6 --- M src/kudu/server/hybrid_clock.cc M src/kudu/server/hybrid_clock.h 2 files changed, 23 insertions(+), 51 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/8530/1 -- To view, visit http://gerrit.cloudera.org:8080/8530 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: branch-1.4.x Gerrit-MessageType: newchange Gerrit-Change-Id: I4583a4d5ef26da791c9f6cba561a9aa2f22d3dd6 Gerrit-Change-Number: 8530 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Alexey Serbin