Hi folks, I'm jumping in on an old thread. We've been running several 64-bit fully-virtualized Xen RHEL5 guests on a RHEL5 host, and have been having odd ntpd/time issues - system gets slowly more and more ahead, and we get lots of 'no servers reachable' in /var/log/messages, though the non-Xen hosts have no problems communicating with these timeservers.
After scouring the 'net for many hours, I found lots of things saying that Xen guests sync off the dom0 (host) clock.... but apparently that's only in the case of *paravirtualized* guests. It turns out that Xen fully virtualized systems grab the dom0's clocktime at boot, and then are responsible for their own time. I could only find one reference to this, though - http://tinyurl.com/256dud - referencing Novell's OES. I finally found this after much searching and references to a default sysctl setting of xen.independent_wallclock=0 which is only relevant for paravirtualized systems. :-/ That said - I'm still seeing a *lot* of synchronizations in /var/log/messages, and am suspecting that things are still not right. I'm seeing some references to tweaking the kernel HZ setting, but am hesitant to muck with that. Are there others running fully-virtualized systems that are having these sorts of time issues? ~ Hope Some details: Red Hat Enterprise Linux Server release 5.1 (Tikanga) Linux 2.6.18-53.el5 #1 SMP x86_64 x86_64 x86_64 GNU/Linux [EMAIL PROTECTED] ~]# cat /etc/ntp.conf driftfile /var/lib/ntp/drift restrict default notrust nomodify notrap restrict 127.0.0.1 # The next three groups of "peer" and "restrict" directives specify the # ITD Time Server addresses, and should not be changed. server 141.211.144.15 restrict 141.211.144.15 server 141.211.125.15 restrict 141.211.125.15 server us.pool.ntp.org # Fake server to allow for outages fudge 127.127.1.0 stratum 10 [EMAIL PROTECTED] ~]# ntpq -np remote refid st t when poll reach delay offset jitter ======================================================================== ====== 141.211.144.15 204.123.2.5 2 u 35 64 17 0.594 -1743.4 1315.84 141.211.125.15 141.211.1.32 3 u - 64 17 0.309 -2843.4 70.318 208.75.85.61 .STEP. 16 u 63 256 0 0.000 0.000 0.000 [EMAIL PROTECTED] ~]# Nov 26 08:20:54 hsg-vtest ntpd[29664]: ntpd [EMAIL PROTECTED] Mon Jun 4 15:13:12 UTC 2007 (1) Nov 26 08:20:54 hsg-vtest ntpd[29665]: precision = 2.000 usec Nov 26 08:20:54 hsg-vtest ntpd[29665]: Listening on interface wildcard, 0.0.0.0#123 Disabled Nov 26 08:20:54 hsg-vtest ntpd[29665]: Listening on interface wildcard, ::#123 Disabled Nov 26 08:20:54 hsg-vtest ntpd[29665]: Listening on interface lo, ::1#123 Enabled Nov 26 08:20:54 hsg-vtest ntpd[29665]: Listening on interface eth0, fe80::216:3eff:fe0a:468#123 Enabled Nov 26 08:20:54 hsg-vtest ntpd[29665]: Listening on interface lo, 127.0.0.1#123 Enabled Nov 26 08:20:54 hsg-vtest ntpd[29665]: Listening on interface eth0, 141.211.232.128#123 Enabled Nov 26 08:20:54 hsg-vtest ntpd[29665]: Listening on interface eth0:0, 141.211.232.129#123 Enabled Nov 26 08:20:54 hsg-vtest ntpd[29665]: kernel time sync status 0040 Nov 26 08:20:54 hsg-vtest ntpd[29665]: frequency initialized -308.293 PPM from /var/lib/ntp/drift Nov 26 08:25:10 hsg-vtest ntpd[29665]: synchronized to 141.211.125.15, stratum 3 Nov 26 07:56:48 hsg-vtest ntpd[29665]: time reset -1701.346214 s Nov 26 07:56:48 hsg-vtest ntpd[29665]: kernel time sync disabled 0001 Nov 26 08:01:26 hsg-vtest ntpd[29665]: synchronized to 141.211.125.15, stratum 3 Nov 26 08:07:55 hsg-vtest ntpd[29665]: synchronized to 141.211.144.15, stratum 2 Nov 26 08:09:32 hsg-vtest ntpd[29665]: no servers reachable Nov 26 08:15:58 hsg-vtest ntpd[29665]: synchronized to 141.211.144.15, stratum 2 Nov 26 08:24:30 hsg-vtest ntpd[29665]: synchronized to 141.211.125.15, stratum 3 Nov 26 08:25:05 hsg-vtest ntpd[29665]: time reset -2.394351 s Nov 26 08:30:01 hsg-vtest ntpd[29665]: synchronized to 141.211.125.15, stratum 3 Nov 26 08:39:43 hsg-vtest ntpd[29665]: time reset -2.424822 s Nov 26 08:49:00 hsg-vtest ntpd[29665]: synchronized to 141.211.144.15, stratum 2 Nov 26 08:58:38 hsg-vtest ntpd[29665]: synchronized to 141.211.125.15, stratum 3 Nov 26 08:59:00 hsg-vtest ntpd[29665]: no servers reachable Nov 26 09:06:15 hsg-vtest ntpd[29665]: synchronized to 141.211.144.15, stratum 2 Nov 26 09:06:06 hsg-vtest ntpd[29665]: time reset -8.216743 s Nov 26 09:10:46 hsg-vtest ntpd[29665]: synchronized to 141.211.125.15, stratum 3 Nov 26 09:15:06 hsg-vtest ntpd[29665]: synchronized to 141.211.144.15, stratum 3 Nov 26 09:15:15 hsg-vtest ntpd[29665]: no servers reachable Nov 26 09:23:38 hsg-vtest ntpd[29665]: synchronized to 141.211.144.15, stratum 2 Nov 26 09:27:01 hsg-vtest ntpd[29665]: time reset -4.652686 s Nov 26 09:39:38 hsg-vtest ntpd[29665]: synchronized to 141.211.144.15, stratum 3 Nov 26 09:45:04 hsg-vtest ntpd[29665]: synchronized to 141.211.125.15, stratum 3 Nov 26 09:44:59 hsg-vtest ntpd[29665]: time reset -5.117679 s ~~~ Hope Dundas Database/Systems Administrator Housing Information Technology Office [HITO] University of Michigan ============================================== Collins, Kevin [MindWorks] Wed, 03 Oct 2007 09:01:40 -0700 The path under proc you listed looks wrong to me - what is /proc/sys/xeno? My systems have /proc/sys/xen/independent_wallclock, but no "xeno" directory. There are numerous other typos in that FAQ... I never even thought about time sync, and have happily been running NTP on all my guests with no problems and the time seems to stay sync'd. I have /proc/sys/xen/independent_wallclock at 0, so it could be the NTP isn't really doing anything... Kevin -----Original Message----- From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Matthias Saou Sent: Wednesday, October 03, 2007 4:30 AM To: [email protected] Subject: Re: [rhelv5-list] Xen guests and time differences Matthias Saou wrote : > On some RHEL5 host/guest Xen setups, I've seen time differences of up > to 106s after 2 weeks uptime. > > Right now I've got a Dell PowerEdge 750 running 2.6.18-8.1.4.el5xen and > running ntpd, with its time perfectly synchronized, acting as the Xen > host. > > This host has two guests running 2.6.18-8.1.10.el5xen, which this > morning had 80s and 106s difference reported by running ntpdate. This > is pretty bad, and according to the Xen documentation, something which > shouldn't happen, ever! > > (In http://xen.epiuse.com/xen-faq.txt see the "Xen time" section) > > What I did in the first guest is to run this : > echo 1 >/proc/sys/xeno/independent_wallclock > Ran ntpdate and enabled ntpd, and now its time is fine. > > The second guest still has its time always 106s off : > # ntpdate ntp > 1 Oct 12:40:27 ntpdate[371]: step time server x offset -106.553042 sec > > Is there any known bug in RHEL5's Xen which explains this? What > solutions do I have apart from changing 'independent_wallclock' on all > my Xen guests and syncing their time as if they were real physical > servers? No answers, no solutions it seems, and quite a nasty bug in enterprise environments, ouch! I've decided to "echo 1 >/proc/sys/xeno/independent_wallclock" on all my Xen guests and make them sync their clock using ntp themselves. Useless added complexity since they're supposed to keep the time of the host, but as this is not happening, I needed a solution. I also just noticed that on 32bit x86 guests, "hwclock --systohc" fails silently, whereas on 64bit x86_64 guests it complains about not being able to access the hardware clock via any known method... maybe a clue, as I've only really investigated the problem on 32bit guests, so maybe the 64bit ones weren't affected, but I didn't check prior to the change. So at least to all 32bit Xen users out there : BEWARE OF SYSTEM TIME PROBLEMS IN YOUR GUESTS! Matthias -- Clean custom Red Hat Linux rpm packages : http://freshrpms.net/ Fedora release 7 (Moonshine) - Linux kernel 2.6.22.6-81.fc7 Load : 0.28 0.29 0.28 _______________________________________________ rhelv5-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/rhelv5-list _______________________________________________ rhelv5-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/rhelv5-list
