Bug#599161: xen-linux-system-2.6.32-5-xen-amd64: Clock moved forward 50 minutes, caused Xen HVM domU restart
Hello, Am 25.09.2011 um 20:23 Uhr schrieb Ian Campbell i...@hellion.org.uk: On Sun, 2011-09-25 at 15:09 +0200, Markus Hochholdinger wrote: I can confirm that there are problems with the clock in PV domUs. [..] I'm wondering if I can set the former behavior of the domU clock to use the dom0s clock? I'm afraid that the old dependent wallclock mode is not available with the upstream kernel so no. The current recommendation is to run ntpd in your guests. many thanks for this information. I'm now going to install and configure ntpd in all my domUs with squeeze-Kernel. -- greetings eMHa signature.asc Description: This is a digitally signed message part.
Bug#599161: xen-linux-system-2.6.32-5-xen-amd64: Clock moved forward 50 minutes, caused Xen HVM domU restart
Hello, I can confirm that there are problems with the clock in PV domUs. I've setups like as follows: * Debian lenny dom0s running Debian squeeze and Debian lenny domUs (PV) * Debian squeeze doms running Debian squeeze and Debian lenny domUs (PV) I only have clock/time problems with my squeeze domUs. They use 2.6.32-5-amd64 and 2.6.32-5-686 kernels from Debian squeeze. It doesn't matter if they run on lenny (2.6.26-*-xen-686) or squeeze (2.6.32-*-xen-686) dom0s. With my Debian lenny dom0s I've set clocksource=jiffies and my lenny domUs also have clocksource=jiffies and independent_wallclock set to 0. All is fine here. I've running ntpd on the dom0s and the domUs use the time from the dom0s. (I've tested to change the time in the dom0 and the time change also show up in the domUs.) so I've no Problems with lenny domUs and time. With debian squeeze there's no clocksource jiffies anymore, so I use the default clocksource=xen in squeeze doms and squeeze domUs. Also there's no independent_wallclock anymore. So here's the problem I see: The time in my squeeze domUs is running slightly slower than the time on my dom0. Changing the time on the dom0 doesn't change the time in the squeeze domUs. But I can run ntpdate/ntp in my squeeze domUs and time gets correct. I'm wondering if I can set the former behavior of the domU clock to use the dom0s clock? -- greetings eMHa signature.asc Description: This is a digitally signed message part.
Bug#599161: xen-linux-system-2.6.32-5-xen-amd64: Clock moved forward 50 minutes, caused Xen HVM domU restart
On Sun, 2011-09-25 at 15:09 +0200, Markus Hochholdinger wrote: Hello, I can confirm that there are problems with the clock in PV domUs. I've setups like as follows: * Debian lenny dom0s running Debian squeeze and Debian lenny domUs (PV) * Debian squeeze doms running Debian squeeze and Debian lenny domUs (PV) I only have clock/time problems with my squeeze domUs. They use 2.6.32-5-amd64 and 2.6.32-5-686 kernels from Debian squeeze. It doesn't matter if they run on lenny (2.6.26-*-xen-686) or squeeze (2.6.32-*-xen-686) dom0s. With my Debian lenny dom0s I've set clocksource=jiffies and my lenny domUs also have clocksource=jiffies and independent_wallclock set to 0. All is fine here. I've running ntpd on the dom0s and the domUs use the time from the dom0s. (I've tested to change the time in the dom0 and the time change also show up in the domUs.) so I've no Problems with lenny domUs and time. With debian squeeze there's no clocksource jiffies anymore, so I use the default clocksource=xen in squeeze doms and squeeze domUs. Also there's no independent_wallclock anymore. So here's the problem I see: The time in my squeeze domUs is running slightly slower than the time on my dom0. Changing the time on the dom0 doesn't change the time in the squeeze domUs. But I can run ntpdate/ntp in my squeeze domUs and time gets correct. I'm wondering if I can set the former behavior of the domU clock to use the dom0s clock? I'm afraid that the old dependent wallclock mode is not available with the upstream kernel so no. The current recommendation is to run ntpd in your guests. Ian. -- Ian Campbell Our missions are peaceful -- not for conquest. When we do battle, it is only because we have no choice. -- Kirk, The Squire of Gothos, stardate 2124.5 signature.asc Description: This is a digitally signed message part
Bug#599161: xen-linux-system-2.6.32-5-xen-amd64: Clock moved forward 50 minutes, caused Xen HVM domU restart
Hello, I am got this issue on several (10+) Xen 4.0 servers without HVM: linux-image-2.6.32-bpo.5-xen-amd64 2.6.32-29~bpo50+1 Linux 2.6.32 for 64-bit PCs, Xen dom0 suppor xen-hypervisor-4.0-amd644.0.1-1The Xen Hypervisor on AMD64 Each times there is this strange 50min : Nov 4 13:33:55 kernel: [591694.109052] Clocksource tsc unstable (delta = -2999660340870 ns) Nov 16 19:04:27 kernel: [102888.814677] Clocksource tsc unstable (delta = -2999660352101 ns) Nov 28 08:16:52 kernel: [2671840.236281] Clocksource tsc unstable (delta = -299966013 ns) Nov 29 08:59:54 kernel: [171581.195202] Clocksource tsc unstable (delta = -2999660341178 ns) Dec 8 12:58:09 kernel: [3108143.298526] Clocksource tsc unstable (delta = -2999660353020 ns) Dec 27 02:34:16 kernel: [1012551.748589] Clocksource tsc unstable (delta = -2999660334211 ns) Jan 12 09:12:53 kernel: [6537645.820016] Clocksource tsc unstable (delta = -2999660339286 ns) Jan 28 11:04:54 kernel: [5352834.035048] Clocksource tsc unstable (delta = -2999660330184 ns) Feb 9 21:04:03 kernel: [6415408.244988] Clocksource tsc unstable (delta = -2999660342333 ns) Feb 10 04:19:24 kernel: [4306193.416722] Clocksource tsc unstable (delta = -2999660332510 ns) after that ntp in Dom0 fails with : ntpd[6834]: time correction of -3000 seconds exceeds sanity limit (1000); set clock manually to the correct UTC time Same things with ntp in domU. I didn't have this kind of issue with Xen Lenny. Regards Olivier
Bug#599161: xen-linux-system-2.6.32-5-xen-amd64: Clock moved forward 50 minutes, caused Xen HVM domU restart
Hi, No, unfortunately not. I just suffered it again aswell over the Christmas break. Let me know if you find any fix. Regards, Mark On Sun, Dec 26, 2010 at 12:50:26PM +0100, Moritz Muehlenhoff wrote: On Wed, Oct 06, 2010 at 12:05:18PM +0100, Mark Adams wrote: As this is when the clock went from 18:00 to 18:50 and started the chain of events (restarted the 2008 domU). Any ideas why this log occurred? The TSC appeared to go backwards by a fairly significant amount, which has upset the kernel. The behaviour of the virtual TSC as seen by an HVM guest is controlled by a combination of the tsc_mode setting in your domain configuration and, I think, by the features of your specific hardware (some have constant rate TSC, others advertise varying levels of synchronisation between cores etc). It's then up to the guest kernel whether it even uses TSC as a timesource at all and how it handles instability etc and how it derives other time sources (such as the wallclock time) from it. Sorry this isn't more helpful, but as I say you will probably get better answers on one of the Xen mailing lists. Hi Ian, Thanks for your notes. I've already tried the xen-users list so I will try xen-devel to see if I can glean any more information about my issue. I will update the report here when I get any more information. Did you contact them? From what I read from the bug report so far, running an NTP client in the HVM host should suffice. Cheers, Moritz -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#599161: xen-linux-system-2.6.32-5-xen-amd64: Clock moved forward 50 minutes, caused Xen HVM domU restart
In addition, I received the below from James Song but when I queried what it changed he did not respond, so I haven't tried it.. added timer_mode =2 and tsc_mode = 1 and viridian=1 into your configure file. On Tue, Jan 04, 2011 at 04:56:00PM +, Mark Adams wrote: Hi, No, unfortunately not. I just suffered it again aswell over the Christmas break. Let me know if you find any fix. Regards, Mark On Sun, Dec 26, 2010 at 12:50:26PM +0100, Moritz Muehlenhoff wrote: On Wed, Oct 06, 2010 at 12:05:18PM +0100, Mark Adams wrote: As this is when the clock went from 18:00 to 18:50 and started the chain of events (restarted the 2008 domU). Any ideas why this log occurred? The TSC appeared to go backwards by a fairly significant amount, which has upset the kernel. The behaviour of the virtual TSC as seen by an HVM guest is controlled by a combination of the tsc_mode setting in your domain configuration and, I think, by the features of your specific hardware (some have constant rate TSC, others advertise varying levels of synchronisation between cores etc). It's then up to the guest kernel whether it even uses TSC as a timesource at all and how it handles instability etc and how it derives other time sources (such as the wallclock time) from it. Sorry this isn't more helpful, but as I say you will probably get better answers on one of the Xen mailing lists. Hi Ian, Thanks for your notes. I've already tried the xen-users list so I will try xen-devel to see if I can glean any more information about my issue. I will update the report here when I get any more information. Did you contact them? From what I read from the bug report so far, running an NTP client in the HVM host should suffice. Cheers, Moritz -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#599161: xen-linux-system-2.6.32-5-xen-amd64: Clock moved forward 50 minutes, caused Xen HVM domU restart
On Wed, Oct 06, 2010 at 12:05:18PM +0100, Mark Adams wrote: As this is when the clock went from 18:00 to 18:50 and started the chain of events (restarted the 2008 domU). Any ideas why this log occurred? The TSC appeared to go backwards by a fairly significant amount, which has upset the kernel. The behaviour of the virtual TSC as seen by an HVM guest is controlled by a combination of the tsc_mode setting in your domain configuration and, I think, by the features of your specific hardware (some have constant rate TSC, others advertise varying levels of synchronisation between cores etc). It's then up to the guest kernel whether it even uses TSC as a timesource at all and how it handles instability etc and how it derives other time sources (such as the wallclock time) from it. Sorry this isn't more helpful, but as I say you will probably get better answers on one of the Xen mailing lists. Hi Ian, Thanks for your notes. I've already tried the xen-users list so I will try xen-devel to see if I can glean any more information about my issue. I will update the report here when I get any more information. Did you contact them? From what I read from the bug report so far, running an NTP client in the HVM host should suffice. Cheers, Moritz -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#599161: xen-linux-system-2.6.32-5-xen-amd64: Clock moved forward 50 minutes, caused Xen HVM domU restart
Hi Ben, Thanks for your response. See my responses inline. On Wed, Oct 06, 2010 at 03:35:23AM +0100, Ben Hutchings wrote: On Tue, 2010-10-05 at 09:31 +0100, Mark Adams wrote: Package: xen-linux-system-2.6.32-5-xen-amd64 Version: 2.6.32-21 Severity: important Hi, Did you receive this bug report? I hadn't received a bug ID even though receiving the copy of the original report. We don't have any other bug report with this description. Likely because the address it was sent from originally was invalid. That would explain it. - Hi All, Im running Xen 4.0.1-rc6 Debian squeeze with pvops 2.6.32-21 kernel. Today I noticed (when kerberos to the domain controllers stopped working..) that the clock was 50 minutes out in dom0 -- This caused the HVM windows domain controllers to have the wrong time. Since you appear to be in the UK, is it possible that the real-time clock is set to local time (GMT+1) while Xen expects it to be GMT, or vice versa? The clock is set with tzdata as BST yes, it is also set to this in the Windows server 2008 domU. We are using localtime=1 to match the clock in dom0 to domU. (This doesn't explain why it's 50 minutes out rather than 1 hour. But ntpd will refuse to correct a large difference and the local clock may then drift further.) [...] Can anyone confirm whether xen controls the time or the kernel? Also when I corrected the time in dom0 it was still wrong in HVM domU -- How long does it take for this to propogate? (I rebooted the VM's to correct it immediately). [...] For HVM guests, the hypervisor emulates a standard PC real-time clock and the guest uses that to initialise the system time, but there is no way to force an update after the guest has booted unless the guest has specific support for Xen; I assume Citrix does provide such software for Windows but I don't know whether it is free software. The citrix WHQL drivers might have this functionality, I don't use them though - prefer the GPL PV drivers! (which don't have any clock support as far as I can tell) For PV guests, I assume you can force an update to the guest time using the Xen management tools. Note, I'm just a general kernel maintainer and don't have any great knowledge of Xen. All good, I have a feeling it might be a kernel issue rather than xen, but I'm still not sure what actually -controls- the time, is it the kernel? I think the key is in the log Oct 2 18:50:33 havhost1 kernel: [623480.977748] Clocksource tsc unstable (delta = -2999660303788 ns) As this is when the clock went from 18:00 to 18:50 and started the chain of events (restarted the 2008 domU). Any ideas why this log occurred? Ben. Regards, Mark -- Ben Hutchings Once a job is fouled up, anything done to improve it makes it worse. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#599161: xen-linux-system-2.6.32-5-xen-amd64: Clock moved forward 50 minutes, caused Xen HVM domU restart
On Wed, 2010-10-06 at 09:33 +0100, Mark Adams wrote: Hi Ben, Thanks for your response. See my responses inline. On Wed, Oct 06, 2010 at 03:35:23AM +0100, Ben Hutchings wrote: On Tue, 2010-10-05 at 09:31 +0100, Mark Adams wrote: Package: xen-linux-system-2.6.32-5-xen-amd64 Version: 2.6.32-21 Severity: important Hi, Did you receive this bug report? I hadn't received a bug ID even though receiving the copy of the original report. We don't have any other bug report with this description. Likely because the address it was sent from originally was invalid. That would explain it. - Hi All, Im running Xen 4.0.1-rc6 Debian squeeze with pvops 2.6.32-21 kernel. Today I noticed (when kerberos to the domain controllers stopped working..) that the clock was 50 minutes out in dom0 -- This caused the HVM windows domain controllers to have the wrong time. Since you appear to be in the UK, is it possible that the real-time clock is set to local time (GMT+1) while Xen expects it to be GMT, or vice versa? The clock is set with tzdata as BST yes, it is also set to this in the Windows server 2008 domU. We are using localtime=1 to match the clock in dom0 to domU. (This doesn't explain why it's 50 minutes out rather than 1 hour. But ntpd will refuse to correct a large difference and the local clock may then drift further.) [...] Can anyone confirm whether xen controls the time or the kernel? Also when I corrected the time in dom0 it was still wrong in HVM domU -- How long does it take for this to propogate? (I rebooted the VM's to correct it immediately). [...] For HVM guests, the hypervisor emulates a standard PC real-time clock and the guest uses that to initialise the system time, but there is no way to force an update after the guest has booted unless the guest has specific support for Xen; I assume Citrix does provide such software for Windows but I don't know whether it is free software. The citrix WHQL drivers might have this functionality, I don't use them though - prefer the GPL PV drivers! (which don't have any clock support as far as I can tell) I think you can run a regular NTP client (assuming one exists for Windows) in an HVM guest to keep wallclock time in sync. For PV guests, I assume you can force an update to the guest time using the Xen management tools. Note, I'm just a general kernel maintainer and don't have any great knowledge of Xen. I should know but time handling (particularly for HVM guests) is something where I basically know enough to know that there is lots I don't know ;-) Mark, you may find you get better answers/support from the xen-users mailing list, or failing that, xen-devel. All good, I have a feeling it might be a kernel issue rather than xen, but I'm still not sure what actually -controls- the time, is it the kernel? I think the key is in the log Oct 2 18:50:33 havhost1 kernel: [623480.977748] Clocksource tsc unstable (delta = -2999660303788 ns) As this is when the clock went from 18:00 to 18:50 and started the chain of events (restarted the 2008 domU). Any ideas why this log occurred? The TSC appeared to go backwards by a fairly significant amount, which has upset the kernel. The behaviour of the virtual TSC as seen by an HVM guest is controlled by a combination of the tsc_mode setting in your domain configuration and, I think, by the features of your specific hardware (some have constant rate TSC, others advertise varying levels of synchronisation between cores etc). It's then up to the guest kernel whether it even uses TSC as a timesource at all and how it handles instability etc and how it derives other time sources (such as the wallclock time) from it. Sorry this isn't more helpful, but as I say you will probably get better answers on one of the Xen mailing lists. Ian. -- Ian Campbell Current Noise: Trouble -
Bug#599161: xen-linux-system-2.6.32-5-xen-amd64: Clock moved forward 50 minutes, caused Xen HVM domU restart
On Wed, Oct 06, 2010 at 11:47:05AM +0100, Ian Campbell wrote: On Wed, 2010-10-06 at 09:33 +0100, Mark Adams wrote: Hi Ben, Thanks for your response. See my responses inline. On Wed, Oct 06, 2010 at 03:35:23AM +0100, Ben Hutchings wrote: On Tue, 2010-10-05 at 09:31 +0100, Mark Adams wrote: Package: xen-linux-system-2.6.32-5-xen-amd64 Version: 2.6.32-21 Severity: important Hi, Did you receive this bug report? I hadn't received a bug ID even though receiving the copy of the original report. We don't have any other bug report with this description. Likely because the address it was sent from originally was invalid. That would explain it. - Hi All, Im running Xen 4.0.1-rc6 Debian squeeze with pvops 2.6.32-21 kernel. Today I noticed (when kerberos to the domain controllers stopped working..) that the clock was 50 minutes out in dom0 -- This caused the HVM windows domain controllers to have the wrong time. Since you appear to be in the UK, is it possible that the real-time clock is set to local time (GMT+1) while Xen expects it to be GMT, or vice versa? The clock is set with tzdata as BST yes, it is also set to this in the Windows server 2008 domU. We are using localtime=1 to match the clock in dom0 to domU. (This doesn't explain why it's 50 minutes out rather than 1 hour. But ntpd will refuse to correct a large difference and the local clock may then drift further.) [...] Can anyone confirm whether xen controls the time or the kernel? Also when I corrected the time in dom0 it was still wrong in HVM domU -- How long does it take for this to propogate? (I rebooted the VM's to correct it immediately). [...] For HVM guests, the hypervisor emulates a standard PC real-time clock and the guest uses that to initialise the system time, but there is no way to force an update after the guest has booted unless the guest has specific support for Xen; I assume Citrix does provide such software for Windows but I don't know whether it is free software. The citrix WHQL drivers might have this functionality, I don't use them though - prefer the GPL PV drivers! (which don't have any clock support as far as I can tell) I think you can run a regular NTP client (assuming one exists for Windows) in an HVM guest to keep wallclock time in sync. For PV guests, I assume you can force an update to the guest time using the Xen management tools. Note, I'm just a general kernel maintainer and don't have any great knowledge of Xen. I should know but time handling (particularly for HVM guests) is something where I basically know enough to know that there is lots I don't know ;-) Mark, you may find you get better answers/support from the xen-users mailing list, or failing that, xen-devel. All good, I have a feeling it might be a kernel issue rather than xen, but I'm still not sure what actually -controls- the time, is it the kernel? I think the key is in the log Oct 2 18:50:33 havhost1 kernel: [623480.977748] Clocksource tsc unstable (delta = -2999660303788 ns) As this is when the clock went from 18:00 to 18:50 and started the chain of events (restarted the 2008 domU). Any ideas why this log occurred? The TSC appeared to go backwards by a fairly significant amount, which has upset the kernel. The behaviour of the virtual TSC as seen by an HVM guest is controlled by a combination of the tsc_mode setting in your domain configuration and, I think, by the features of your specific hardware (some have constant rate TSC, others advertise varying levels of synchronisation between cores etc). It's then up to the guest kernel whether it even uses TSC as a timesource at all and how it handles instability etc and how it derives other
Bug#599161: xen-linux-system-2.6.32-5-xen-amd64: Clock moved forward 50 minutes, caused Xen HVM domU restart
Package: xen-linux-system-2.6.32-5-xen-amd64 Version: 2.6.32-21 Severity: important Hi, Did you receive this bug report? I hadn't received a bug ID even though receiving the copy of the original report. Likely because the address it was sent from originally was invalid. - Hi All, Im running Xen 4.0.1-rc6 Debian squeeze with pvops 2.6.32-21 kernel. Today I noticed (when kerberos to the domain controllers stopped working..) that the clock was 50 minutes out in dom0 -- This caused the HVM windows domain controllers to have the wrong time. I'm not sure if this is a kernel issue or a xen issue, but the only thing related is I can see the following in the kernel log: Oct 2 18:50:33 havhost1 kernel: [623480.977748] Clocksource tsc unstable (delta = -2999660303788 ns) But I also see in the dmesg log that xen is using it's own clock. [7.676563] Switching to clocksource xen I can't identify anything else in the logs to indicate when the time might have changed. I have a few other dom0 at the same level that haven't decided to change the time. Can anyone confirm whether xen controls the time or the kernel? Also when I corrected the time in dom0 it was still wrong in HVM domU -- How long does it take for this to propogate? (I rebooted the VM's to correct it immediately). It appears the HVM domU (windows server 2008) unexpectedly shut down at 18:51, after the unstable clocksource error. qemu-dm logs show a reset reset requested in cpu_handle_ioreq. and xend.log shows a reboot Any other pointers on how to ensure stability of clocks from dom0 to domU HVM hosts (and pv for that matter..) would be appreciated. NTP was running but also appeared to have crashed at the same time. Cheers, Mark
Bug#599161: xen-linux-system-2.6.32-5-xen-amd64: Clock moved forward 50 minutes, caused Xen HVM domU restart
On Tue, 2010-10-05 at 09:31 +0100, Mark Adams wrote: Package: xen-linux-system-2.6.32-5-xen-amd64 Version: 2.6.32-21 Severity: important Hi, Did you receive this bug report? I hadn't received a bug ID even though receiving the copy of the original report. We don't have any other bug report with this description. Likely because the address it was sent from originally was invalid. That would explain it. - Hi All, Im running Xen 4.0.1-rc6 Debian squeeze with pvops 2.6.32-21 kernel. Today I noticed (when kerberos to the domain controllers stopped working..) that the clock was 50 minutes out in dom0 -- This caused the HVM windows domain controllers to have the wrong time. Since you appear to be in the UK, is it possible that the real-time clock is set to local time (GMT+1) while Xen expects it to be GMT, or vice versa? (This doesn't explain why it's 50 minutes out rather than 1 hour. But ntpd will refuse to correct a large difference and the local clock may then drift further.) [...] Can anyone confirm whether xen controls the time or the kernel? Also when I corrected the time in dom0 it was still wrong in HVM domU -- How long does it take for this to propogate? (I rebooted the VM's to correct it immediately). [...] For HVM guests, the hypervisor emulates a standard PC real-time clock and the guest uses that to initialise the system time, but there is no way to force an update after the guest has booted unless the guest has specific support for Xen; I assume Citrix does provide such software for Windows but I don't know whether it is free software. For PV guests, I assume you can force an update to the guest time using the Xen management tools. Note, I'm just a general kernel maintainer and don't have any great knowledge of Xen. Ben. -- Ben Hutchings Once a job is fouled up, anything done to improve it makes it worse. signature.asc Description: This is a digitally signed message part
