RE: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
We have noticed a similar problem for years now, from freebsd 9X up through 10XR. When a Freebsd VM is livemigrated under xenserver 6.5 - it will almost always change it's date/time back to 1970 or thereabouts. Windows VM's done at the same time never have the issue. Servers are HP DL1000 (4 node). It happens on all of them. We've just come to live with it, and check the date/time on any Freebsd vm's whenever they are migrated. Jay West -Original Message- From: owner-freebsd-...@freebsd.org [mailto:owner-freebsd-...@freebsd.org] On Behalf Of Roger Pau Monné Sent: Monday, August 22, 2016 9:42 AM To: Hoyer-Reuther, Christian Cc: freebsd-xen@freebsd.org Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? On Mon, Aug 22, 2016 at 04:04:46PM +0200, Hoyer-Reuther, Christian wrote: > Hello, > > with 11.0-RC1 I see the same problem. > > I wonder if I am the only one with this problem. Are you able to reproduce it? Hello, I don't seem to be able to reproduce this, and Karl who also uses XenServer recently told me that he was able to perform ~100 migrations without issues [0]. Ccing him in case he has something to add. Can you provide some description about the hardware that you use on your pool? Thanks, Roger. [0] https://lists.freebsd.org/pipermail/freebsd-xen/2016-July/002779.html ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org" ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
On Mon, Aug 22, 2016 at 04:04:46PM +0200, Hoyer-Reuther, Christian wrote: > Hello, > > with 11.0-RC1 I see the same problem. > > I wonder if I am the only one with this problem. Are you able to reproduce it? Hello, I don't seem to be able to reproduce this, and Karl who also uses XenServer recently told me that he was able to perform ~100 migrations without issues [0]. Ccing him in case he has something to add. Can you provide some description about the hardware that you use on your pool? Thanks, Roger. [0] https://lists.freebsd.org/pipermail/freebsd-xen/2016-July/002779.html ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
Hello, with 11.0-RC1 I see the same problem. I wonder if I am the only one with this problem. Are you able to reproduce it? Regards, Christian > -Original Message- > From: Hoyer-Reuther, Christian > Sent: Friday, August 12, 2016 10:28 AM > To: 'Roger Pau Monné' > Cc: Wei Liu; freebsd-xen@freebsd.org > Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? > > Hm, could you give a try to 11.0-RC1 when it comes out? It should be out > > today or tomorrow (builds are already underway). > > > > Roger. > > No, I'm not booting with boot_verbose="YES". > > I will give 11.0-RC1 a try when it is available. > > Regards, > > Christian ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
Hello, > -Original Message- > From: Roger Pau Monné [mailto:roger@citrix.com] > Sent: Friday, August 12, 2016 10:01 AM > To: Hoyer-Reuther, Christian > Cc: Wei Liu; freebsd-xen@freebsd.org > Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? > > On Thu, Aug 11, 2016 at 02:29:10PM +0200, Hoyer-Reuther, Christian wrote: > > Hello, > > > > sorry for the late response, but now I'm back from vacation and did some > new tests. > > > > I installed a new VM using the FreeBSD-11.0-ALPHA6-amd64-20160701-r302303- > disc1.iso, then updated to HEAD r303908 and installed xe-guest-utilities- > 6.2.0_2 from HEAD ports r420055. Xenguest_enable="YES" is set in > /etc/rc.conf. > > > > I tested VM migration from XenCenter (XenMotion). > > > > There are no more time issues after migration. > > > > I did 4 migrations and at 2 migrations I noticed the following problems: > > > > When the migration finishes then the VM console switches to the VGA BIOS > screen (Plex86/Bochs VGABios ...), the VM console freezes, the VM is not > reachable over network and in XenCenter I see the first 2 CPU's raising up > to 100 percent. Screenshots attached. > > Hm, not sure what's going on there, and why you seem to be the only one to > hit this issue. Are you maybe booting with boot_verbose="YES"? > > > Then after 10 minutes the CPU usage goes down, the VM console is > responsive again and the machine is reachable over network. > > Hm, could you give a try to 11.0-RC1 when it comes out? It should be out > today or tomorrow (builds are already underway). > > Roger. No, I'm not booting with boot_verbose="YES". I will give 11.0-RC1 a try when it is available. Regards, Christian ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
On Thu, Aug 11, 2016 at 02:29:10PM +0200, Hoyer-Reuther, Christian wrote: > Hello, > > sorry for the late response, but now I'm back from vacation and did some new > tests. > > I installed a new VM using the > FreeBSD-11.0-ALPHA6-amd64-20160701-r302303-disc1.iso, then updated to HEAD > r303908 and installed xe-guest-utilities-6.2.0_2 from HEAD ports r420055. > Xenguest_enable="YES" is set in /etc/rc.conf. > > I tested VM migration from XenCenter (XenMotion). > > There are no more time issues after migration. > > I did 4 migrations and at 2 migrations I noticed the following problems: > > When the migration finishes then the VM console switches to the VGA BIOS > screen (Plex86/Bochs VGABios ...), the VM console freezes, the VM is not > reachable over network and in XenCenter I see the first 2 CPU's raising up to > 100 percent. Screenshots attached. Hm, not sure what's going on there, and why you seem to be the only one to hit this issue. Are you maybe booting with boot_verbose="YES"? > Then after 10 minutes the CPU usage goes down, the VM console is responsive > again and the machine is reachable over network. Hm, could you give a try to 11.0-RC1 when it comes out? It should be out today or tomorrow (builds are already underway). Roger. ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
Hello,
sorry for the late response, but now I'm back from vacation and did some new
tests.
I installed a new VM using the
FreeBSD-11.0-ALPHA6-amd64-20160701-r302303-disc1.iso, then updated to HEAD
r303908 and installed xe-guest-utilities-6.2.0_2 from HEAD ports r420055.
Xenguest_enable="YES" is set in /etc/rc.conf.
I tested VM migration from XenCenter (XenMotion).
There are no more time issues after migration.
I did 4 migrations and at 2 migrations I noticed the following problems:
When the migration finishes then the VM console switches to the VGA BIOS screen
(Plex86/Bochs VGABios ...), the VM console freezes, the VM is not reachable
over network and in XenCenter I see the first 2 CPU's raising up to 100
percent. Screenshots attached.
Then after 10 minutes the CPU usage goes down, the VM console is responsive
again and the machine is reachable over network.
Regards,
Christian
> -Original Message-
> From: Roger Pau Monné [mailto:roger@citrix.com]
> Sent: Friday, July 29, 2016 10:29 AM
> To: Wei Liu
> Cc: Karl Pielorz; Hoyer-Reuther, Christian; freebsd-xen@freebsd.org
> Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
>
> On Mon, Jul 25, 2016 at 04:37:14PM +0100, Wei Liu wrote:
> > On Mon, Jul 25, 2016 at 04:43:43PM +0200, Roger Pau Monné wrote:
> > > Adding Wei to the Cc list since he added the multiqueue functionality.
> > >
> > > On Mon, Jul 25, 2016 at 02:59:02PM +0100, Karl Pielorz wrote:
> > > >
> > > > --On 22 July 2016 13:55 +0200 Roger Pau Monné
> wrote:
> > > >
> > > > > In my environment I've migrated a FreeBSD VM with 2 cpus for > 100
> > > > > consecutive times without seeing any issues (or freezes), although
> this
> > > > > was with OSS Xen and without xe-guest-utilities. Karl, have you
> tested
> > > > > HEAD recently?
> > > >
> > > > Ok, I have tested this with r303286 - it seems to work OK. The hosts
> gain no
> > > > time that I can see while migrating, and NTP stays happy.
> > > >
> > > > I did get a panic after about 40 migrations - but that seems to be
> some
> > > > network issue or something...
> > > >
> > > > ('panic called with 0 available queues / dbt_trace_self_wrapper /
> vpanic /
> > > > kassert_panic / xn_txq_mq_start / ether_output / udp_send /
> sosend_dgram /
> > > > kern_sendit / sendit / sys_sendto / amd64_syscall / Xfast_syscall)
> > >
> > > I haven't been able to reproduce this, but I think it's possible that if
> you
> > > migrate an active netfront xn_txq_mq_start might be called during the
> > > migration, just in the middle of the setup_device reconfiguation (while
> > > info->num_queues is 0).
> > >
> > > Wei, I think netif_disconnect_backend should set IFF_DRV_OACTIVE in
> order to
> > > notify the net subsystem that the queues are full, so no further calls
> to
> > > xn_txq_mq_start happen until the resume has finished, do you agree?
> > >
> >
> > Perhaps clear IFF_DRV_RUNNING and only set it when the device is ready?
> > Looking at the manpage is seems more appropriate to me semantically.
>
> Hello Karl and Christian, I have the following patches that solve all the
> issues I've seen with live migration, with those I've been able to migrate a
> VM > 100 times without seeing any issues. Could you give them a try?
>
> BTW, I haven't been able to reproduce Karl's crash ("called with 0 available
> queues"), but I've added a condition that should prevent it from triggering
> anyway. Patches are here:
>
> https://reviews.freebsd.org/D7349
> https://reviews.freebsd.org/D7362
> https://reviews.freebsd.org/D7363
>
> It doesn't really matter in which order you apply them as long as both 3 are
> applied. Ideally I would like to commit them on Monday, so that I can MFC
> them to stable/11 before the releng/11 branch, could you please provide some
> feedback before then?
>
> Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
--On 29 July 2016 10:29 +0200 Roger Pau Monné wrote:
Hello Karl and Christian, I have the following patches that solve all the
issues I've seen with live migration, with those I've been able to
migrate a VM > 100 times without seeing any issues. Could you give them
a try?
BTW, I haven't been able to reproduce Karl's crash ("called with 0
available queues"), but I've added a condition that should prevent it
from triggering anyway. Patches are here:
https://reviews.freebsd.org/D7349
https://reviews.freebsd.org/D7362
https://reviews.freebsd.org/D7363
It doesn't really matter in which order you apply them as long as both 3
are applied. Ideally I would like to commit them on Monday, so that I
can MFC them to stable/11 before the releng/11 branch, could you please
provide some feedback before then?
Patched, and have been running migrations back & forth between two pool
members all day (must have done ~100), time has stayed in sync - and I've
not experienced any panics.
This is on the same VM as before - I've also tried leaving more heavy
network processes running in the background, and still seems Ok.
Regards,
-Karl
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
On Mon, Jul 25, 2016 at 04:37:14PM +0100, Wei Liu wrote:
> On Mon, Jul 25, 2016 at 04:43:43PM +0200, Roger Pau Monné wrote:
> > Adding Wei to the Cc list since he added the multiqueue functionality.
> >
> > On Mon, Jul 25, 2016 at 02:59:02PM +0100, Karl Pielorz wrote:
> > >
> > > --On 22 July 2016 13:55 +0200 Roger Pau Monné
> > > wrote:
> > >
> > > > In my environment I've migrated a FreeBSD VM with 2 cpus for > 100
> > > > consecutive times without seeing any issues (or freezes), although this
> > > > was with OSS Xen and without xe-guest-utilities. Karl, have you tested
> > > > HEAD recently?
> > >
> > > Ok, I have tested this with r303286 - it seems to work OK. The hosts gain
> > > no
> > > time that I can see while migrating, and NTP stays happy.
> > >
> > > I did get a panic after about 40 migrations - but that seems to be some
> > > network issue or something...
> > >
> > > ('panic called with 0 available queues / dbt_trace_self_wrapper /
> > > vpanic /
> > > kassert_panic / xn_txq_mq_start / ether_output / udp_send / sosend_dgram /
> > > kern_sendit / sendit / sys_sendto / amd64_syscall / Xfast_syscall)
> >
> > I haven't been able to reproduce this, but I think it's possible that if
> > you
> > migrate an active netfront xn_txq_mq_start might be called during the
> > migration, just in the middle of the setup_device reconfiguation (while
> > info->num_queues is 0).
> >
> > Wei, I think netif_disconnect_backend should set IFF_DRV_OACTIVE in order
> > to
> > notify the net subsystem that the queues are full, so no further calls to
> > xn_txq_mq_start happen until the resume has finished, do you agree?
> >
>
> Perhaps clear IFF_DRV_RUNNING and only set it when the device is ready?
> Looking at the manpage is seems more appropriate to me semantically.
Hello Karl and Christian, I have the following patches that solve all the
issues I've seen with live migration, with those I've been able to migrate a
VM > 100 times without seeing any issues. Could you give them a try?
BTW, I haven't been able to reproduce Karl's crash ("called with 0 available
queues"), but I've added a condition that should prevent it from triggering
anyway. Patches are here:
https://reviews.freebsd.org/D7349
https://reviews.freebsd.org/D7362
https://reviews.freebsd.org/D7363
It doesn't really matter in which order you apply them as long as both 3 are
applied. Ideally I would like to commit them on Monday, so that I can MFC
them to stable/11 before the releng/11 branch, could you please provide some
feedback before then?
Thanks, Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
On Mon, Jul 25, 2016 at 04:43:43PM +0200, Roger Pau Monné wrote:
> Adding Wei to the Cc list since he added the multiqueue functionality.
>
> On Mon, Jul 25, 2016 at 02:59:02PM +0100, Karl Pielorz wrote:
> >
> > --On 22 July 2016 13:55 +0200 Roger Pau Monné wrote:
> >
> > > In my environment I've migrated a FreeBSD VM with 2 cpus for > 100
> > > consecutive times without seeing any issues (or freezes), although this
> > > was with OSS Xen and without xe-guest-utilities. Karl, have you tested
> > > HEAD recently?
> >
> > Ok, I have tested this with r303286 - it seems to work OK. The hosts gain no
> > time that I can see while migrating, and NTP stays happy.
> >
> > I did get a panic after about 40 migrations - but that seems to be some
> > network issue or something...
> >
> > ('panic called with 0 available queues / dbt_trace_self_wrapper / vpanic /
> > kassert_panic / xn_txq_mq_start / ether_output / udp_send / sosend_dgram /
> > kern_sendit / sendit / sys_sendto / amd64_syscall / Xfast_syscall)
>
> I haven't been able to reproduce this, but I think it's possible that if you
> migrate an active netfront xn_txq_mq_start might be called during the
> migration, just in the middle of the setup_device reconfiguation (while
> info->num_queues is 0).
>
> Wei, I think netif_disconnect_backend should set IFF_DRV_OACTIVE in order to
> notify the net subsystem that the queues are full, so no further calls to
> xn_txq_mq_start happen until the resume has finished, do you agree?
>
Perhaps clear IFF_DRV_RUNNING and only set it when the device is ready?
Looking at the manpage is seems more appropriate to me semantically.
Wei.
> Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
Adding Wei to the Cc list since he added the multiqueue functionality.
On Mon, Jul 25, 2016 at 02:59:02PM +0100, Karl Pielorz wrote:
>
> --On 22 July 2016 13:55 +0200 Roger Pau Monné wrote:
>
> > In my environment I've migrated a FreeBSD VM with 2 cpus for > 100
> > consecutive times without seeing any issues (or freezes), although this
> > was with OSS Xen and without xe-guest-utilities. Karl, have you tested
> > HEAD recently?
>
> Ok, I have tested this with r303286 - it seems to work OK. The hosts gain no
> time that I can see while migrating, and NTP stays happy.
>
> I did get a panic after about 40 migrations - but that seems to be some
> network issue or something...
>
> ('panic called with 0 available queues / dbt_trace_self_wrapper / vpanic /
> kassert_panic / xn_txq_mq_start / ether_output / udp_send / sosend_dgram /
> kern_sendit / sendit / sys_sendto / amd64_syscall / Xfast_syscall)
I haven't been able to reproduce this, but I think it's possible that if you
migrate an active netfront xn_txq_mq_start might be called during the
migration, just in the middle of the setup_device reconfiguation (while
info->num_queues is 0).
Wei, I think netif_disconnect_backend should set IFF_DRV_OACTIVE in order to
notify the net subsystem that the queues are full, so no further calls to
xn_txq_mq_start happen until the resume has finished, do you agree?
Roger.
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
--On 22 July 2016 13:55 +0200 Roger Pau Monné wrote:
In my environment I've migrated a FreeBSD VM with 2 cpus for > 100
consecutive times without seeing any issues (or freezes), although this
was with OSS Xen and without xe-guest-utilities. Karl, have you tested
HEAD recently?
Ok, I have tested this with r303286 - it seems to work OK. The hosts gain
no time that I can see while migrating, and NTP stays happy.
I did get a panic after about 40 migrations - but that seems to be some
network issue or something...
('panic called with 0 available queues / dbt_trace_self_wrapper / vpanic
/ kassert_panic / xn_txq_mq_start / ether_output / udp_send / sosend_dgram
/ kern_sendit / sendit / sys_sendto / amd64_syscall / Xfast_syscall)
I don't have a crashdump (failed). I did get a backtrace, for what it's
worth.
I'm running the test again now (in case it panics again - I'll try harder
to get a dump just in case).
-Karl
___
freebsd-xen@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
--On 22 July 2016 13:55 +0200 Roger Pau Monné wrote: Hello, In my environment I've migrated a FreeBSD VM with 2 cpus for > 100 consecutive times without seeing any issues (or freezes), although this was with OSS Xen and without xe-guest-utilities. Karl, have you tested HEAD recently? I'm waiting for it to build on our XenServer test pool - I'll get it tested, and get back to you as soon as I can, Cheers, -Karl ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
On Fri, Jul 22, 2016 at 12:01:16PM +0200, Hoyer-Reuther, Christian wrote: > Hello Roger, > > my test machine is now running HEAD (r303131) amd64 with > xe-guest-utilities-6.0.2 (from HEAD ports r418911). My XenServer environment > is still the same. > > When I migrate the VM to another host then the migration finishes but at the > end of migration the VM console switches to the VGA BIOS screen (Plex86/Bochs > VGABios ...). > > The VM console freezes, the VM is not reachable over network and in XenCenter > I see the first 2 CPU's go up to 100 percent. > > Then after 10 minutes the CPU usage goes down, the VM console is responsive > and the machine is reachable over network. And the time is in sync (not in > future). > > Maybe I used the wrong path to setup a machine with HEAD - I upgraded > 10.3-RELEASE to HEAD. > > I will do a 2nd test using the > FreeBSD-11.0-ALPHA6-amd64-20160701-r302303-disc1.iso and then update to the > latest revision. Maybe this way is more clean and will bring other results. > > But I'm not sure if I will get this finished today. I will be in vacation > from Monday and then let you know when I'm finished. Hello, In my environment I've migrated a FreeBSD VM with 2 cpus for > 100 consecutive times without seeing any issues (or freezes), although this was with OSS Xen and without xe-guest-utilities. Karl, have you tested HEAD recently? Roger. ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
RE: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
Hello Roger, my test machine is now running HEAD (r303131) amd64 with xe-guest-utilities-6.0.2 (from HEAD ports r418911). My XenServer environment is still the same. When I migrate the VM to another host then the migration finishes but at the end of migration the VM console switches to the VGA BIOS screen (Plex86/Bochs VGABios ...). The VM console freezes, the VM is not reachable over network and in XenCenter I see the first 2 CPU's go up to 100 percent. Then after 10 minutes the CPU usage goes down, the VM console is responsive and the machine is reachable over network. And the time is in sync (not in future). Maybe I used the wrong path to setup a machine with HEAD - I upgraded 10.3-RELEASE to HEAD. I will do a 2nd test using the FreeBSD-11.0-ALPHA6-amd64-20160701-r302303-disc1.iso and then update to the latest revision. Maybe this way is more clean and will bring other results. But I'm not sure if I will get this finished today. I will be in vacation from Monday and then let you know when I'm finished. Regards, Christian > -Original Message- > From: Roger Pau Monné [mailto:roger@citrix.com] > Sent: Wednesday, July 20, 2016 11:31 AM > To: Hoyer-Reuther, Christian > Cc: freebsd-xen@freebsd.org > Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? > > It seems that HEAD is in a good shape now, starting from r303076 (or any > later revision) you should be able to test live migration without issues. > I will MFC the fix to stable/11 in a couple of days, until then can you give > HEAD a try? ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
RE: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
Hello Roger, I will try to test it until Friday. Regards, Christian > -Original Message- > From: Roger Pau Monné [mailto:roger@citrix.com] > Sent: Wednesday, July 20, 2016 11:31 AM > To: Hoyer-Reuther, Christian > Cc: freebsd-xen@freebsd.org > Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? > > It seems that HEAD is in a good shape now, starting from r303076 (or any > later revision) you should be able to test live migration without issues. > I will MFC the fix to stable/11 in a couple of days, until then can you give > HEAD a try? ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
On Thu, Jul 14, 2016 at 12:47:40PM +0200, Hoyer-Reuther, Christian wrote: > Hello, > > thanks for the update. And yes, please let me know when the new issues are > fixed. I will test it then and report back the results. Hello, It seems that HEAD is in a good shape now, starting from r303076 (or any later revision) you should be able to test live migration without issues. I will MFC the fix to stable/11 in a couple of days, until then can you give HEAD a try? Thanks, Roger. ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
RE: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
Hello, thanks for the update. And yes, please let me know when the new issues are fixed. I will test it then and report back the results. Regards, Christian > -Original Message- > From: Roger Pau Monné [mailto:roger@citrix.com] > Sent: Thursday, July 14, 2016 12:30 PM > To: Hoyer-Reuther, Christian > Cc: freebsd-xen@freebsd.org > Subject: Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions? > > On Tue, Jul 12, 2016 at 02:04:12PM +0200, Hoyer-Reuther, Christian wrote: > > > > We run a XenServer 6.5 SP1 pool with all patches up to XS65ESP1033 on > Fujitsu Primergy RX300 S6 servers. My test VM is FreeBSD 10.3 with xe-guest- > utilities-6.2.0_2 and xen-guest-tools-4.6.1 installed. > > > > After live migration the VM's clock is also several seconds in future. > > Hello, > > I've done some fixes in HEAD some time ago, that are also present in 11. > Those have been tested by Karl, and seem to solve the situation (or at least > the skew is much smaller with those). Sadly migration in both 11 and HEAD > is currently broken due to other issues, so you cannot test it at the > moment. Once that's solved I will ping you so that you can test it. > > Roger. ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
On Tue, Jul 12, 2016 at 02:04:12PM +0200, Hoyer-Reuther, Christian wrote: > Hello list, > > I know this is old but I discovered the same problem and found this thread > but no newer information. > > --On Mon Nov 9 10:23:04 UTC 2015 Karl Pielorz > wrote: > > > I've tested this on two pools now. The original production pool (Xeon > > E3-1220 v3 @ 3.10GHz) has the issue. > > > > Our office / test pool (Xeon L5630 @ 2.13Ghz) also -has the issue-. Both > > are setup pretty much identical (same storage solution, both running > > FreeBSD 10.1/10.2-R with 'xe-guest-utilities' installed). Running or not > > running NTP doesn't make a difference. > > > > Live migrates result in the domU gaining several seconds (thus breaking NTP > > if it is running) else leaving the host 'running in the future' vs. both > > external clocks, and NTP sync'd clocks on the xenserver. > > > > As it's easily reproducible I guess I'll file it as a bug... > > We run a XenServer 6.5 SP1 pool with all patches up to XS65ESP1033 on Fujitsu > Primergy RX300 S6 servers. My test VM is FreeBSD 10.3 with > xe-guest-utilities-6.2.0_2 and xen-guest-tools-4.6.1 installed. > > After live migration the VM's clock is also several seconds in future. Hello, I've done some fixes in HEAD some time ago, that are also present in 11. Those have been tested by Karl, and seem to solve the situation (or at least the skew is much smaller with those). Sadly migration in both 11 and HEAD is currently broken due to other issues, so you cannot test it at the moment. Once that's solved I will ping you so that you can test it. Roger. ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
Hello list, I know this is old but I discovered the same problem and found this thread but no newer information. --On Mon Nov 9 10:23:04 UTC 2015 Karl Pielorz wrote: > I've tested this on two pools now. The original production pool (Xeon > E3-1220 v3 @ 3.10GHz) has the issue. > > Our office / test pool (Xeon L5630 @ 2.13Ghz) also -has the issue-. Both > are setup pretty much identical (same storage solution, both running > FreeBSD 10.1/10.2-R with 'xe-guest-utilities' installed). Running or not > running NTP doesn't make a difference. > > Live migrates result in the domU gaining several seconds (thus breaking NTP > if it is running) else leaving the host 'running in the future' vs. both > external clocks, and NTP sync'd clocks on the xenserver. > > As it's easily reproducible I guess I'll file it as a bug... We run a XenServer 6.5 SP1 pool with all patches up to XS65ESP1033 on Fujitsu Primergy RX300 S6 servers. My test VM is FreeBSD 10.3 with xe-guest-utilities-6.2.0_2 and xen-guest-tools-4.6.1 installed. After live migration the VM's clock is also several seconds in future. I use "tinker panic 0" in /etc/ntp.conf as workaround. Then some minutes after the migration the time is in sync again. Here is some output from a peerstats log: ... VM running on "old" host 57581 41601.319 172.30.99.31 965a 0.000911290 0.000156970 0.437525532 0.79273 57581 41603.285 172.30.99.31 965a 0.000931010 0.000158095 0.187527895 0.87612 ... VM migrated to "new" host 57581 41675.500 172.30.99.31 965a -6.986332827 0.000539032 0.062950133 6.987199663 57581 41744.489 172.30.99.31 905a -6.98823 0.000480657 0.000739123 6.467141976 57581 41811.509 172.30.99.31 905a -6.982751485 0.000298934 0.000864689 5.902246019 57581 41879.519 172.30.99.31 905a -6.980983259 0.000466260 0.000933010 5.277805891 57581 41945.518 172.30.99.31 905a -6.979267346 0.000409787 0.000950354 4.569599170 57581 42014.502 172.30.99.31 905a -6.977464766 0.000497814 0.000979475 3.730107446 57581 42080.531 172.30.99.31 905a -6.975764545 0.000413500 0.000969749 2.636950596 57581 42148.485 172.30.99.31 905a -6.973984885 0.000336825 0.000977892 0.007845778 57581 42211.541 172.30.99.31 9024 0.036648577 0.000355228 7.937500599 0.00238 57581 42213.518 172.30.99.31 963a 0.037403812 0.000173852 3.937508397 0.000755235 ... VM still running on the "new" host, now synced Regards, Christian ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
--On 03 November 2015 12:21 +0100 Roger Pau Monné wrote: Is XenServer also synchronized using ntpd? When a guest resumes from migration is updates it's current time based on the time provided by the hypervisor, so if the hosts themselves are not synchronized you could see skews like that. I've tested this on two pools now. The original production pool (Xeon E3-1220 v3 @ 3.10GHz) has the issue. Our office / test pool (Xeon L5630 @ 2.13Ghz) also -has the issue-. Both are setup pretty much identical (same storage solution, both running FreeBSD 10.1/10.2-R with 'xe-guest-utilities' installed). Running or not running NTP doesn't make a difference. Live migrates result in the domU gaining several seconds (thus breaking NTP if it is running) else leaving the host 'running in the future' vs. both external clocks, and NTP sync'd clocks on the xenserver. As it's easily reproducible I guess I'll file it as a bug... -Karl ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
--On 3 November 2015 at 12:21:20 +0100 Roger Pau Monné wrote: Can't you tell ntpd to just resynchronize no matter what skew it detects? Probably - I'll have to do some digging - I had hoped someone had hit this already, though I'm also a little concerned about throwing a 7 second 'back in time' jump on the system. Are either of those fixes likely to cure this? Hm, I don't think so. Both commits are present from 4.3 onwards, and IIRC XenServer 6.5 is based on Xen 4.4, which should contain both fixes. Is XenServer also synchronized using ntpd? When a guest resumes from migration is updates it's current time based on the time provided by the hypervisor, so if the hosts themselves are not synchronized you could see skews like that. All the clocks on the XenServers are NTP sync'd - and are in sync (confirmed from the consoles). When the domU moves - it's clock moves >ahead< in time of both the server it's leaving, and the server it's moved to. I re-ran the test adding checks for the source (Xen1) and destination (Xen2) XenServer clocks - and I get: Me: Tue Nov 3 18:56:29 GMT 2015 Them: Tue Nov 3 18:56:29 GMT 2015 Xen1: Tue Nov 3 18:56:29 GMT 2015 Xen2: Tue Nov 3 18:56:29 GMT 2015 (All in sync) [moving FreeBSD from Xen1 to Xen2] Me: Tue Nov 3 18:56:39 GMT 2015 (Gained 4 seconds) Them: Tue Nov 3 18:56:35 GMT 2015 Xen1: Tue Nov 3 18:56:35 GMT 2015 Xen2: Tue Nov 3 18:56:35 GMT 2015 Me: Tue Nov 3 18:56:43 GMT 2015 (Completed - gained 7 seconds total) Them: Tue Nov 3 18:56:36 GMT 2015 Xen1: Tue Nov 3 18:56:36 GMT 2015 Xen2: Tue Nov 3 18:56:36 GMT 2015 Me: Tue Nov 3 18:56:44 GMT 2015 Them: Tue Nov 3 18:56:37 GMT 2015 Xen1: Tue Nov 3 18:56:37 GMT 2015 Xen2: Tue Nov 3 18:56:37 GMT 2015 So you can see it's now 7 seconds ahead of both 'Them' (external bare metal machine running ntp) and Xen1/Xen2. I re-ran the test without ntp loaded on the domU - and the same thing happens - so it's not ntp causing it. We've kind of 'lived with it' until now - but as the quantity of VM's increases - it becomes more of a pain :( -Karl ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
El 03/11/15 a les 11.03, Karl Pielorz ha escrit: > > Hi, > > Ok, I've done some proper research into this now... > > When Migrating a domU FreeBSD (10.1-R p6) between hosts, it "more often > than not" gains time. > > This breaks NTP on the domU. > > I setup a script on the domU being migrated that read out it's clock - > and called out to an external machine to read it's clock - i.e. > > " > Me: Tue Nov 3 09:55:51 GMT 2015 > Them: Tue Nov 3 09:55:51 GMT 2015 > > Me: Tue Nov 3 09:55:52 GMT 2015 > Them: Tue Nov 3 09:55:52 GMT 2015 > " > > As you can see 'Me' and 'Them' is in sync. > > At this point I live migrated the FreeBSD box from one XenServer > (6.5-SP1 w/hotfixes) to another in the pool (same hardware spec / > XenServer etc.) > > The domU's clock got advanced by a number of seconds: > > " > Me: Tue Nov 3 09:55:52 GMT 2015 > Them: Tue Nov 3 09:55:52 GMT 2015 > > [Migrate 'Me' Happens Now] > > Me: Tue Nov 3 09:56:02 GMT 2015 > Them: Tue Nov 3 09:55:58 GMT 2015 > > Me: Tue Nov 3 09:56:06 GMT 2015 > Them: Tue Nov 3 09:55:59 GMT 2015 > > Me: Tue Nov 3 09:56:07 GMT 2015 > Them: Tue Nov 3 09:56:00 GMT 2015 > > Me: Tue Nov 3 09:56:08 GMT 2015 > Them: Tue Nov 3 09:56:01 GMT 2015 > " > > So you can see during the migrate the 'Me' (domU being moved) clock > picks up 3 seconds, then another 4 seconds to make it 'ahead' 7 seconds > in total - at which point it stabilises). Can't you tell ntpd to just resynchronize no matter what skew it detects? > It stays this way then - which is enough to cause NTP to "give up" > trying to bring that back into alignment. > >> http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=f8e8fd56bd7d5675e8331 >> b4ec74bae76c9dbf24e >> http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=32c864a35ece2c24a336d >> 183869a546798a4b241 > > Are either of those fixes likely to cure this? Hm, I don't think so. Both commits are present from 4.3 onwards, and IIRC XenServer 6.5 is based on Xen 4.4, which should contain both fixes. Is XenServer also synchronized using ntpd? When a guest resumes from migration is updates it's current time based on the time provided by the hypervisor, so if the hosts themselves are not synchronized you could see skews like that. Roger. ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
Hi, Ok, I've done some proper research into this now... When Migrating a domU FreeBSD (10.1-R p6) between hosts, it "more often than not" gains time. This breaks NTP on the domU. I setup a script on the domU being migrated that read out it's clock - and called out to an external machine to read it's clock - i.e. " Me: Tue Nov 3 09:55:51 GMT 2015 Them: Tue Nov 3 09:55:51 GMT 2015 Me: Tue Nov 3 09:55:52 GMT 2015 Them: Tue Nov 3 09:55:52 GMT 2015 " As you can see 'Me' and 'Them' is in sync. At this point I live migrated the FreeBSD box from one XenServer (6.5-SP1 w/hotfixes) to another in the pool (same hardware spec / XenServer etc.) The domU's clock got advanced by a number of seconds: " Me: Tue Nov 3 09:55:52 GMT 2015 Them: Tue Nov 3 09:55:52 GMT 2015 [Migrate 'Me' Happens Now] Me: Tue Nov 3 09:56:02 GMT 2015 Them: Tue Nov 3 09:55:58 GMT 2015 Me: Tue Nov 3 09:56:06 GMT 2015 Them: Tue Nov 3 09:55:59 GMT 2015 Me: Tue Nov 3 09:56:07 GMT 2015 Them: Tue Nov 3 09:56:00 GMT 2015 Me: Tue Nov 3 09:56:08 GMT 2015 Them: Tue Nov 3 09:56:01 GMT 2015 " So you can see during the migrate the 'Me' (domU being moved) clock picks up 3 seconds, then another 4 seconds to make it 'ahead' 7 seconds in total - at which point it stabilises). It stays this way then - which is enough to cause NTP to "give up" trying to bring that back into alignment. http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=f8e8fd56bd7d5675e8331 b4ec74bae76c9dbf24e http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=32c864a35ece2c24a336d 183869a546798a4b241 Are either of those fixes likely to cure this? Thanks, -Karl ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
--On 2 November 2015 at 18:40:43 +0100 Roger Pau Monné wrote: I'm sorry, but I have no idea about ntpd, what does the above mean? The offset is too big with other peers and ntpd simply disconnects? There were some issues that I fixed some time ago regarding migration and PVHVM guests: I'll work out (more scientifically) what's actually happening tomorrow - rather than now, while I'm trying to get the servers all patched / updated out of hours. It's pretty reproducible - so I'll run some proper tests that show what's happening (to the VM's time, and to ntpd) and post back as it doesn't there's a known fix / workaround etc. -Karl ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
--On 2 November 2015 at 18:40:43 +0100 Roger Pau Monné wrote: remoterefid st t when poll reach delay offset jitter === ntp0 193.67.79.2022 u 931 1024 3770.232 -4977.5 4207.29 ntp1 66.228.38.733 u 890 1024 3770.267 -4988.5 4616.94 ntp2 64.246.132.142 u 990 1024 3770.324 -4978.0 4207.98 " I'm sorry, but I have no idea about ntpd, what does the above mean? The offset is too big with other peers and ntpd simply disconnects? Sorry for being so vague (I half hoped someone must have run into this before and posted a "Yes, we noticed the same - this fixes it" reply :) The above seems to show that time moved forward on the VM by a few seconds. This doesn't always happen - I just moved another VM and that shows 'offset = 8000'. Yet another VM moved at the same time is fine. Comparing the output of 'date' for that offset 8000 VM with a known 'good' clock - it shows the VM is indeed 8 seconds in the future after the move. ntp is fussy - and kind of understandably decides that's too much time difference to slowly ebb away - so just gives up and leaves the VM clock 'as is'. The XenServer the VM was residing on (and the new one it's resident on) both on their server consoles show the correct time (i.e. neither are 8 seconds 'ahead') - so it looks like the move somehow gained us 8 seconds (or 5 seconds in the original example). There were some issues that I fixed some time ago regarding migration and PVHVM guests: http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=f8e8fd56bd7d5675e8331 b4ec74bae76c9dbf24e http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=32c864a35ece2c24a336d 183869a546798a4b241 You should make sure XenServer has both of this commits. I'll have a look at those - we might be a bit stuck for implementing them if they're XenServer changes, as we run the XenServer ISO's (6.5 SP1 w/hotfixes). -Karl ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
Re: 'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
El 02/11/15 a les 16.09, Karl Pielorz ha escrit: > > Hi, > > We run a number of 10.x boxes under XenServer 6.5. On those boxes we run > NTP as a client with a few local time servers on the network to keep > the time in sync on that VM. > > There's a number of big threads on the Citrix forums about whether you > should, or shouldn't run NTP in domU's - but the consensus seems to be > you need to in order to keep the domU's clock accurately in sync. > > This works fine. > > However if we run a live migrate (to another XenServer node) or Xen > Storage Motion migration - the domU's ntp gets messed up. The time on > the domU is generally "ok" - but ntpq gives output like: > > " > ntpq> pe > remoterefid st t when poll reach delay offset jitter > === > ntp0 193.67.79.2022 u 931 1024 3770.232 -4977.5 4207.29 > ntp1 66.228.38.733 u 890 1024 3770.267 -4988.5 4616.94 > ntp2 64.246.132.142 u 990 1024 3770.324 -4978.0 4207.98 > " I'm sorry, but I have no idea about ntpd, what does the above mean? The offset is too big with other peers and ntpd simply disconnects? There were some issues that I fixed some time ago regarding migration and PVHVM guests: http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=f8e8fd56bd7d5675e8331b4ec74bae76c9dbf24e http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=32c864a35ece2c24a336d183869a546798a4b241 You should make sure XenServer has both of this commits. Roger. ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
'Live' Migrate messes up NTP on FreeBSD domU - any suggestions?
Hi, We run a number of 10.x boxes under XenServer 6.5. On those boxes we run NTP as a client with a few local time servers on the network to keep the time in sync on that VM. There's a number of big threads on the Citrix forums about whether you should, or shouldn't run NTP in domU's - but the consensus seems to be you need to in order to keep the domU's clock accurately in sync. This works fine. However if we run a live migrate (to another XenServer node) or Xen Storage Motion migration - the domU's ntp gets messed up. The time on the domU is generally "ok" - but ntpq gives output like: " ntpq> pe remoterefid st t when poll reach delay offset jitter === ntp0 193.67.79.2022 u 931 1024 3770.232 -4977.5 4207.29 ntp1 66.228.38.733 u 890 1024 3770.267 -4988.5 4616.94 ntp2 64.246.132.142 u 990 1024 3770.324 -4978.0 4207.98 " It *never* recovers from this (you have to log into the box, and restart ntpd). I realise this may be more a 'ntpd' than Xen question - but does anyone have any suggestions for avoiding this happening? - i.e. Some option for ntp.conf or something? Thanks, -Karl ___ freebsd-xen@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
