Re: bhyve clock issue
Il 26/04/16 16:50, Neel Natu ha scritto: What was the timecounter being used when you were seeing the time skew? HPET -- *Ing. Ilario Febi Schema 31 S.p.a. Socio Unico* *System Administrator *BO - FI - ROMA - PA ITALY Tel: +39. 06.98.358.472 Cell: +39. 366.69.45.384 Fax: +39. 055.71.880.466 Società del Gruppo *SC31 ITALIA* ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve clock issue
What was the timecounter being used when you were seeing the time skew? Best Neel > On Apr 26, 2016, at 6:46 AM, Ilario Febiwrote: > > Yes, > > I found errors in /var/log/messages and fix adding: > > kern.timecounter.hardware=TSC-low > into /etc/sysctl.conf file > > Thanks for support. > > > > Il 14/04/16 18:13, Neel Natu ha scritto: >> Hi, >> >> Did you see anything interesting in the ntpd log file around the time the >> large skew happened? >> >> Best >> Neel >> >>> On Apr 13, 2016, at 3:44 AM, Ilario Febi wrote: >>> >>> Hi All >>> >>> I have a strange issue about the clock of some FreeBSD Bhyve guests, >>> maybe someone of you can help me to diagnose it and work on a >>> solution. >>> >>> In detail, sometimes (without an apparent regularity) one of four >>> bhyve guest's clock goes exactly 7 minutes and 10 seconds in the >>> future. >>> >>> This clock stays unaligned for about 20 minutes, then, without any >>> human intervention, the clock realign to proper condition. >>> >>> My configuration is: >>> >>> envrm02 (bhyve HOST, 10.2-RELEASE-p14) >>> -> og-qa (bhyve guest - 10.2-RELEASE-p14) >>> -> og-ci (bhyve guest - 10.2-RELEASE-p14) >>> -> og-test (bhyve guest - 10.2-RELEASE-p14) >>> -> og-prod-rm (bhyve guest - 10.2-RELEASE-p14) >>> All the og-* machines just have a running copy of tomcat. >>> >>> I built a simple test to catch the error with python/fabric... the test is: >>> >>> fab -R ENV,OG,OG-PROD -- date | awk -F ': ' '/2016/{print $2, $1}' >>> >>> which simply run the command "date" on all hosts. >>> >>> So, the result is: >>> @Mac-mini ~/p/f/> ./check_clock.sh >>> Thu Apr 7 17:38:00 CEST 2016 [envrm02] out >>> Thu Apr 7 17:38:00 CEST 2016 [og-qa] out >>> Thu Apr 7 17:38:00 CEST 2016 [og-ci] out >>> Thu Apr 7 17:38:00 CEST 2016 [og-test] out >>> Thu Apr 7 17:38:00 CEST 2016 [og-prod-rm] out >>> >>> @Mac-mini ~/p/f/> ./check_clock.sh >>> Thu Apr 7 18:38:00 CEST 2016 [envrm02] out >>> Thu Apr 7 18:45:10 CEST 2016 [og-qa] out >>> Thu Apr 7 18:38:00 CEST 2016 [og-ci] out >>> Thu Apr 7 18:38:00 CEST 2016 [og-test] out >>> Thu Apr 7 18:38:00 CEST 2016 [og-prod-rm] out >>> >>> @Mac-mini ~/p/f/> ./check_clock.sh >>> Thu Apr 7 18:41:05 CEST 2016 [envrm02] out >>> Thu Apr 7 18:48:15 CEST 2016 [og-qa] out >>> Thu Apr 7 18:41:05 CEST 2016 [og-ci] out >>> Thu Apr 7 18:41:05 CEST 2016 [og-test] out >>> Thu Apr 7 18:41:05 CEST 2016 [og-prod-rm] out >>> >>> @Mac-mini ~/p/f/> ./check_clock.sh >>> Thu Apr 7 18:46:51 CEST 2016 [envrm02] out >>> Thu Apr 7 18:46:52 CEST 2016 [og-qa] out >>> Thu Apr 7 18:46:52 CEST 2016 [og-ci] out >>> Thu Apr 7 18:46:53 CEST 2016 [og-test] out >>> Thu Apr 7 18:46:53 CEST 2016 [og-prod-rm] out >>> >>> The configuration is the same for all hosts and is this: >>> @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/rc.conf | grep ntpd >>> [envrm02] out: ntpd_enable="YES" >>> [og-qa] out: ntpd_enable="YES" >>> [og-qa] out: ntpd_sync_on_start="YES" >>> [og-ci] out: ntpd_enable="YES" >>> [og-ci] out: ntpd_sync_on_start="YES" >>> [og-test] out: ntpd_enable="YES" >>> [og-test] out: ntpd_sync_on_start="YES" >>> [og-prod-rm] out: ntpd_enable="YES" >>> [og-prod-rm] out: ntpd_sync_on_start="YES" >>> >>> @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/ntp.conf | grep -v >>> 'out: #' >>> [envrm02] Executing task '' >>> [envrm02] run: cat /etc/ntp.conf >>> [envrm02] out: >>> [envrm02] out: server 0.freebsd.pool.ntp.org iburst >>> [envrm02] out: server 1.freebsd.pool.ntp.org iburst >>> [envrm02] out: server 2.freebsd.pool.ntp.org iburst >>> [envrm02] out: >>> [envrm02] out: restrict default limited kod nomodify notrap nopeer noquery >>> [envrm02] out: restrict -6 default limited kod nomodify notrap nopeer >>> noquery >>> [envrm02] out: restrict 127.0.0.1 >>> [envrm02] out: restrict -6 ::1 >>> [envrm02] out: restrict 127.127.1.0 >>> >>> [og-qa] Executing task '' >>> [og-qa] run: cat /etc/ntp.conf >>> [og-qa] out: >>> [og-qa] out: server 0.freebsd.pool.ntp.org iburst >>> [og-qa] out: server 1.freebsd.pool.ntp.org iburst >>> [og-qa] out: server 2.freebsd.pool.ntp.org iburst >>> [og-qa] out: >>> [og-qa] out: restrict default limited kod nomodify notrap nopeer noquery >>> [og-qa] out: restrict -6 default limited kod nomodify notrap nopeer noquery >>> [og-qa] out: restrict 127.0.0.1 >>> [og-qa] out: restrict -6 ::1 >>> [og-qa] out: restrict 127.127.1.0 >>> >>> [og-ci] Executing task '' >>> [og-ci] run: cat /etc/ntp.conf >>> [og-ci] out: >>> [og-ci] out: server 0.freebsd.pool.ntp.org iburst >>> [og-ci] out: server 1.freebsd.pool.ntp.org iburst >>> [og-ci] out: server 2.freebsd.pool.ntp.org iburst >>> [og-ci] out: >>> [og-ci] out: restrict default limited kod nomodify notrap nopeer noquery >>> [og-ci] out: restrict -6 default limited kod nomodify notrap nopeer noquery >>> [og-ci] out: restrict 127.0.0.1 >>> [og-ci] out: restrict -6 ::1 >>> [og-ci] out: restrict 127.127.1.0 >>> >>> [og-test] Executing task ''
Re: bhyve clock issue
Yes, I found errors in /var/log/messages and fix adding: kern.timecounter.hardware=TSC-low into /etc/sysctl.conf file Thanks for support. Il 14/04/16 18:13, Neel Natu ha scritto: Hi, Did you see anything interesting in the ntpd log file around the time the large skew happened? Best Neel On Apr 13, 2016, at 3:44 AM, Ilario Febiwrote: Hi All I have a strange issue about the clock of some FreeBSD Bhyve guests, maybe someone of you can help me to diagnose it and work on a solution. In detail, sometimes (without an apparent regularity) one of four bhyve guest's clock goes exactly 7 minutes and 10 seconds in the future. This clock stays unaligned for about 20 minutes, then, without any human intervention, the clock realign to proper condition. My configuration is: envrm02 (bhyve HOST, 10.2-RELEASE-p14) -> og-qa (bhyve guest - 10.2-RELEASE-p14) -> og-ci (bhyve guest - 10.2-RELEASE-p14) -> og-test (bhyve guest - 10.2-RELEASE-p14) -> og-prod-rm (bhyve guest - 10.2-RELEASE-p14) All the og-* machines just have a running copy of tomcat. I built a simple test to catch the error with python/fabric... the test is: fab -R ENV,OG,OG-PROD -- date | awk -F ': ' '/2016/{print $2, $1}' which simply run the command "date" on all hosts. So, the result is: @Mac-mini ~/p/f/> ./check_clock.sh Thu Apr 7 17:38:00 CEST 2016 [envrm02] out Thu Apr 7 17:38:00 CEST 2016 [og-qa] out Thu Apr 7 17:38:00 CEST 2016 [og-ci] out Thu Apr 7 17:38:00 CEST 2016 [og-test] out Thu Apr 7 17:38:00 CEST 2016 [og-prod-rm] out @Mac-mini ~/p/f/> ./check_clock.sh Thu Apr 7 18:38:00 CEST 2016 [envrm02] out Thu Apr 7 18:45:10 CEST 2016 [og-qa] out Thu Apr 7 18:38:00 CEST 2016 [og-ci] out Thu Apr 7 18:38:00 CEST 2016 [og-test] out Thu Apr 7 18:38:00 CEST 2016 [og-prod-rm] out @Mac-mini ~/p/f/> ./check_clock.sh Thu Apr 7 18:41:05 CEST 2016 [envrm02] out Thu Apr 7 18:48:15 CEST 2016 [og-qa] out Thu Apr 7 18:41:05 CEST 2016 [og-ci] out Thu Apr 7 18:41:05 CEST 2016 [og-test] out Thu Apr 7 18:41:05 CEST 2016 [og-prod-rm] out @Mac-mini ~/p/f/> ./check_clock.sh Thu Apr 7 18:46:51 CEST 2016 [envrm02] out Thu Apr 7 18:46:52 CEST 2016 [og-qa] out Thu Apr 7 18:46:52 CEST 2016 [og-ci] out Thu Apr 7 18:46:53 CEST 2016 [og-test] out Thu Apr 7 18:46:53 CEST 2016 [og-prod-rm] out The configuration is the same for all hosts and is this: @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/rc.conf | grep ntpd [envrm02] out: ntpd_enable="YES" [og-qa] out: ntpd_enable="YES" [og-qa] out: ntpd_sync_on_start="YES" [og-ci] out: ntpd_enable="YES" [og-ci] out: ntpd_sync_on_start="YES" [og-test] out: ntpd_enable="YES" [og-test] out: ntpd_sync_on_start="YES" [og-prod-rm] out: ntpd_enable="YES" [og-prod-rm] out: ntpd_sync_on_start="YES" @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/ntp.conf | grep -v 'out: #' [envrm02] Executing task '' [envrm02] run: cat /etc/ntp.conf [envrm02] out: [envrm02] out: server 0.freebsd.pool.ntp.org iburst [envrm02] out: server 1.freebsd.pool.ntp.org iburst [envrm02] out: server 2.freebsd.pool.ntp.org iburst [envrm02] out: [envrm02] out: restrict default limited kod nomodify notrap nopeer noquery [envrm02] out: restrict -6 default limited kod nomodify notrap nopeer noquery [envrm02] out: restrict 127.0.0.1 [envrm02] out: restrict -6 ::1 [envrm02] out: restrict 127.127.1.0 [og-qa] Executing task '' [og-qa] run: cat /etc/ntp.conf [og-qa] out: [og-qa] out: server 0.freebsd.pool.ntp.org iburst [og-qa] out: server 1.freebsd.pool.ntp.org iburst [og-qa] out: server 2.freebsd.pool.ntp.org iburst [og-qa] out: [og-qa] out: restrict default limited kod nomodify notrap nopeer noquery [og-qa] out: restrict -6 default limited kod nomodify notrap nopeer noquery [og-qa] out: restrict 127.0.0.1 [og-qa] out: restrict -6 ::1 [og-qa] out: restrict 127.127.1.0 [og-ci] Executing task '' [og-ci] run: cat /etc/ntp.conf [og-ci] out: [og-ci] out: server 0.freebsd.pool.ntp.org iburst [og-ci] out: server 1.freebsd.pool.ntp.org iburst [og-ci] out: server 2.freebsd.pool.ntp.org iburst [og-ci] out: [og-ci] out: restrict default limited kod nomodify notrap nopeer noquery [og-ci] out: restrict -6 default limited kod nomodify notrap nopeer noquery [og-ci] out: restrict 127.0.0.1 [og-ci] out: restrict -6 ::1 [og-ci] out: restrict 127.127.1.0 [og-test] Executing task '' [og-test] run: cat /etc/ntp.conf [og-test] out: [og-test] out: server 0.freebsd.pool.ntp.org iburst [og-test] out: server 1.freebsd.pool.ntp.org iburst [og-test] out: server 2.freebsd.pool.ntp.org iburst [og-test] out: [og-test] out: restrict default limited kod nomodify notrap nopeer noquery [og-test] out: restrict -6 default limited kod nomodify notrap nopeer noquery [og-test] out: restrict 127.0.0.1 [og-test] out: restrict -6 ::1 [og-test] out: restrict 127.127.1.0 Some observation: * The clock of envrm02 (the HOST) always seems to be correct. * The issue happens on a (apparently) random guest (not always the same one). * The issue happens always on
Re: Booting Windows/UEFI (was Re: Booting r298488 as Xen Dom0 may break ZFS pool?)
On Tue, Apr 26, 2016 at 10:39:06AM +, Marcin Cieslak wrote: > Yes, should be easy. Will try to plug a binary port in. Thanks :). > By the way, I just managed to crash a whole machine by trying to > boot Solaris 11.3: > > # builder = "hvm" > memory = 2048 > vcpus = 1 > name = "Solaris0" > disk = [ 'file:/root/sol-11_3-text-x86.iso,6:cdrom,r', > '/dev/zvol/zroot/solaris0,raw,hda,w' ] > kernel = "/root/xen/solaris/unix" > ramdisk = "/root/xen/solaris/boot_archive" > extra = "/platform/i86xpv/kernel/amd64/unix -B console=ttya,livemode=text" > #boot = "c" > usbdevice = 'tablet' > vnc = 1 > vnclisten = '0.0.0.0' > vif = [ 'bridge=bridge0' ] > > /root/xen/solaris/unix and /root/xen/solaris/boot_archive where extracted > from sol-11_3-text-x86.iso > ("Intel text-only image"). I don't think you can boot Solaris as a PV guest anymore, you should instead boot it as a HVM guest. You will have to remove the kernel and ramdisk options and instead add builder="hvm" (that you have left commented out). Roger. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Booting Windows/UEFI (was Re: Booting r298488 as Xen Dom0 may break ZFS pool?)
On Tue, 26 Apr 2016, Roger Pau Monné wrote: > On Tue, Apr 26, 2016 at 09:24:30AM +, Marcin Cieslak wrote: > > I got lazy and I have downloaded > > > > https://sourceforge.net/projects/edk2/files/OVMF/OVMF-X64-r15214.zip/download > > > > recompiled xen-tools by adding to xen-tools/Makefile > > > > CONFIGURE_ARGS+=--enable-ovmf > > CONFIGURE_ARGS+=--with-system-ovmf=/root/xen/OVMF.fd > > > > and worked pretty much out of the box. > > Oh, thanks for the info, I didn't know OVMF provided pre-compiled binaries. > This should make adding a OVMF port trivial. Would you like to take a stab > at adding such a port and wiring it into the xen-tools package? Yes, should be easy. Will try to plug a binary port in. By the way, I just managed to crash a whole machine by trying to boot Solaris 11.3: # builder = "hvm" memory = 2048 vcpus = 1 name = "Solaris0" disk = [ 'file:/root/sol-11_3-text-x86.iso,6:cdrom,r', '/dev/zvol/zroot/solaris0,raw,hda,w' ] kernel = "/root/xen/solaris/unix" ramdisk = "/root/xen/solaris/boot_archive" extra = "/platform/i86xpv/kernel/amd64/unix -B console=ttya,livemode=text" #boot = "c" usbdevice = 'tablet' vnc = 1 vnclisten = '0.0.0.0' vif = [ 'bridge=bridge0' ] /root/xen/solaris/unix and /root/xen/solaris/boot_archive where extracted from sol-11_3-text-x86.iso ("Intel text-only image"). Marcin ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Booting Windows/UEFI (was Re: Booting r298488 as Xen Dom0 may break ZFS pool?)
On Tue, Apr 26, 2016 at 09:24:30AM +, Marcin Cieslak wrote: > I got lazy and I have downloaded > > https://sourceforge.net/projects/edk2/files/OVMF/OVMF-X64-r15214.zip/download > > recompiled xen-tools by adding to xen-tools/Makefile > > CONFIGURE_ARGS+=--enable-ovmf > CONFIGURE_ARGS+=--with-system-ovmf=/root/xen/OVMF.fd > > and worked pretty much out of the box. Oh, thanks for the info, I didn't know OVMF provided pre-compiled binaries. This should make adding a OVMF port trivial. Would you like to take a stab at adding such a port and wiring it into the xen-tools package? Roger. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Booting Windows/UEFI (was Re: Booting r298488 as Xen Dom0 may break ZFS pool?)
On Tue, 26 Apr 2016, Marcin Cieslak wrote: > On Tue, 26 Apr 2016, Roger Pau Monné wrote: > > > You will have to boot it using OVMF, which is not compiled with Xen by > > default. I will try to add an OVMF package (like the SeaBIOS one that we > > already have), and wire it into the xen-tools package. > > > > If you want to try it yourself, you can add "--enable-ovmf" to the > > xen-tools > > package configure and see what breaks ;). > > Thanks, lots of bad Linux-only scripts (#!/bin/bash and the like), will try! I got lazy and I have downloaded https://sourceforge.net/projects/edk2/files/OVMF/OVMF-X64-r15214.zip/download recompiled xen-tools by adding to xen-tools/Makefile CONFIGURE_ARGS+=--enable-ovmf CONFIGURE_ARGS+=--with-system-ovmf=/root/xen/OVMF.fd and worked pretty much out of the box. This is Windows Server 2016 Technical Preview 2016, with GPL PV drivers added to get networking. My config: builder = "hvm" memory = 4096 vcpus = 2 name = "Windows" disk = [ '/dev/zvol/zroot/windows0,raw,hda,w','/root/fat.bin,raw,hdb,w' ] boot = "c" # Boot to hard disk image vnc = 2 vnclisten = "0.0.0.0" usbdevice = 'tablet' on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart' acpi = 1 bios = 'ovmf' vif = [ 'bridge=bridge0' ] /root/fat.bin is a little partitioned "hard drive" with FAT16 partition used to provide GPL drivers. Marcin ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Booting r298488 as Xen Dom0 may break ZFS pool?
On Tue, 26 Apr 2016, Roger Pau Monné wrote: > You will have to boot it using OVMF, which is not compiled with Xen by > default. I will try to add an OVMF package (like the SeaBIOS one that we > already have), and wire it into the xen-tools package. > > If you want to try it yourself, you can add "--enable-ovmf" to the xen-tools > package configure and see what breaks ;). Thanks, lots of bad Linux-only scripts (#!/bin/bash and the like), will try! Marcin ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Booting r298488 as Xen Dom0 may break ZFS pool?
On Tue, Apr 26, 2016 at 07:16:25AM +, Marcin Cieslak wrote: > On Tue, 26 Apr 2016, Roger Pau Monné wrote: > > > > > libxl: error: libxl.c:6121:libxl_xen_console_read_line: reading console > > > > ring buffer: Cannot allocate memory > > > > > > It seems that my problem was ... > > > > > > vm.max_wired=1 > > > > > > in /boot/loader.conf > > > > > > instead of > > > > > > vm.max_wired=-1 > > > > And this also caused the ZFS corruption? > > Looks like it to me... Pretty strange. I thought the max_wired sysctl node only affected the usage of wired memory by userspace applications, but it may have also prevented ZFS from using memory. > Btw. is there any way to boot UEFI Windows partition under Xen? > (it works fine under bhyve) You will have to boot it using OVMF, which is not compiled with Xen by default. I will try to add an OVMF package (like the SeaBIOS one that we already have), and wire it into the xen-tools package. If you want to try it yourself, you can add "--enable-ovmf" to the xen-tools package configure and see what breaks ;). Roger. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Booting r298488 as Xen Dom0 may break ZFS pool?
On Tue, 26 Apr 2016, Roger Pau Monné wrote: > > > libxl: error: libxl.c:6121:libxl_xen_console_read_line: reading console > > > ring buffer: Cannot allocate memory > > > > It seems that my problem was ... > > > > vm.max_wired=1 > > > > in /boot/loader.conf > > > > instead of > > > > vm.max_wired=-1 > > And this also caused the ZFS corruption? Looks like it to me... Pretty strange. Btw. is there any way to boot UEFI Windows partition under Xen? (it works fine under bhyve) Marcin ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: Booting r298488 as Xen Dom0 may break ZFS pool?
On Mon, Apr 25, 2016 at 08:50:53PM +, Marcin Cieslak wrote: > On Mon, 25 Apr 2016, Marcin Cieslak wrote: > > > On Mon, 25 Apr 2016, Marcin Cieslak wrote: > > > > > On Mon, 25 Apr 2016, Roger Pau Monné wrote: > > > > TBH, I have no idea. Can you also paste the log of the system (Xen + > > > > FreeBSD) when it fails to boot? If that's not possible, can you at > > > > least > > > > paste the output of `xl dmesg` when booted correctly under Xen? > > > > I managed to boot it again... > > > > root@o:~ # xl dmesg > > xc: error: Could not bounce buffer for version hypercall (35 = Resource > > temporarily unavailabl): Internal error > > xc: error: Could not bounce buffer for version hypercall (35 = Resource > > temporarily unavailabl): Internal error > > xc: error: Could not bounce buffer for version hypercall (35 = Resource > > temporarily unavailabl): Internal error > > xc: error: Could not bounce buffer for version hypercall (35 = Resource > > temporarily unavailabl): Internal error > > xc: error: Could not bounce buffer for version hypercall (35 = Resource > > temporarily unavailabl): Internal error > > xc: error: Could not bounce buffer for version hypercall (35 = Resource > > temporarily unavailabl): Internal error > > libxl: error: libxl.c:6121:libxl_xen_console_read_line: reading console > > ring buffer: Cannot allocate memory > > It seems that my problem was ... > > vm.max_wired=1 > > in /boot/loader.conf > > instead of > > vm.max_wired=-1 And this also caused the ZFS corruption? Roger. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"