Re: host, bhyve vm and ntpd

2017-10-24 Thread Boris Samorodov
25.10.2017 01:14, Ian Lepore пишет:

> Can you show the /var/db/ntpd.drift file contents of the host and
> guest?  Ideally, now that it's stable, the two values should be very
> close.  If they're not, maybe this isn't the right fix.

Sorry, no. :-( I experimented with the host and bhyve vm now has
unstable ntp values (stepping). I'll try to revert all my changes
and report back in a couple of days.

-- 
WBR, bsam
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: host, bhyve vm and ntpd

2017-10-24 Thread Ian Lepore
On Wed, 2017-10-25 at 00:43 +0300, Boris Samorodov wrote:
> Hi Ian, All!
> 
> 22.10.2017 01:15, Ian Lepore пишет:
> > 
> > On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote:
> > > 
> > > Ian Lepore writes:
> > > > 
> > > > 
> > > > Beyond that, I'm not sure what else to try.  It might be necessary to
> > > > get some bhyve developers involved (I know almost nothing about it).
> > > NTPD behaves more normally on uniprocessor VMs.
> > > 
> > > A FreeBSD bhyve-guest running on a freebsd host will select a
> > > different timecounter depending on whether it is a multiprocessor or a
> > > uniprocessor.  My uniprocessor bhyve-vm selected TSC-low as the best
> > > timecounter in a uniprocessor.  NTP functions there as expected.
> > > 
> > > kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) 
> > > dummy(-100)
> > > kern.timecounter.hardware: TSC-low
> > > 
> > > The very same VM, when given two total CPUs, selected HPET (if I
> > > recall) and the timekeeping with NTPD was unreliable, with many
> > > step-resets to the clock.
> > > 
> > Hmm, I just had glance at the code in sys/amd64/vmm/io/vhpet.c and it
> > looks right.  I wonder if this is just a simple roundoff error in
> > converting between 10.0MHz and SBT units?  If so, that could be wished
> > away easily by using a power-of-2 frequency for the virtual HPET.  I
> > wonder if the attached patch is all that's needed?
> I suppose the answer is "yes", the patch helped. Here are two samples
> (host for bhyve VM without your patch and after patching):
> ---
> https://poudriere.passap.ru/misc/ntpd.jot.log-HPET.frequency.1000.txt
> https://poudriere.passap.ru/misc/ntpd.jot.log-HPET.frequency.16777216.txt
> ---
> 
> The command was:
> % for t in `jot 1000`; do ntpq -pn; sleep 64; done
> 
> The patch made the system to stabilize the process.
> Ian, thank you!
> 

Hmmm.  The startup behavior wasn't great, it took a long time and
several clock steps for it to figure out the frequency error and get
the clock under control.  But, as you say, it did eventually stabilize
this time.

Can you show the /var/db/ntpd.drift file contents of the host and
guest?  Ideally, now that it's stable, the two values should be very
close.  If they're not, maybe this isn't the right fix.

-- Ian
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: host, bhyve vm and ntpd

2017-10-24 Thread Boris Samorodov
Hi Ian, All!

22.10.2017 01:15, Ian Lepore пишет:
> On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote:
>> Ian Lepore writes:
>>>
>>> Beyond that, I'm not sure what else to try.  It might be necessary to
>>> get some bhyve developers involved (I know almost nothing about it).
>> NTPD behaves more normally on uniprocessor VMs.
>>
>> A FreeBSD bhyve-guest running on a freebsd host will select a
>> different timecounter depending on whether it is a multiprocessor or a
>> uniprocessor.  My uniprocessor bhyve-vm selected TSC-low as the best
>> timecounter in a uniprocessor.  NTP functions there as expected.
>>
>> kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) 
>> dummy(-100)
>> kern.timecounter.hardware: TSC-low
>>
>> The very same VM, when given two total CPUs, selected HPET (if I
>> recall) and the timekeeping with NTPD was unreliable, with many
>> step-resets to the clock.
>>
> 
> Hmm, I just had glance at the code in sys/amd64/vmm/io/vhpet.c and it
> looks right.  I wonder if this is just a simple roundoff error in
> converting between 10.0MHz and SBT units?  If so, that could be wished
> away easily by using a power-of-2 frequency for the virtual HPET.  I
> wonder if the attached patch is all that's needed?

I suppose the answer is "yes", the patch helped. Here are two samples
(host for bhyve VM without your patch and after patching):
---
https://poudriere.passap.ru/misc/ntpd.jot.log-HPET.frequency.1000.txt
https://poudriere.passap.ru/misc/ntpd.jot.log-HPET.frequency.16777216.txt
---

The command was:
% for t in `jot 1000`; do ntpq -pn; sleep 64; done

The patch made the system to stabilize the process.
Ian, thank you!

-- 
WBR, bsam
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: host, bhyve vm and ntpd

2017-10-22 Thread Boris Samorodov
22.10.2017 19:02, Ian Lepore пишет:
> On Sun, 2017-10-22 at 11:31 +0300, Boris Samorodov wrote:
>> 22.10.2017 01:15, Ian Lepore пишет:
>>>
>>> On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote:

 Ian Lepore writes:
>
>
> Beyond that, I'm not sure what else to try.  It might be necessary to
> get some bhyve developers involved (I know almost nothing about it).
 NTPD behaves more normally on uniprocessor VMs.

 A FreeBSD bhyve-guest running on a freebsd host will select a
 different timecounter depending on whether it is a multiprocessor or a
 uniprocessor.  My uniprocessor bhyve-vm selected TSC-low as the best
 timecounter in a uniprocessor.  NTP functions there as expected.

 kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) 
 dummy(-100)
 kern.timecounter.hardware: TSC-low

 The very same VM, when given two total CPUs, selected HPET (if I
 recall) and the timekeeping with NTPD was unreliable, with many
 step-resets to the clock.

>>> Hmm, I just had glance at the code in sys/amd64/vmm/io/vhpet.c and it
>>> looks right.  I wonder if this is just a simple roundoff error in
>>> converting between 10.0MHz and SBT units?  If so, that could be wished
>>> away easily by using a power-of-2 frequency for the virtual HPET.  I
>>> wonder if the attached patch is all that's needed?
>> I've tried the patch (at bhyve guest) and nothing has changed. Should
>> the patched system be tested at bhyve guest or bhyve host?
>>
> 
> Oh, I'm sorry, I should have mentioned that's for the host side.

NP, that's OK.

However, the host is busy now, and I'll have an opportunity to test host
only tomorrow evening.

Ian, thank you for your help!

-- 
WBR, bsam
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: host, bhyve vm and ntpd

2017-10-22 Thread Ian Lepore
On Sun, 2017-10-22 at 11:31 +0300, Boris Samorodov wrote:
> 22.10.2017 01:15, Ian Lepore пишет:
> > 
> > On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote:
> > > 
> > > Ian Lepore writes:
> > > > 
> > > > 
> > > > Beyond that, I'm not sure what else to try.  It might be necessary to
> > > > get some bhyve developers involved (I know almost nothing about it).
> > > NTPD behaves more normally on uniprocessor VMs.
> > > 
> > > A FreeBSD bhyve-guest running on a freebsd host will select a
> > > different timecounter depending on whether it is a multiprocessor or a
> > > uniprocessor.  My uniprocessor bhyve-vm selected TSC-low as the best
> > > timecounter in a uniprocessor.  NTP functions there as expected.
> > > 
> > > kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) 
> > > dummy(-100)
> > > kern.timecounter.hardware: TSC-low
> > > 
> > > The very same VM, when given two total CPUs, selected HPET (if I
> > > recall) and the timekeeping with NTPD was unreliable, with many
> > > step-resets to the clock.
> > > 
> > Hmm, I just had glance at the code in sys/amd64/vmm/io/vhpet.c and it
> > looks right.  I wonder if this is just a simple roundoff error in
> > converting between 10.0MHz and SBT units?  If so, that could be wished
> > away easily by using a power-of-2 frequency for the virtual HPET.  I
> > wonder if the attached patch is all that's needed?
> I've tried the patch (at bhyve guest) and nothing has changed. Should
> the patched system be tested at bhyve guest or bhyve host?
> 

Oh, I'm sorry, I should have mentioned that's for the host side.

-- Ian
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: host, bhyve vm and ntpd

2017-10-22 Thread Boris Samorodov
22.10.2017 18:22, Rodney W. Grimes пишет:
> [ Charset UTF-8 unsupported, converting... ]
>> 22.10.2017 01:15, Ian Lepore ?:
>>> On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote:
 Ian Lepore writes:
>
> Beyond that, I'm not sure what else to try. ?It might be necessary to
> get some bhyve developers involved (I know almost nothing about it).
 NTPD behaves more normally on uniprocessor VMs.

 A FreeBSD bhyve-guest running on a freebsd host will select a
 different timecounter depending on whether it is a multiprocessor or a
 uniprocessor.??My uniprocessor bhyve-vm selected TSC-low as the best
 timecounter in a uniprocessor.??NTP functions there as expected.

 kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) 
 dummy(-100)
 kern.timecounter.hardware: TSC-low

 The very same VM, when given two total CPUs, selected HPET (if I
 recall) and the timekeeping with NTPD was unreliable, with many
 step-resets to the clock.

>>>
>>> Hmm, I just had glance at the code in?sys/amd64/vmm/io/vhpet.c and it
>>> looks right. ?I wonder if this is just a simple roundoff error in
>>> converting between 10.0MHz and SBT units? ?If so, that could be wished
>>> away easily by using a power-of-2 frequency for the virtual HPET. ?I
>>> wonder if the attached patch is all that's needed?
>> I've tried the patch (at bhyve guest) and nothing has changed. Should
>> the patched system be tested at bhyve guest or bhyve host?
> 
> I believe the suggested patch would have to be made to the bhyve
> host

OK, I'd do it tomorrow and report back.

>.  Also on the host and guest what are the values of
>   sysctl kern.timecounter.tc.HPET
>   sysctl kern.timecounter.tc.i8254

Here they are:
---
bhyve-host% sysctl kern.timecounter.tc.HPET
kern.timecounter.tc.HPET.quality: 950
kern.timecounter.tc.HPET.frequency: 14318180
kern.timecounter.tc.HPET.counter: 2138094157
kern.timecounter.tc.HPET.mask: 4294967295

bhyve-host% sysctl kern.timecounter.tc.i8254
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 54883
kern.timecounter.tc.i8254.mask: 65535
---
bhyve-guest% sysctl kern.timecounter.tc.HPET
kern.timecounter.tc.HPET.quality: 950
kern.timecounter.tc.HPET.frequency: 1000
kern.timecounter.tc.HPET.counter: 969429421
kern.timecounter.tc.HPET.mask: 4294967295

bhyve-guest% sysctl kern.timecounter.tc.i8254
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 39893
kern.timecounter.tc.i8254.mask: 65535
---

> Getting good ntpd behavior in a VM guest of any kind is sometimes a
> non trivial thing to do.

As a side note, I have a CentOS-7 bhyve VM at the same host.
And it was enough to run chronyd with default config. Which stepped
twice and is stable (no messages) for several days, current log:
---
Oct 19 16:01:03 c.vpn systemd[1]: Starting NTP client/server...
Oct 19 16:01:03 c.vpn chronyd[27043]: chronyd version 3.1 starting
(+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SECHASH +SIGND
+ASYNCDNS +IPV6 +DEBUG)
Oct 19 16:01:03 c.vpn chronyd[27043]: Frequency 0.000 +/- 100.000
ppm read from /var/lib/chrony/drift
Oct 19 16:01:03 c.vpn systemd[1]: Started NTP client/server.
Oct 19 16:01:07 c.vpn chronyd[27043]: Selected source XX.XX.XX.1
Oct 19 16:01:07 c.vpn chronyd[27043]: System clock wrong by -44.392782
seconds, adjustment started
Oct 19 16:00:23 c.vpn chronyd[27043]: System clock was stepped by
-44.392782 seconds
Oct 19 16:00:34 c.vpn chronyd[27043]: System clock was stepped by
0.01 seconds
---

-- 
WBR, bsam
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: host, bhyve vm and ntpd

2017-10-22 Thread Rodney W. Grimes
[ Charset UTF-8 unsupported, converting... ]
> 22.10.2017 01:15, Ian Lepore ?:
> > On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote:
> >> Ian Lepore writes:
> >>>
> >>> Beyond that, I'm not sure what else to try. ?It might be necessary to
> >>> get some bhyve developers involved (I know almost nothing about it).
> >> NTPD behaves more normally on uniprocessor VMs.
> >>
> >> A FreeBSD bhyve-guest running on a freebsd host will select a
> >> different timecounter depending on whether it is a multiprocessor or a
> >> uniprocessor.??My uniprocessor bhyve-vm selected TSC-low as the best
> >> timecounter in a uniprocessor.??NTP functions there as expected.
> >>
> >> kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) 
> >> dummy(-100)
> >> kern.timecounter.hardware: TSC-low
> >>
> >> The very same VM, when given two total CPUs, selected HPET (if I
> >> recall) and the timekeeping with NTPD was unreliable, with many
> >> step-resets to the clock.
> >>
> > 
> > Hmm, I just had glance at the code in?sys/amd64/vmm/io/vhpet.c and it
> > looks right. ?I wonder if this is just a simple roundoff error in
> > converting between 10.0MHz and SBT units? ?If so, that could be wished
> > away easily by using a power-of-2 frequency for the virtual HPET. ?I
> > wonder if the attached patch is all that's needed?
> I've tried the patch (at bhyve guest) and nothing has changed. Should
> the patched system be tested at bhyve guest or bhyve host?

I believe the suggested patch would have to be made to the bhyve
host.  Also on the host and guest what are the values of
sysctl kern.timecounter.tc.HPET
sysctl kern.timecounter.tc.i8254

Getting good ntpd behavior in a VM guest of any kind is sometimes a
non trivial thing to do.


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"


Re: host, bhyve vm and ntpd

2017-10-22 Thread Boris Samorodov
22.10.2017 01:15, Ian Lepore пишет:
> On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote:
>> Ian Lepore writes:
>>>
>>> Beyond that, I'm not sure what else to try.  It might be necessary to
>>> get some bhyve developers involved (I know almost nothing about it).
>> NTPD behaves more normally on uniprocessor VMs.
>>
>> A FreeBSD bhyve-guest running on a freebsd host will select a
>> different timecounter depending on whether it is a multiprocessor or a
>> uniprocessor.  My uniprocessor bhyve-vm selected TSC-low as the best
>> timecounter in a uniprocessor.  NTP functions there as expected.
>>
>> kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) 
>> dummy(-100)
>> kern.timecounter.hardware: TSC-low
>>
>> The very same VM, when given two total CPUs, selected HPET (if I
>> recall) and the timekeeping with NTPD was unreliable, with many
>> step-resets to the clock.
>>
> 
> Hmm, I just had glance at the code in sys/amd64/vmm/io/vhpet.c and it
> looks right.  I wonder if this is just a simple roundoff error in
> converting between 10.0MHz and SBT units?  If so, that could be wished
> away easily by using a power-of-2 frequency for the virtual HPET.  I
> wonder if the attached patch is all that's needed?
I've tried the patch (at bhyve guest) and nothing has changed. Should
the patched system be tested at bhyve guest or bhyve host?

-- 
WBR, bsam
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: host, bhyve vm and ntpd

2017-10-22 Thread Boris Samorodov
22.10.2017 00:07, Michael Voorhis пишет:
> Ian Lepore writes:
>> Beyond that, I'm not sure what else to try.  It might be necessary to
>> get some bhyve developers involved (I know almost nothing about it).
> 
> NTPD behaves more normally on uniprocessor VMs.
> 
> A FreeBSD bhyve-guest running on a freebsd host will select a
> different timecounter depending on whether it is a multiprocessor or a
> uniprocessor.  My uniprocessor bhyve-vm selected TSC-low as the best
> timecounter in a uniprocessor.  NTP functions there as expected.
> 
> kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) 
> dummy(-100)
> kern.timecounter.hardware: TSC-low
> 
> The very same VM, when given two total CPUs, selected HPET (if I
> recall) and the timekeeping with NTPD was unreliable, with many
> step-resets to the clock.

Yep, the same here. I've switched to TSC-low at Bhyve guest and there
is no stepping per 24 hours.

-- 
WBR, bsam
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: host, bhyve vm and ntpd

2017-10-21 Thread Ian Lepore
On Sat, 2017-10-21 at 17:07 -0400, Michael Voorhis wrote:
> Ian Lepore writes:
> > 
> > Beyond that, I'm not sure what else to try.  It might be necessary to
> > get some bhyve developers involved (I know almost nothing about it).
> NTPD behaves more normally on uniprocessor VMs.
> 
> A FreeBSD bhyve-guest running on a freebsd host will select a
> different timecounter depending on whether it is a multiprocessor or a
> uniprocessor.  My uniprocessor bhyve-vm selected TSC-low as the best
> timecounter in a uniprocessor.  NTP functions there as expected.
> 
> kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) 
> dummy(-100)
> kern.timecounter.hardware: TSC-low
> 
> The very same VM, when given two total CPUs, selected HPET (if I
> recall) and the timekeeping with NTPD was unreliable, with many
> step-resets to the clock.
> 

Hmm, I just had glance at the code in sys/amd64/vmm/io/vhpet.c and it
looks right.  I wonder if this is just a simple roundoff error in
converting between 10.0MHz and SBT units?  If so, that could be wished
away easily by using a power-of-2 frequency for the virtual HPET.  I
wonder if the attached patch is all that's needed?

-- Ian
Index: sys/amd64/vmm/io/vhpet.c
===
--- sys/amd64/vmm/io/vhpet.c	(revision 324176)
+++ sys/amd64/vmm/io/vhpet.c	(working copy)
@@ -51,7 +51,7 @@ __FBSDID("$FreeBSD$");
 
 static MALLOC_DEFINE(M_VHPET, "vhpet", "bhyve virtual hpet");
 
-#define	HPET_FREQ	1000		/* 10.0 Mhz */
+#define	HPET_FREQ	16777216		/* 16.7 (2^24) Mhz */
 #define	FS_PER_S	1000ul
 
 /* Timer N Configuration and Capabilities Register */
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: host, bhyve vm and ntpd

2017-10-21 Thread Michael Voorhis
Ian Lepore writes:
> Beyond that, I'm not sure what else to try.  It might be necessary to
> get some bhyve developers involved (I know almost nothing about it).

NTPD behaves more normally on uniprocessor VMs.

A FreeBSD bhyve-guest running on a freebsd host will select a
different timecounter depending on whether it is a multiprocessor or a
uniprocessor.  My uniprocessor bhyve-vm selected TSC-low as the best
timecounter in a uniprocessor.  NTP functions there as expected.

kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) 
dummy(-100)
kern.timecounter.hardware: TSC-low

The very same VM, when given two total CPUs, selected HPET (if I
recall) and the timekeeping with NTPD was unreliable, with many
step-resets to the clock.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"


Re: host, bhyve vm and ntpd

2017-10-20 Thread Ian Lepore
On Fri, 2017-10-20 at 21:15 +0300, Boris Samorodov wrote:
> 20.10.2017 21:04, Ian Lepore пишет:
> > 
> > On Fri, 2017-10-20 at 20:20 +0300, Boris Samorodov wrote:
> > > 
> > > (CC to freebsd-virtualization@)
> > > 
> > > 20.10.2017 19:32, Ian Lepore пишет:
> > > > 
> > > > 
> > > > On Fri, 2017-10-20 at 18:36 +0300, Boris Samorodov wrote:
> > > > > 
> > > > > 
> > > > > 20.10.2017 18:31, Boris Samorodov пишет:
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 20.10.2017 18:12, Ian Lepore пишет:
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > On Fri, 2017-10-20 at 14:46 +0300, Boris Samorodov wrote:
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > Hi All,
> > > > > > > > 
> > > > > > > > I have got a host:
> > > > > > > > ---
> > > > > > > > bhyve-host% uname -a
> > > > > > > > FreeBSD sm.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #3 r322868: 
> > > > > > > > Fri Aug
> > > > > > > > 25 05:25:26 MSK 2017
> > > > > > > > bsam@builder.bsnet:/usr/obj/usr/src/sys/GENERIC-FAST  amd64 
> > > > > > > > amd64
> > > > > > > > ---
> > > > > > > > 
> > > > > > > > And a bhyve vm:
> > > > > > > > ---
> > > > > > > > bhyve-vm: uname -a
> > > > > > > > FreeBSD builder.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #58 
> > > > > > > > r324782: Fri
> > > > > > > > Oct 20 05:12:17 MSK 2017
> > > > > > > > bsam@builder.bsnet:/usr/obj/usr/src/sys/PKG64X  amd64 amd64
> > > > > > > > ---
> > > > > > > > 
> > > > > > > > The only difference at kernel configs is a colored console. :-)
> > > > > > > > 
> > > > > > > > And here I get some weird (is it?) result at the VM (I expect 
> > > > > > > > ntpd to be
> > > > > > > > more stable):
> > > > > > > > ---
> > > > > > > > bhyve-vm% for t in `jot 10`; do ntpq -pn; sleep 64; done
> > > > > > > >  remote   refid  st t when poll reach   delay   
> > > > > > > > offset
> > > > > > > > jitter
> > > > > > > > ==
> > > > > > > >  XX.XX.XX.1  XX.XX.XX.245 4 u9   6430.605   
> > > > > > > > -1.202
> > > > > > > > 316.407
> > > > > > > >  XX.XX.XX.1  XX.XX.XX.245 4 u7   6470.605   
> > > > > > > > -1.202
> > > > > > > > 358.395
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u5   64   170.615  
> > > > > > > > -328.42
> > > > > > > > 181.405
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u3   64   370.615  
> > > > > > > > -328.42
> > > > > > > > 214.868
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   67   64   370.615  
> > > > > > > > -328.42
> > > > > > > > 214.868
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   63   64   770.615  
> > > > > > > > -328.42
> > > > > > > > 268.618
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   60   64  1770.615  
> > > > > > > > -328.42
> > > > > > > > 333.175
> > > > > > > >  XX.XX.XX.1  .STEP.  16 u 1910   6400.000   
> > > > > > > >  0.000
> > > > > > > > 0.000
> > > > > > > >  XX.XX.XX.1  XX.XX.XX.245 4 u   27   6410.703  
> > > > > > > > -262.63
> > > > > > > > 0.004
> > > > > > > >  XX.XX.XX.1  XX.XX.XX.245 4 u   31   6410.649  
> > > > > > > > -331.43
> > > > > > > > 68.800
> > > > > > > > ---
> > > > > > > > 
> > > > > > > > At the same time host's results are very stable:
> > > > > > > > ---
> > > > > > > > bhyve-host% for t in `jot 10`; do ntpq -pn; sleep 64; done
> > > > > > > >  remote   refid  st t when poll reach   delay   
> > > > > > > > offset
> > > > > > > > jitter
> > > > > > > > ==
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u1   6410.401   
> > > > > > > >  0.176
> > > > > > > > 0.106
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u6   6430.401   
> > > > > > > >  0.176
> > > > > > > > 0.459
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u3   6470.401   
> > > > > > > >  0.176
> > > > > > > > 0.940
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   67   6470.401   
> > > > > > > >  0.176
> > > > > > > > 0.940
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   64   64   170.401   
> > > > > > > >  0.176
> > > > > > > > 1.566
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   60   64   370.448   
> > > > > > > >  1.275
> > > > > > > > 1.739
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   55   64   770.448   
> > > > > > > >  1.275
> > > > > > > > 2.365
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   53   64  1770.448   
> > > > > > > >  1.275
> > > > > > > > 3.110
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   50   64  3770.448   
> > > > > > > >  1.275
> > > > > > > > 3.929
> > > > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   45   64  3770.443   
> > > > > > > >  8.750
> > > > > > > > 4.722
> > > > > > > > ---

Re: host, bhyve vm and ntpd

2017-10-20 Thread Boris Samorodov
20.10.2017 21:04, Ian Lepore пишет:
> On Fri, 2017-10-20 at 20:20 +0300, Boris Samorodov wrote:
>> (CC to freebsd-virtualization@)
>>
>> 20.10.2017 19:32, Ian Lepore пишет:
>>>
>>> On Fri, 2017-10-20 at 18:36 +0300, Boris Samorodov wrote:

 20.10.2017 18:31, Boris Samorodov пишет:
>
>
> 20.10.2017 18:12, Ian Lepore пишет:
>>
>>
>> On Fri, 2017-10-20 at 14:46 +0300, Boris Samorodov wrote:
>>>
>>>
>>> Hi All,
>>>
>>> I have got a host:
>>> ---
>>> bhyve-host% uname -a
>>> FreeBSD sm.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #3 r322868: Fri Aug
>>> 25 05:25:26 MSK 2017
>>> bsam@builder.bsnet:/usr/obj/usr/src/sys/GENERIC-FAST  amd64 amd64
>>> ---
>>>
>>> And a bhyve vm:
>>> ---
>>> bhyve-vm: uname -a
>>> FreeBSD builder.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #58 r324782: Fri
>>> Oct 20 05:12:17 MSK 2017
>>> bsam@builder.bsnet:/usr/obj/usr/src/sys/PKG64X  amd64 amd64
>>> ---
>>>
>>> The only difference at kernel configs is a colored console. :-)
>>>
>>> And here I get some weird (is it?) result at the VM (I expect ntpd to be
>>> more stable):
>>> ---
>>> bhyve-vm% for t in `jot 10`; do ntpq -pn; sleep 64; done
>>>  remote   refid  st t when poll reach   delay   offset
>>> jitter
>>> ==
>>>  XX.XX.XX.1  XX.XX.XX.245 4 u9   6430.605   -1.202
>>> 316.407
>>>  XX.XX.XX.1  XX.XX.XX.245 4 u7   6470.605   -1.202
>>> 358.395
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u5   64   170.615  -328.42
>>> 181.405
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u3   64   370.615  -328.42
>>> 214.868
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u   67   64   370.615  -328.42
>>> 214.868
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u   63   64   770.615  -328.42
>>> 268.618
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u   60   64  1770.615  -328.42
>>> 333.175
>>>  XX.XX.XX.1  .STEP.  16 u 1910   6400.0000.000
>>> 0.000
>>>  XX.XX.XX.1  XX.XX.XX.245 4 u   27   6410.703  -262.63
>>> 0.004
>>>  XX.XX.XX.1  XX.XX.XX.245 4 u   31   6410.649  -331.43
>>> 68.800
>>> ---
>>>
>>> At the same time host's results are very stable:
>>> ---
>>> bhyve-host% for t in `jot 10`; do ntpq -pn; sleep 64; done
>>>  remote   refid  st t when poll reach   delay   offset
>>> jitter
>>> ==
>>>
>>>
>>>
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u1   6410.4010.176
>>> 0.106
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u6   6430.4010.176
>>> 0.459
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u3   6470.4010.176
>>> 0.940
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u   67   6470.4010.176
>>> 0.940
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u   64   64   170.4010.176
>>> 1.566
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u   60   64   370.4481.275
>>> 1.739
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u   55   64   770.4481.275
>>> 2.365
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u   53   64  1770.4481.275
>>> 3.110
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u   50   64  3770.4481.275
>>> 3.929
>>> *XX.XX.XX.1  XX.XX.XX.245 4 u   45   64  3770.4438.750
>>> 4.722
>>> ---
>>>
>>> The network is organized via bridge -- host igb and vm tap interfaces
>>> are members of one bridge.
>>>
>>> Are those results expected? Does it smell like a bug? Should I dig
>>> furter?
>>>
>> So it is repeatedly stepping the clock in the VM? (Set
>> kern.timecounter.stepwarnings=1 to log steps).
> No kernel/ntpd messages for 20 minutes after setting this sysctl.
>
>>
>>
>>  That is usually a sign
>> that the chosen timecounter is running at a different frequency than it
>> claimed to be when it registered itself -- the host may not be
>> emulating the timer hardware properly in the guest.  What is the output
>> of sysctl kern.timecounter in the vm?
> ---
> bhyve-vm% sysctl kern.timecounter
>
> kern.timecounter.tsc_shift: 1
> kern.timecounter.smp_tsc_adjust: 0
> kern.timecounter.smp_tsc: 0
> kern.timecounter.invariant_tsc: 1
> kern.timecounter.fast_gettime: 1
> kern.timecounter.tick: 1
> kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(-100)
> dummy(-100)
> kern.timecounter.hardware: HPET
> kern.timecounter.alloweddeviation: 5
> kern.timecounter.stepwarnings: 1
> kern.timecounter.tc.ACPI-fast.quality: 900
> 

Re: host, bhyve vm and ntpd

2017-10-20 Thread Ian Lepore
On Fri, 2017-10-20 at 20:20 +0300, Boris Samorodov wrote:
> (CC to freebsd-virtualization@)
> 
> 20.10.2017 19:32, Ian Lepore пишет:
> > 
> > On Fri, 2017-10-20 at 18:36 +0300, Boris Samorodov wrote:
> > > 
> > > 20.10.2017 18:31, Boris Samorodov пишет:
> > > > 
> > > > 
> > > > 20.10.2017 18:12, Ian Lepore пишет:
> > > > > 
> > > > > 
> > > > > On Fri, 2017-10-20 at 14:46 +0300, Boris Samorodov wrote:
> > > > > > 
> > > > > > 
> > > > > > Hi All,
> > > > > > 
> > > > > > I have got a host:
> > > > > > ---
> > > > > > bhyve-host% uname -a
> > > > > > FreeBSD sm.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #3 r322868: Fri 
> > > > > > Aug
> > > > > > 25 05:25:26 MSK 2017
> > > > > > bsam@builder.bsnet:/usr/obj/usr/src/sys/GENERIC-FAST  amd64 amd64
> > > > > > ---
> > > > > > 
> > > > > > And a bhyve vm:
> > > > > > ---
> > > > > > bhyve-vm: uname -a
> > > > > > FreeBSD builder.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #58 
> > > > > > r324782: Fri
> > > > > > Oct 20 05:12:17 MSK 2017
> > > > > > bsam@builder.bsnet:/usr/obj/usr/src/sys/PKG64X  amd64 amd64
> > > > > > ---
> > > > > > 
> > > > > > The only difference at kernel configs is a colored console. :-)
> > > > > > 
> > > > > > And here I get some weird (is it?) result at the VM (I expect ntpd 
> > > > > > to be
> > > > > > more stable):
> > > > > > ---
> > > > > > bhyve-vm% for t in `jot 10`; do ntpq -pn; sleep 64; done
> > > > > >  remote   refid  st t when poll reach   delay   
> > > > > > offset
> > > > > > jitter
> > > > > > ==
> > > > > >  XX.XX.XX.1  XX.XX.XX.245 4 u9   6430.605   
> > > > > > -1.202
> > > > > > 316.407
> > > > > >  XX.XX.XX.1  XX.XX.XX.245 4 u7   6470.605   
> > > > > > -1.202
> > > > > > 358.395
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u5   64   170.615  
> > > > > > -328.42
> > > > > > 181.405
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u3   64   370.615  
> > > > > > -328.42
> > > > > > 214.868
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   67   64   370.615  
> > > > > > -328.42
> > > > > > 214.868
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   63   64   770.615  
> > > > > > -328.42
> > > > > > 268.618
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   60   64  1770.615  
> > > > > > -328.42
> > > > > > 333.175
> > > > > >  XX.XX.XX.1  .STEP.  16 u 1910   6400.000
> > > > > > 0.000
> > > > > > 0.000
> > > > > >  XX.XX.XX.1  XX.XX.XX.245 4 u   27   6410.703  
> > > > > > -262.63
> > > > > > 0.004
> > > > > >  XX.XX.XX.1  XX.XX.XX.245 4 u   31   6410.649  
> > > > > > -331.43
> > > > > > 68.800
> > > > > > ---
> > > > > > 
> > > > > > At the same time host's results are very stable:
> > > > > > ---
> > > > > > bhyve-host% for t in `jot 10`; do ntpq -pn; sleep 64; done
> > > > > >  remote   refid  st t when poll reach   delay   
> > > > > > offset
> > > > > > jitter
> > > > > > ==
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u1   6410.401
> > > > > > 0.176
> > > > > > 0.106
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u6   6430.401
> > > > > > 0.176
> > > > > > 0.459
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u3   6470.401
> > > > > > 0.176
> > > > > > 0.940
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   67   6470.401
> > > > > > 0.176
> > > > > > 0.940
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   64   64   170.401
> > > > > > 0.176
> > > > > > 1.566
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   60   64   370.448
> > > > > > 1.275
> > > > > > 1.739
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   55   64   770.448
> > > > > > 1.275
> > > > > > 2.365
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   53   64  1770.448
> > > > > > 1.275
> > > > > > 3.110
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   50   64  3770.448
> > > > > > 1.275
> > > > > > 3.929
> > > > > > *XX.XX.XX.1  XX.XX.XX.245 4 u   45   64  3770.443
> > > > > > 8.750
> > > > > > 4.722
> > > > > > ---
> > > > > > 
> > > > > > The network is organized via bridge -- host igb and vm tap 
> > > > > > interfaces
> > > > > > are members of one bridge.
> > > > > > 
> > > > > > Are those results expected? Does it smell like a bug? Should I dig
> > > > > > furter?
> > > > > > 
> > > > > So it is repeatedly stepping the clock in the VM? (Set
> > > > > kern.timecounter.stepwarnings=1 to log steps).
> > > > No kernel/ntpd messages for 20 minutes after setting this sysctl.
> > > > 
> > > > > 
> > > > > 
> > > > >  That is usually a sign
> > > > > that the chosen timecounter is running at a different frequency than 
> > > > > it
> > > > > claimed to be when it registered 

Re: host, bhyve vm and ntpd

2017-10-20 Thread Boris Samorodov
(CC to freebsd-virtualization@)

20.10.2017 19:32, Ian Lepore пишет:
> On Fri, 2017-10-20 at 18:36 +0300, Boris Samorodov wrote:
>> 20.10.2017 18:31, Boris Samorodov пишет:
>>>
>>> 20.10.2017 18:12, Ian Lepore пишет:

 On Fri, 2017-10-20 at 14:46 +0300, Boris Samorodov wrote:
>
> Hi All,
>
> I have got a host:
> ---
> bhyve-host% uname -a
> FreeBSD sm.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #3 r322868: Fri Aug
> 25 05:25:26 MSK 2017
> bsam@builder.bsnet:/usr/obj/usr/src/sys/GENERIC-FAST  amd64 amd64
> ---
>
> And a bhyve vm:
> ---
> bhyve-vm: uname -a
> FreeBSD builder.bsnet 12.0-CURRENT FreeBSD 12.0-CURRENT #58 r324782: Fri
> Oct 20 05:12:17 MSK 2017
> bsam@builder.bsnet:/usr/obj/usr/src/sys/PKG64X  amd64 amd64
> ---
>
> The only difference at kernel configs is a colored console. :-)
>
> And here I get some weird (is it?) result at the VM (I expect ntpd to be
> more stable):
> ---
> bhyve-vm% for t in `jot 10`; do ntpq -pn; sleep 64; done
>  remote   refid  st t when poll reach   delay   offset
> jitter
> ==
>  XX.XX.XX.1  XX.XX.XX.245 4 u9   6430.605   -1.202
> 316.407
>  XX.XX.XX.1  XX.XX.XX.245 4 u7   6470.605   -1.202
> 358.395
> *XX.XX.XX.1  XX.XX.XX.245 4 u5   64   170.615  -328.42
> 181.405
> *XX.XX.XX.1  XX.XX.XX.245 4 u3   64   370.615  -328.42
> 214.868
> *XX.XX.XX.1  XX.XX.XX.245 4 u   67   64   370.615  -328.42
> 214.868
> *XX.XX.XX.1  XX.XX.XX.245 4 u   63   64   770.615  -328.42
> 268.618
> *XX.XX.XX.1  XX.XX.XX.245 4 u   60   64  1770.615  -328.42
> 333.175
>  XX.XX.XX.1  .STEP.  16 u 1910   6400.0000.000
> 0.000
>  XX.XX.XX.1  XX.XX.XX.245 4 u   27   6410.703  -262.63
> 0.004
>  XX.XX.XX.1  XX.XX.XX.245 4 u   31   6410.649  -331.43
> 68.800
> ---
>
> At the same time host's results are very stable:
> ---
> bhyve-host% for t in `jot 10`; do ntpq -pn; sleep 64; done
>  remote   refid  st t when poll reach   delay   offset
> jitter
> ==
>
>
>
> *XX.XX.XX.1  XX.XX.XX.245 4 u1   6410.4010.176
> 0.106
> *XX.XX.XX.1  XX.XX.XX.245 4 u6   6430.4010.176
> 0.459
> *XX.XX.XX.1  XX.XX.XX.245 4 u3   6470.4010.176
> 0.940
> *XX.XX.XX.1  XX.XX.XX.245 4 u   67   6470.4010.176
> 0.940
> *XX.XX.XX.1  XX.XX.XX.245 4 u   64   64   170.4010.176
> 1.566
> *XX.XX.XX.1  XX.XX.XX.245 4 u   60   64   370.4481.275
> 1.739
> *XX.XX.XX.1  XX.XX.XX.245 4 u   55   64   770.4481.275
> 2.365
> *XX.XX.XX.1  XX.XX.XX.245 4 u   53   64  1770.4481.275
> 3.110
> *XX.XX.XX.1  XX.XX.XX.245 4 u   50   64  3770.4481.275
> 3.929
> *XX.XX.XX.1  XX.XX.XX.245 4 u   45   64  3770.4438.750
> 4.722
> ---
>
> The network is organized via bridge -- host igb and vm tap interfaces
> are members of one bridge.
>
> Are those results expected? Does it smell like a bug? Should I dig
> furter?
>
 So it is repeatedly stepping the clock in the VM? (Set
 kern.timecounter.stepwarnings=1 to log steps).
>>> No kernel/ntpd messages for 20 minutes after setting this sysctl.
>>>

  That is usually a sign
 that the chosen timecounter is running at a different frequency than it
 claimed to be when it registered itself -- the host may not be
 emulating the timer hardware properly in the guest.  What is the output
 of sysctl kern.timecounter in the vm?
>>> ---
>>> bhyve-vm% sysctl kern.timecounter
>>>
>>> kern.timecounter.tsc_shift: 1
>>> kern.timecounter.smp_tsc_adjust: 0
>>> kern.timecounter.smp_tsc: 0
>>> kern.timecounter.invariant_tsc: 1
>>> kern.timecounter.fast_gettime: 1
>>> kern.timecounter.tick: 1
>>> kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(-100)
>>> dummy(-100)
>>> kern.timecounter.hardware: HPET
>>> kern.timecounter.alloweddeviation: 5
>>> kern.timecounter.stepwarnings: 1
>>> kern.timecounter.tc.ACPI-fast.quality: 900
>>> kern.timecounter.tc.ACPI-fast.frequency: 3579545
>>> kern.timecounter.tc.ACPI-fast.counter: 4161213491
>>> kern.timecounter.tc.ACPI-fast.mask: 4294967295
>>> kern.timecounter.tc.HPET.quality: 950
>>> kern.timecounter.tc.HPET.frequency: 1000
>>> kern.timecounter.tc.HPET.counter: 3518036865
>>> kern.timecounter.tc.HPET.mask: 4294967295
>>> kern.timecounter.tc.i8254.quality: 0
>>>