Re: Several bhyve quirks

2015-03-27 Thread Neel Natu
Hi Julian,

On Wed, Mar 25, 2015 at 2:24 AM, Julian Hsiao wrote:

 I'm running bhyve on 10.1, mostly with OpenBSD (5.7) guests, and I ran into
 a few strange issues:

 1. The guest RTC is several hours off every time I start bhyve.  The host
 RTC is set to UTC, and /etc/localtime on both the host and guests are set to
 US/Pacific (currently PDT).  I thought maybe bhyve is setting the RTC to the
 local time, and indeed changing TZ environment variable affects the guest's
 RTC.  However, with TZ=UTC the guest is still off by an hour, and to get the
 correct offset I set TZ='UTC+1'; perhaps something's not handling DST

 Also, one time the offset was mysteriously tens of hours off (i.e. the guest
 RTS is a day or two ahead), and the condition persisted across multiple host
 and guest reboots.  Unfortunately, the problem went away a few hours later
 and I was unable to reproduce it since.

The problem is that in 10.1 (and earlier) bhyve defaulted to a 12-hour
RTC format but some guests like OpenBSD and Linux assume that it is
configured in the 24-hour format.

The 12-hour format indicates PM time by setting the most significant
bit in the 'hour' byte. Since the guest is not prepared to mask this
bit it thinks that the time is 68 hours ahead of the actual time (but
only for PM times - everything goes back to normal during AM times).

This is fixed in HEAD where the RTC device model defaults to 24-hour time.
 suggests that I'm on the right track, but it doesn't explain the off-by-one
 nor the (one time) multi-day offset.

The one-hour offset is a bug due to my interpretation of the 12-hour format.

I am going to fix this in HEAD shortly but here is a patch for 10.1 and earlier:

 As an aside, the commit message implies that this only affects OpenBSD
 guest, when in fact this probably affects all guests (at least also Linux).
 Perhaps he meant you cannot configure OpenBSD to assume that the RTC is set
 to local time instead of UTC.

 2. What's the preferred solution for minimizing guest clock drift in bhyve?
 Based on some Google searches, I run ntpd in the guests and set
 kern.timecounter.hardware=acpitimer0 instead of the default acpihpet0.
 acpitimer0 drifts by ~600 ppm while acpihpet0 drifts by ~1500 ppm; why?

I don't know but I am running experiments that I hope will provide some insight.


 3. Even moderate guest disk I/O completely kills guest network performance.
 For example, whenever security(8) (security(7) in FreeBSD) runs, guest
 network throughput drops from 150+ Mbps to ~20 Mbps, and jitter from ping
 jumps from 0.01 ms to 100+ ms.  If I try to build something in the guest,
 then network becomes almost unusable.

 The network performance degradation only affects the guest that's generating
 the I/O; high I/O on guest B doesn't affect guest A, nor would high I/O on
 the host.

 I'm using both virtio-blk and virio-net drivers, and the guests' disk images
 are backed by zvol+geli.  Removing geli has no effect.

 There are some commits in CURRENT that suggests improved virtio performance,
 but I'm not comfortable running CURRENT.  Is there a workaround I could use
 for 10.1?

 4. virtio-blk always reports the virtual disk as having 512-byte sectors,
 and so I get I/O errors on OpenBSD guests when the disk image is backed by
 zvol+geli with 4K sector size.  Curiously, this only seems to affect
 zvol+geli; with just zvol it seems to work.  Also, it works either way on
 Linux guests.

 ATM I changed the zvol / geli sector size to 512 bytes, which probably made
 #2 worse.  I think this bug / feature is addressed by:,
 but again is there a workaround to force a specific sector size for 10.1?

 5. This may be better directed at OpenBSD but I'll ask here anyway: if I
 enable virtio-rnd then OpenBSD would not boot with couldn't map interrupt
 error.  The kernel in bsd.rd will boot, but not the installed kernel (or the
 one built from STABLE; I forgot).  Again, Linux seems unaffected, but I
 couldn't tell if it's actually working.

 Julian Hsiao

 ___ mailing list
 To unsubscribe, send any mail to
___ mailing list
To unsubscribe, send any mail to

Re: Several bhyve quirks

2015-03-26 Thread Julian Hsiao

On 2015-03-25 15:44:35 +, Peter Grehan said:

In 10.1, virtio-blk i/o is done sychronously in the context of the
guest vCPU exit. If it's a single vCPU guest, or the virtio-net
interrupt happens to be delivered to that vCPU, performance will suffer.

A workaround is to use ahci-hd for the disk emulation and not
virtio-blk. The AHCI emulation does i/o in a dedicated thread and
doesn't block the vCPU thread.

Thank you for your explanation and tips, Peter.  I just tried changing 
virtio-blk - ahci-hd and preliminary results are good.  And now you've 
mentioned it, I do recall seeing slightly less performance degradation 
on guests with 2 vCPUs vs. ones with just one.

I've always assumed virtio driver  emulated driver so it didn't occur 
to me to try ahci-hd.

The only workaround for 10.1 would be to use ahci-hd instead of
virtio-blk. The correct sector size will be reported there.

I haven't had a chance to test this; next time I spin up a guest from 
scratch I'll try it out.

Try using the -W option to bhyve. This will force the bhyve virtio
code to advertize (non-standard) MSI interrupt capability which OpenBSD
will then use to allocate vectors.

Unfortunately -W didn't help.  This is not critical, however, and I'll 
ask around in the OpenBSD mailing list.

Thanks again for your help.

Julian Hsiao

___ mailing list
To unsubscribe, send any mail to