Re: vmm timer for linux guests

[Sivaram Gowkanapalli]

Hello Dave, Pratik,

> My hacky "vmm-clock" implementation is here if you'd like to compare
> notes: https://git.sr.ht/~voutilad/linux/log/linux-5.4-obsd

The clock is in sync with these patches. It is keeping the clock in sync with 
the host.

sudo dmesg | grep -i clock
[0.034850] clocksource: refined-jiffies: mask: 0x max_cycles: 
0x, max_idle_ns: 1911260446275 ns
[0.099077] [vmm_get_wallclock]
[0.099083] [vmm_get_wallclock] wallclock: 1590341994.140345631
[0.204807] clocksource: tsc-early: mask: 0x max_cycles: 
0x618c5af5a59, max_idle_ns: 440795234522 ns
[1.214829] clocksource: timekeeping watchdog on CPU0: Marking clocksource 
'tsc-early' as unstable because the skew is too large:
[1.214831] clocksource:   'vmm-clock' wd_now: 
1612079e9ed2e025 wd_last: 1612079e63384222 mask: 
[1.214832] clocksource:   'tsc-early' cs_now: 
824f14258d7a cs_last: 824e48e69b4f mask: 
[1.214833] tsc: Marking TSC unstable due to clocksource watchdog
[1.241836] clocksource: jiffies: mask: 0x max_cycles: 0x, 
max_idle_ns: 1911260446275 ns
[1.246960] PTP clock support registered
[1.254545] clocksource: Switched to clocksource vmm-clock
[3.412453] rtc_cmos rtc_cmos: setting system clock to 2020-05-24T17:39:57 
UTC (1590341997)
[3.412468] Unstable clock detected, switching default tracing clock to 
"global"
[3.412468] If you want to keep using the local clock, then add:
[3.412468]   "trace_clock=local"

For the next guy with this issue on alpine, http://ix.io/2nfZ is the patch I 
used against the alpine aports git repo to build the kernel.

Thanks

Re: vmm timer for linux guests

[Sivaram Gowkanapalli]

Hello Pratik,

I thought the serial console has been stable since this patch. I just noticed a 
hang. So, the issue seems to still persist.

Thanks

Re: vmm timer for linux guests

[Sivaram Gowkanapalli]

Hello Pratik,

Thanks for the patch. I am about to test it out too.

If you do not mind sharing, do you have your patches hosted somewhere?

Thanks


From: Pratik Vyas [m...@pd.io]
Sent: Friday, May 22, 2020 6:15 AM
To: Renato Aguiar
Cc: tech@openbsd.org; Sivaram Gowkanapalli; d...@sisu.io
Subject: Re: vmm timer for linux guests

* Renato Aguiar  [2020-05-21 12:55:45 -0700]:

>Hi Sivaram,
>
>I'm the author of the e-mail thread that you mentioned. After feedback
>I got from OpenBSD community, I created a patch for Linux to enable
>kvm-clock when booting on VMM. It managed to keep clock in sync, but I
>experienced random crashes in vmd after some time running the VM.
>
>Multiple fixes have been merged to vmm/vmd since then, so I'm planning
>to give it another try with OpenBSD 6.7.
>
>If you are interested, I can share the patch with you.
>
>Regards,
>
>--
>Renato Aguiar
>

Hi Sivaram and Renato,

I have been looking at this issue this week with Dave.  (funnily Dave
and I independently also decided to write a linux patch to attach
pvclock and debugging these crashes).

If you have your linux vm with kvm-clock around, can you try this rather
crude patch and see if it still crashes?  This might also fix the '100%
cpu when using alpine via console' problem.

For ref, this is my linux side patch to attach kvm-clock http://ix.io/2lzK

--
Pratik

diff --git a/usr.sbin/vmd/ns8250.c b/usr.sbin/vmd/ns8250.c
index 497e6fad550..33f1a371c16 100644
--- a/usr.sbin/vmd/ns8250.c
+++ b/usr.sbin/vmd/ns8250.c
@@ -36,6 +36,8 @@

  extern char *__progname;
  struct ns8250_dev com1_dev;
+struct event read_delay_ev;
+struct timeval read_delay_tv;

  static void com_rcv_event(int, short, void *);
  static void com_rcv(struct ns8250_dev *, uint32_t, uint32_t);
@@ -61,6 +63,11 @@ ratelimit(int fd, short type, void *arg)
vcpu_deassert_pic_irq(com1_dev.vmid, 0, com1_dev.irq);
  }

+static void
+arm_read(int fd, short type, void *arg) {
+   event_add(&com1_dev.event, NULL);
+}
+
  void
  ns8250_init(int fd, uint32_t vmid)
  {
@@ -96,7 +103,7 @@ ns8250_init(int fd, uint32_t vmid)
 */
com1_dev.pause_ct = (com1_dev.baudrate / 8) / 1000 * 10;

-   event_set(&com1_dev.event, com1_dev.fd, EV_READ | EV_PERSIST,
+   event_set(&com1_dev.event, com1_dev.fd, EV_READ,
com_rcv_event, (void *)(intptr_t)vmid);

/*
@@ -112,6 +119,9 @@ ns8250_init(int fd, uint32_t vmid)
timerclear(&com1_dev.rate_tv);
com1_dev.rate_tv.tv_usec = 1;
evtimer_set(&com1_dev.rate, ratelimit, NULL);
+   read_delay_tv.tv_usec = 1;
+   evtimer_set(&read_delay_ev, arm_read,
+   (void *)(intptr_t)vmid);
  }

  static void
@@ -131,6 +141,7 @@ com_rcv_event(int fd, short kind, void *arg)
 */
if (com1_dev.rcv_pending) {
mutex_unlock(&com1_dev.mutex);
+   evtimer_add(&read_delay_ev, &read_delay_tv);
return;
}

@@ -146,6 +157,7 @@ com_rcv_event(int fd, short kind, void *arg)
vcpu_deassert_pic_irq((uintptr_t)arg, 0, com1_dev.irq);
}
}
+   event_add(&com1_dev.event, NULL);

mutex_unlock(&com1_dev.mutex);
  }

Re: vmm timer for linux guests

[Renato Aguiar]

Hi Sivaram and David,

Here is my patch for Linux 4.19: 
https://gitlab.com/renatoaguiar/linux/-/commit/325afccf1e2d156be745f811411327e22bbd4c20


I also tried same patch with Linux 5.4 
, 
but for some reason 4.19 seemed more stable.


Regards,

--
Renato Aguiar

Re: vmm timer for linux guests

[Sivaram Gowkanapalli]

Hello Dave,

Thanks for sharing this. I have been using your hacked up version of 
virtio_vmmci to keep the timer in sync all this time.

btw, would you recommend a linux distro other than alpine for vmm? I need to 
run an Oracle Java app which needs glibc, hence, I cannot use alpine. I tried 
tiny core but had trouble with the install in my initial attempts. I have not 
dug into the issue yet. Just want to check if there is a better option instead.

I am using devuan and it has been working fine in the vm.

Thanks again

Re: vmm timer for linux guests

[Dave Voutila]

On Fri, May 22, 2020 at 6:15 AM Pratik Vyas  wrote:
> Hi Sivaram and Renato,
>
> I have been looking at this issue this week with Dave.  (funnily Dave
> and I independently also decided to write a linux patch to attach
> pvclock and debugging these crashes).

My hacky "vmm-clock" implementation is here if you'd like to compare
notes: https://git.sr.ht/~voutilad/linux/log/linux-5.4-obsd

My goal is to strip down the implementation to the bare minimum needed
for an OpenBSD guest similar to what I did with my vmmci(4)
implementation: https://github.com/voutilad/virtio_vmmci

One thing I've found is (at least with busybox's top(1)), the
accounting for cpu utilization is wrong if you just wire up the
kvm-clock clocksource.

>
> If you have your linux vm with kvm-clock around, can you try this rather
> crude patch and see if it still crashes?  This might also fix the '100%
> cpu when using alpine via console' problem.

>From my quick testing, with both the standard Linux 5.4 kernel with
Alpine 3.11 and my 5.4.40 fork, it resolves the CPU utilization issue
from the serial console, but doesn't resolve the serial console lockup
when spamming the return key. I haven't dug into why yet.

> For ref, this is my linux side patch to attach kvm-clock http://ix.io/2lzK
>
> @@ -96,7 +103,7 @@ ns8250_init(int fd, uint32_t vmid)
>  */
> com1_dev.pause_ct = (com1_dev.baudrate / 8) / 1000 * 10;
>
> -   event_set(&com1_dev.event, com1_dev.fd, EV_READ | EV_PERSIST,
> +   event_set(&com1_dev.event, com1_dev.fd, EV_READ,
> com_rcv_event, (void *)(intptr_t)vmid);

pd@ I think you're missing the analogous code in ns8250_restore().

-- 
Dave Voutila

Re: vmm timer for linux guests

[Sivaram Gowkanapalli]

Hello Renato,

Could you please share the linux patches? We can try with them too.

Thanks


From: Renato Aguiar [ren...@renatoaguiar.net]
Sent: Thursday, May 21, 2020 3:55 PM
To: tech@openbsd.org
Cc: Sivaram Gowkanapalli
Subject: Re: vmm timer for linux guests

Hi Sivaram,

I'm the author of the e-mail thread that you mentioned. After
feedback I got from OpenBSD community, I created a patch for Linux
to enable kvm-clock when booting on VMM. It managed to keep clock
in sync, but I experienced random crashes in vmd after some time
running the VM.

Multiple fixes have been merged to vmm/vmd since then, so I'm
planning to give it another try with OpenBSD 6.7.

If you are interested, I can share the patch with you.

Regards,

--
Renato Aguiar

Re: vmm timer for linux guests

[Sivaram Gowkanapalli]

Hello Pratik,

> For ref, this is my linux side patch to attach kvm-clock http://ix.io/2lzK

Thanks, will try the below patches.

Re: vmm timer for linux guests

[Pratik Vyas]

* Renato Aguiar  [2020-05-21 12:55:45 -0700]:


Hi Sivaram,

I'm the author of the e-mail thread that you mentioned. After feedback 
I got from OpenBSD community, I created a patch for Linux to enable 
kvm-clock when booting on VMM. It managed to keep clock in sync, but I 
experienced random crashes in vmd after some time running the VM.


Multiple fixes have been merged to vmm/vmd since then, so I'm planning 
to give it another try with OpenBSD 6.7.


If you are interested, I can share the patch with you.

Regards,

--
Renato Aguiar



Hi Sivaram and Renato,

I have been looking at this issue this week with Dave.  (funnily Dave
and I independently also decided to write a linux patch to attach
pvclock and debugging these crashes).

If you have your linux vm with kvm-clock around, can you try this rather
crude patch and see if it still crashes?  This might also fix the '100%
cpu when using alpine via console' problem.

For ref, this is my linux side patch to attach kvm-clock http://ix.io/2lzK

--
Pratik

diff --git a/usr.sbin/vmd/ns8250.c b/usr.sbin/vmd/ns8250.c
index 497e6fad550..33f1a371c16 100644
--- a/usr.sbin/vmd/ns8250.c
+++ b/usr.sbin/vmd/ns8250.c
@@ -36,6 +36,8 @@
 
 extern char *__progname;

 struct ns8250_dev com1_dev;
+struct event read_delay_ev;
+struct timeval read_delay_tv;
 
 static void com_rcv_event(int, short, void *);

 static void com_rcv(struct ns8250_dev *, uint32_t, uint32_t);
@@ -61,6 +63,11 @@ ratelimit(int fd, short type, void *arg)
vcpu_deassert_pic_irq(com1_dev.vmid, 0, com1_dev.irq);
 }
 
+static void

+arm_read(int fd, short type, void *arg) {
+   event_add(&com1_dev.event, NULL);
+}
+
 void
 ns8250_init(int fd, uint32_t vmid)
 {
@@ -96,7 +103,7 @@ ns8250_init(int fd, uint32_t vmid)
 */
com1_dev.pause_ct = (com1_dev.baudrate / 8) / 1000 * 10;
 
-	event_set(&com1_dev.event, com1_dev.fd, EV_READ | EV_PERSIST,

+   event_set(&com1_dev.event, com1_dev.fd, EV_READ,
com_rcv_event, (void *)(intptr_t)vmid);
 
 	/*

@@ -112,6 +119,9 @@ ns8250_init(int fd, uint32_t vmid)
timerclear(&com1_dev.rate_tv);
com1_dev.rate_tv.tv_usec = 1;
evtimer_set(&com1_dev.rate, ratelimit, NULL);
+   read_delay_tv.tv_usec = 1;
+   evtimer_set(&read_delay_ev, arm_read,
+   (void *)(intptr_t)vmid);
 }
 
 static void

@@ -131,6 +141,7 @@ com_rcv_event(int fd, short kind, void *arg)
 */
if (com1_dev.rcv_pending) {
mutex_unlock(&com1_dev.mutex);
+   evtimer_add(&read_delay_ev, &read_delay_tv);
return;
}
 
@@ -146,6 +157,7 @@ com_rcv_event(int fd, short kind, void *arg)

vcpu_deassert_pic_irq((uintptr_t)arg, 0, com1_dev.irq);
}
}
+   event_add(&com1_dev.event, NULL);
 
 	mutex_unlock(&com1_dev.mutex);

 }

Re: vmm timer for linux guests

[Renato Aguiar]

Hi Sivaram,

I'm the author of the e-mail thread that you mentioned. After 
feedback I got from OpenBSD community, I created a patch for Linux 
to enable kvm-clock when booting on VMM. It managed to keep clock 
in sync, but I experienced random crashes in vmd after some time 
running the VM.


Multiple fixes have been merged to vmm/vmd since then, so I'm 
planning to give it another try with OpenBSD 6.7.


If you are interested, I can share the patch with you.

Regards,

--
Renato Aguiar

vmm timer for linux guests

[Sivaram Gowkanapalli]

Hello Mike,

>From #openbsd, I gather that you are working on the timer issue (time falling 
>behind) that linux guests have in vmm.

I run linux guests in vmm and I am using a linux module just to sync the timer 
every 5 seconds from the host. I presume that there might be a better solution 
than that. Just want to check if you might have any test code or material that 
can help me work around the issue.

I tried this and the timer in the linux guest is kvm-clock and it avoids the 
timer issue. But, the linux guests hang once in a while. I presume that it 
might be due to some kvm instruction unsupported by vmm. I got this idea from 
https://marc.info/?l=openbsd-tech&m=155903945227814&w=2

Index: sys/arch/amd64/amd64/vmm.c
===
RCS file: /cvs/src/sys/arch/amd64/amd64/vmm.c,v
retrieving revision 1.273
diff -u -p -u -r1.273 vmm.c
--- sys/arch/amd64/amd64/vmm.c  19 Apr 2020 19:29:52 -  1.273
+++ sys/arch/amd64/amd64/vmm.c  20 May 2020 17:28:12 -
@@ -242,6 +242,7 @@ extern uint64_t tsc_frequency;
 extern int tsc_is_invariant;
 
 const char *vmm_hv_signature = VMM_HV_SIGNATURE;
+const char *kvm_hv_signature = KVM_HV_SIGNATURE;
 
 const struct kmem_pa_mode vmm_kp_contig = {
.kp_constraint = &no_constraint,
@@ -6830,7 +6831,14 @@ vmm_handle_cpuid(struct vcpu *vcpu)
*rcx = *((uint32_t *)&vmm_hv_signature[4]);
*rdx = *((uint32_t *)&vmm_hv_signature[8]);
break;
+   case 0x4100: /* KVM CPUID signature */
+   *rax = 0;
+   *rbx = *((uint32_t *)&kvm_hv_signature[0]);
+   *rcx = *((uint32_t *)&kvm_hv_signature[4]);
+   *rdx = *((uint32_t *)&kvm_hv_signature[8]);
+   break;
case 0x4001:/* KVM hypervisor features */
+   case 0x4101:
*rax = (1 << KVM_FEATURE_CLOCKSOURCE2) |
(1 << KVM_FEATURE_CLOCKSOURCE_STABLE_BIT);
*rbx = 0;
Index: sys/arch/amd64/include/vmmvar.h
===
RCS file: /cvs/src/sys/arch/amd64/include/vmmvar.h,v
retrieving revision 1.70
diff -u -p -u -r1.70 vmmvar.h
--- sys/arch/amd64/include/vmmvar.h 8 Apr 2020 07:39:48 -   1.70
+++ sys/arch/amd64/include/vmmvar.h 20 May 2020 17:28:12 -
@@ -22,6 +22,7 @@
 #define _MACHINE_VMMVAR_H_
 
 #define VMM_HV_SIGNATURE   "OpenBSDVMM58"
+#define KVM_HV_SIGNATURE   "KVMKVMKVM\0\0\0"
 
 #define VMM_MAX_MEM_RANGES 16
 #define VMM_MAX_DISKS_PER_VM   4

Thanks