Hi Greg, On Sun, Nov 25, 2018 at 10:13:52AM -0800, Greg Steuck wrote: > Hi Anton, > > I tried to boot a kernel with kcov based on GENERIC.MP and the machine > reboots without a peep immediately after > > vmm0 at mainbus0: VMX (using slow L1TF mitigation) > > Switching off either of kcov or MP results in normally working kernels. I'm > attaching two concatenated dmesgs. The effect is reproducible on real HW > and on GCE VM. Broken config is just: > $ cat /sys/arch/amd64/conf/SYZKALLER > include "arch/amd64/conf/GENERIC.MP" > pseudo-device kcov 1 > > Disabling either vmm or kcov in broken kernel UKC doesn't prevent crashes.
Known limitation, I haven't spent much time on making kcov MP-safe. Especially since it's primarily used inside a VM through vmm which currently is limited to a single CPU. However, I did some investigation before and concluded that the problem resides in the trace routine which is called from cpu_boot_secondary_processors() before the secondary CPU is accessible through curcpu(). I came up with a hackish solution to this problem (see diff below) that got rejected; kettenis@ mentioned that we instead should set MSR_GSBASE earlier in cpu_hatch() but I never managed to get the right people involved with knowledge in this area. I might take a look myself. In the meantime, you could give the diff a try. It might be the case that more functions are not eligible for tracing. OpenBSD as no method of turning of tracing for a given source file like Linux does. This might become necessary since I fear many more functions will not cope with tracing. Index: dev/kcov.c =================================================================== RCS file: /cvs/src/sys/dev/kcov.c,v retrieving revision 1.4 diff -u -p -r1.4 kcov.c --- dev/kcov.c 27 Aug 2018 15:57:39 -0000 1.4 +++ dev/kcov.c 8 Sep 2018 21:51:20 -0000 @@ -49,6 +49,7 @@ struct kcov_dev { }; void kcovattach(int); +void kcov_attachhook(struct device *); int kd_alloc(struct kcov_dev *, unsigned long); void kd_free(struct kcov_dev *); @@ -57,6 +58,7 @@ struct kcov_dev *kd_lookup(int); static inline int inintr(void); TAILQ_HEAD(, kcov_dev) kd_list = TAILQ_HEAD_INITIALIZER(kd_list); +int kcov_attached = 0; #ifdef KCOV_DEBUG int kcov_debug = 1; @@ -76,12 +78,11 @@ int kcov_debug = 1; void __sanitizer_cov_trace_pc(void) { - extern int cold; struct kcov_dev *kd; uint64_t idx; - /* Do not trace during boot. */ - if (cold) + /* Do not trace before the root file system is mounted. */ + if (!kcov_attached) return; /* Do not trace in interrupts to prevent noisy coverage. */ @@ -102,6 +103,13 @@ __sanitizer_cov_trace_pc(void) void kcovattach(int count) { + config_mountroot(NULL, kcov_attachhook); +} + +void +kcov_attachhook(struct device *dev) +{ + kcov_attached = 1; } int