Re: [PATCH kvm-unit-tests 0/6] Improve the output of test runners
On Tue, Dec 15, 2015 at 02:13:25PM +0100, Radim Krčmář wrote: > 2015-12-15 11:19+0100, Paolo Bonzini: > > On 14/12/2015 22:24, Radim Krčmář wrote: > >> This series is a mix of patches that change the output of run_tests.sh > >> and x86-run. The output of ./run_tests.sh now looks like this: > > > > I like the idea, thanks! I agree with Andrew about pretty much > > everything, except that I like having the summary close to PASS/FAIL. > > I'm planning a summary without useless informaton in v2, which might be > acceptable for everyone :) Yeah, I can live that :-) drew -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH kvm-unit-tests 0/6] Improve the output of test runners
2015-12-15 11:19+0100, Paolo Bonzini: > On 14/12/2015 22:24, Radim Krčmář wrote: >> This series is a mix of patches that change the output of run_tests.sh >> and x86-run. The output of ./run_tests.sh now looks like this: > > I like the idea, thanks! I agree with Andrew about pretty much > everything, except that I like having the summary close to PASS/FAIL. I'm planning a summary without useless informaton in v2, which might be acceptable for everyone :) >>> qemu-kvm: Property '.hv-synic' not found >>> skip hyperv_synic (failed $(echo quit | $qemu -enable-kvm -cpu >>> kvm64,hv_synic -device hyperv-testdev -monitor stdio > /dev/null)) > > Here I'd just print "failed check". Ok, thanks. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH kvm-unit-tests 0/6] Improve the output of test runners
2015-12-14 16:20-0600, Andrew Jones: > On Mon, Dec 14, 2015 at 10:24:15PM +0100, Radim Krčmář wrote: >> > skip vmx (0 tests, 0 unexpected failures) >> > PASS debug (7 tests, 0 unexpected failures) >> > qemu-kvm: Property '.hv-synic' not found >> > skip hyperv_synic (failed $(echo quit | $qemu -enable-kvm -cpu >> > kvm64,hv_synic -device hyperv-testdev -monitor stdio > /dev/null)) > > I'm not sure I like the (summary) addition. A summary of why we skip > would be useful, like the (i386 only) stuff, but otherwise it doesn't > seem necessary, and the "(failed $(echo quit | $qemu -enable-kvm -cpu > kvm64,hv_synic -device hyperv-testdev -monitor stdio > /dev/null))" > summary is a bit ugly, wrapping on many terminals. The summary should help to make callers more certain of what happened without spending time with tests.log. I'd like the summary more if we didn't print "0 unexpected failures" in passing/skipped cases, so maybe we can agree on that? I'll shorten the failed case. > Another comment that is series wide is that all these changes need to be > tested with 'make standalone' (and many of your patches will require > changes to scripts/mkstandalone.sh, for which you will never forgive > me for having written :-) First time I heard about it. :) -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH kvm-unit-tests 0/6] Improve the output of test runners
On 14/12/2015 22:24, Radim Krčmář wrote: > This series is a mix of patches that change the output of run_tests.sh > and x86-run. The output of ./run_tests.sh now looks like this: I like the idea, thanks! I agree with Andrew about pretty much everything, except that I like having the summary close to PASS/FAIL. >> qemu-kvm: Property '.hv-synic' not found >> skip hyperv_synic (failed $(echo quit | $qemu -enable-kvm -cpu >> kvm64,hv_synic -device hyperv-testdev -monitor stdio > /dev/null)) Here I'd just print "failed check". Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH kvm-unit-tests 0/6] Improve the output of test runners
On Mon, Dec 14, 2015 at 10:24:15PM +0100, Radim Krčmář wrote: > This series is a mix of patches that change the output of run_tests.sh > and x86-run. The output of ./run_tests.sh now looks like this: > > > PASS apic (14 tests, 0 unexpected failures) > > PASS ioapic (19 tests, 0 unexpected failures) > > PASS smptest (1 tests, 0 unexpected failures) > > PASS smptest3 (1 tests, 0 unexpected failures) > > PASS vmexit_cpuid > > PASS vmexit_vmcall > > PASS vmexit_mov_from_cr8 > > PASS vmexit_mov_to_cr8 > > PASS vmexit_inl_pmtimer > > PASS vmexit_ipi > > PASS vmexit_ipi_halt > > PASS vmexit_ple_round_robin > > PASS access > > skip smap (0 tests, 0 unexpected failures) > > skip pku (0 tests, 0 unexpected failures) > > PASS emulator (132 tests, 0 unexpected failures, 1 skipped) > > PASS eventinj (13 tests, 0 unexpected failures) > > PASS hypercall (2 tests, 0 unexpected failures) > > PASS idt_test (4 tests, 0 unexpected failures) > > PASS msr (13 tests, 0 unexpected failures) > > PASS pmu (67 tests, 0 unexpected failures, 1 expected failures) > > PASS port80 > > PASS realmode > > PASS s3 > > PASS sieve > > PASS tsc (3 tests, 0 unexpected failures) > > PASS tsc_adjust (5 tests, 0 unexpected failures) > > PASS xsave (17 tests, 0 unexpected failures) > > PASS rmap_chain > > skip svm (0 tests, 0 unexpected failures) > > skip svm-disabled (0 tests, 0 unexpected failures) > > skip taskswitch (i386 only) > > skip taskswitch2 (i386 only) > > PASS kvmclock_test > > PASS pcid (3 tests, 0 unexpected failures) > > skip vmx (0 tests, 0 unexpected failures) > > PASS debug (7 tests, 0 unexpected failures) > > qemu-kvm: Property '.hv-synic' not found > > skip hyperv_synic (failed $(echo quit | $qemu -enable-kvm -cpu > > kvm64,hv_synic -device hyperv-testdev -monitor stdio > /dev/null)) I'm not sure I like the (summary) addition. A summary of why we skip would be useful, like the (i386 only) stuff, but otherwise it doesn't seem necessary, and the "(failed $(echo quit | $qemu -enable-kvm -cpu kvm64,hv_synic -device hyperv-testdev -monitor stdio > /dev/null))" summary is a bit ugly, wrapping on many terminals. Another comment that is series wide is that all these changes need to be tested with 'make standalone' (and many of your patches will require changes to scripts/mkstandalone.sh, for which you will never forgive me for having written :-) Thanks, drew -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH kvm-unit-tests 0/6] Improve the output of test runners
This series is a mix of patches that change the output of run_tests.sh and x86-run. The output of ./run_tests.sh now looks like this: > PASS apic (14 tests, 0 unexpected failures) > PASS ioapic (19 tests, 0 unexpected failures) > PASS smptest (1 tests, 0 unexpected failures) > PASS smptest3 (1 tests, 0 unexpected failures) > PASS vmexit_cpuid > PASS vmexit_vmcall > PASS vmexit_mov_from_cr8 > PASS vmexit_mov_to_cr8 > PASS vmexit_inl_pmtimer > PASS vmexit_ipi > PASS vmexit_ipi_halt > PASS vmexit_ple_round_robin > PASS access > skip smap (0 tests, 0 unexpected failures) > skip pku (0 tests, 0 unexpected failures) > PASS emulator (132 tests, 0 unexpected failures, 1 skipped) > PASS eventinj (13 tests, 0 unexpected failures) > PASS hypercall (2 tests, 0 unexpected failures) > PASS idt_test (4 tests, 0 unexpected failures) > PASS msr (13 tests, 0 unexpected failures) > PASS pmu (67 tests, 0 unexpected failures, 1 expected failures) > PASS port80 > PASS realmode > PASS s3 > PASS sieve > PASS tsc (3 tests, 0 unexpected failures) > PASS tsc_adjust (5 tests, 0 unexpected failures) > PASS xsave (17 tests, 0 unexpected failures) > PASS rmap_chain > skip svm (0 tests, 0 unexpected failures) > skip svm-disabled (0 tests, 0 unexpected failures) > skip taskswitch (i386 only) > skip taskswitch2 (i386 only) > PASS kvmclock_test > PASS pcid (3 tests, 0 unexpected failures) > skip vmx (0 tests, 0 unexpected failures) > PASS debug (7 tests, 0 unexpected failures) > qemu-kvm: Property '.hv-synic' not found > skip hyperv_synic (failed $(echo quit | $qemu -enable-kvm -cpu kvm64,hv_synic > -device hyperv-testdev -monitor stdio > /dev/null)) I don't like it too much, but it still seems like an improvement. The main difference is in extra information for tests using report_summary(), that smap and pku tests don't fail, pmu test isn't completely skipped, and svm, vmx, and hyperv_synic are skipped. The old one looked like this: > PASS apic > PASS ioapic > PASS smptest > PASS smptest3 > PASS vmexit_cpuid > PASS vmexit_vmcall > PASS vmexit_mov_from_cr8 > PASS vmexit_mov_to_cr8 > PASS vmexit_inl_pmtimer > PASS vmexit_ipi > PASS vmexit_ipi_halt > PASS vmexit_ple_round_robin > PASS access > FAIL smap > FAIL pku > PASS emulator > PASS eventinj > PASS hypercall > PASS idt_test > PASS msr > skip pmu (/proc/sys/kernel/nmi_watchdog not equal to 0) > PASS port80 > PASS realmode > PASS s3 > PASS sieve > PASS tsc > PASS tsc_adjust > PASS xsave > PASS rmap_chain > PASS svm > PASS svm-disabled > skip taskswitch (i386 only) > skip taskswitch2 (i386 only) > PASS kvmclock_test > PASS pcid > PASS vmx > PASS debug > qemu-kvm: Property '.hv-synic' not found > PASS hyperv_synic Radim Krčmář (6): lib/report: allow test skipping x86/*: report that the test was skipped x86/pmu: expect failure with nmi_watchdog run_tests: generalize check x86/hyperv_synic: check for support before testing run_tests: print summary lib/libcflat.h | 1 + lib/report.c | 43 +-- run_tests.sh | 35 ++- x86/apic.c | 7 +++ x86/emulator.c | 2 +- x86/hyperv_synic.c | 2 +- x86/pku.c | 2 +- x86/pmu.c | 11 +-- x86/smap.c | 2 +- x86/svm.c | 2 +- x86/tsc.c | 2 +- x86/unittests.cfg | 4 ++-- 12 files changed, 68 insertions(+), 45 deletions(-) -- 2.6.4 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
