Daniel P. Berrangé <berra...@redhat.com> writes: > When reviewing tracetool patches it is often very unclear what the > expected output will be for the generated backends. Compounding > this is that a default build will only enable the 'log' trace > backend, so developers won't see generated code for other backends > without making a special effort. Some backends are also platform > specific, so can't be enabled in QEMU builds, even though tracetool > could generate the code. > > To address this, introduce a test suite for tracetool which is > conceptually similar to the qapi-schema test. It is a simple > python program that runs tracetool and compares the actual output > to historical reference output kept in git. The test directly > emits TAP format logs for ease of integration with meson. > > This can be run with > > make check-tracetool > > to make it easier for developers changing generated output, the > sample expected content can be auto-recreated > > QEMU_TEST_REGENERATE=1 make check-tracetool
make check-qapi-schema uses QAPI_TEST_UPDATE for this. Should we use a single environment variable for this purpose? I'd be fine with changing QAPI_TEST_UPDATE. > and the changes reviewed and added to the commit. This will also > assist reviewers interpreting the change. > > Developers are reminded of this in the test output on failure: > > $ make check-tracetool > 1/6 qemu:tracetool / dtrace OK 0.14s > 2/6 qemu:tracetool / ftrace FAIL 0.06s exit status 1 > ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――― > 1..2 > ok 1 - ftrace.c > # > not ok 1 - ftrace.h (set QEMU_TEST_REGENERATE=1 to recreate reference > output if tracetool generator was intentionally changed) Neat! Nice to have in tests/qapi-schema/test-qapi.py, too. Observation, not a demand. > ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――― > > 3/6 qemu:tracetool / log OK 0.06s > 4/6 qemu:tracetool / simple OK 0.06s > 5/6 qemu:tracetool / syslog OK 0.06s > 6/6 qemu:tracetool / ust OK 0.11s > > Summary of Failures: > > 2/6 qemu:tracetool / ftrace FAIL 0.06s exit status 1 > > Signed-off-by: Daniel P. Berrangé <berra...@redhat.com> [...] > diff --git a/MAINTAINERS b/MAINTAINERS > index 28cea34271..c08c51f4c6 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -3556,6 +3556,7 @@ F: scripts/tracetool/ > F: scripts/qemu-trace-stap* > F: docs/tools/qemu-trace-stap.rst > F: docs/devel/tracing.rst > +F: tests/tracetool/ > T: git https://github.com/stefanha/qemu.git tracing > > Simpletrace > diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst > index 2b5cb0c148..11f05c0006 100644 > --- a/docs/devel/testing/main.rst > +++ b/docs/devel/testing/main.rst > @@ -178,6 +178,34 @@ parser (either fixing a bug or extending/modifying the > syntax). To do this: > > ``qapi-schema += foo.json`` > > +.. _tracetool-tests: > + > +Tracetool tests > +~~~~~~~~~~~~~~~ > + > +The tracetool tests validate the generated source files used for defining > +probes for various tracing backends and source formats. The test operates > +by running the tracetool program against a sample trace-events file, and > +comparing the generated output against known good reference output. The > +tests can be run with: > + > +.. code:: > + > + make check-tracetool > + > +The reference output is stored in files under tests/tracetool, and when > +the tracetool backend/format output is intentionally changed, the reference > +files need to be updated. This can be automated by setting the > +QEMU_TEST_REGENERATE=1 environment variable: > + > +.. code:: > + > + QEMU_TEST_REGENERATE=1 make check-tracetool > + > +The resulting changes must be reviewed by the author to ensure they match > +the intended results, before adding the updated reference output to the > +same commit that alters the generator code. Section "QAPI schema tests" could use similar advice on how to update reference output. Observation, not a demand. > + > check-block > ~~~~~~~~~~~ > [...]
