On Tue, Apr 17, 2018 at 02:01:53PM +0200, Markus Armbruster wrote: > Eduardo Habkost <ehabk...@redhat.com> writes: > > > See cover letter for a description of the new test system. > > > > TODO: code clean up > > TODO: write description. > > > > Signed-off-by: Eduardo Habkost <ehabk...@redhat.com> > > --- > > scripts/validator.py | 724 > > +++++++++++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 724 insertions(+) > > create mode 100755 scripts/validator.py > > > > diff --git a/scripts/validator.py b/scripts/validator.py > > new file mode 100755 > > index 0000000000..4312571feb > > --- /dev/null > > +++ b/scripts/validator.py > > @@ -0,0 +1,724 @@ > > +#!/usr/bin/env python > > +# > > +# Copyright (c) 2018 Red Hat Inc > > +# > > +# Author: > > +# Eduardo Habkost <ehabk...@redhat.com> > > +# > > +# This program is free software; you can redistribute it and/or modify > > +# it under the terms of the GNU General Public License as published by > > +# the Free Software Foundation; either version 2 of the License, or > > +# (at your option) any later version. > > +# > > +# This program is distributed in the hope that it will be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > +# GNU General Public License for more details. > > +# > > +# You should have received a copy of the GNU General Public License along > > +# with this program; if not, write to the Free Software Foundation, Inc., > > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > > + > > +""" > > +QEMU validator script > > +===================== > > + > > +This script will get test YAML test case specifications or Python > > +modules as input, and generate/run test cases based on them. > > + > > +USAGE > > +----- > > + > > +validator.py <specification-file>... -V VAR1=value1 VAR1=value2 VAR2=value3 > > + > > +specification-file is a YAML file containing the test specification. > > I can see the "test YAML test case specifications", but not the "Python > modules". > > Should we introduce yet another markup language into QEMU? (pardon the > pun).
Fair question. What are the existing markup languages in QEMU we could use? JSON is an option, but I believe YAML is more readable. > > > + > > +Example:: > > + > > + # this test specification is equivalent to the > > + # "device/introspect/list" test case in device-introspect-test.c > > If we merge this, not for long :) > > > + command-line: '$QEMU -nodefaults -machine none' > > + monitor-commands: > > + - qmp: > > + - execute: qom-list-types > > + arguments: > > + implements: 'device' > > + abstract: true > > + - hmp: 'device_add help' > > + > > + > > +VARIABLE EXPANSION > > +------------------ > > + > > +The test runner will try to run the test cases with all possible values > > +for variables appearing in the test specification. > > + > > +Some built-in variables are automatically expanded: > > + > > +* `$MACHINE` - Expands to a machine-type name supported by $QEMU > > +* `$ACCEL` - Expands to an accelerator name supported by $QEMU > > +* `$DEVICE` - Expands to a (user-creatable) device type name supported by > > $QEMU > > +* `$CPU` - Expands to a CPU model name supported by $QEMU > > + > > +Note that the $QEMU variable must be specified in th > > Yes? Heh. It must be specified in the command-line. I will fix that. > > > + > > +TEST SPECIFICATION FIELDS > > +------------------------- > > + > > +command-line > > +~~~~~~~~~~~~ > > + > > +List or string, containing the QEMU command-line to be run. > > + > > +Default: '$QEMU' > > + > > + > > +monitor-commands > > +~~~~~~~~~~~~~~~~ > > + > > +Mapping or list-of-mappings containing monitor commands to run. The key > > on each > > +item can be ``hmp`` or ``qmp``. The value on each entry can be a string, > > +mapping, or list. > > + > > +Default: None. > > + > > +TODO: not implemented yet. > > The whole monitor-commands feature? Outdated comment, sorry. > > Can I do > > monitor-commands: > - qmp: > - execute: eins > - hmp: zwei > - qmp: > - execute: drei > > to execute eins, zwei, drei in this order? Yes, it can. See the test specification examples. > > > + > > + > > +qmp > > +~~~ > > + > > +Boolean. If true (the default), a QMP monitor is configured on the > > command-line > > +automatically. > > + > > +If true, the test runner will issue a ``quit`` command automatically when > > +testing is finished. If false, the test runner will wait until QEMU exits > > by > > +itself. > > + > > +Example:: > > + > > + # just run $QEMU -help and ensure it won't crash > > + command-line: ['$QEMU', '-help'] > > + qmp: false > > + > > + > > +TODO: whitelist > > +TODO: validate output against reference output > > +TODO: configure defaults for variables > > +TODO: compatibility with Avocado multiplexer? > > +""" > > This is a DSL to write tests. I applaud the idea to write more tests at > a higher level than C. We just need to be careful not to create too > many different ways to write tests, or else readability will suffer. > Ideally, the way to use for a test should be fairly obvious. Agreed we need to keep this in mind. Replacing the DSL with very short and simple Python code is a possibility I was considering, but I'm not sure yet if this is the way to go. I don't want this to create a whole new test framework. > > Looking forward to your next iteration. Thanks! -- Eduardo
