Re: [Qemu-devel] [RFC 17/18] validator.py script

[Eduardo Habkost]

On Wed, Apr 18, 2018 at 08:58:00AM +0200, Markus Armbruster wrote:
> Eduardo Habkost  writes:
> 
> > On Tue, Apr 17, 2018 at 02:01:53PM +0200, Markus Armbruster wrote:
> >> Eduardo Habkost  writes:
> [...]
> >> > +command-line: '$QEMU -nodefaults -machine none'
> >> > +monitor-commands:
> >> > +- qmp:
> >> > +  - execute: qom-list-types
> >> > +arguments:
> >> > +  implements: 'device'
> >> > +  abstract: true
> >> > +- hmp: 'device_add help'
> [...]
> >> > +monitor-commands
> >> > +
> >> > +
> >> > +Mapping or list-of-mappings containing monitor commands to run.
> 
> http://yaml.org/spec/1.2/spec.html talks about mappings and sequences.
> Recommend to say "sequence" rather than "list".

Noted.

> 
> I can't see how sequences come into play.  Your example's
> monitor-command is a mapping, not a sequence of mappings.  Am I
> confused?

The example is supposed to be a sequence of mappings.

> 
> >> >   The 
> >> > key on each
> >> > +item can be ``hmp`` or ``qmp``.  The value on each entry can be a 
> >> > string,
> >> > +mapping, or list.
> >> > +
> >> > +Default: None.
> [...]
> >> Can I do
> >> 
> >> monitor-commands:
> >> - qmp:
> >>   - execute: eins
> >> - hmp: zwei
> >> - qmp:
> >>   - execute: drei
> >> 
> >> to execute eins, zwei, drei in this order?
> >
> > Yes, it can.  See the test specification examples.
> 
> I asked because the YAML spec I quoted above says mappings are unordered
> and must be unique.  My example violates "unique", and your "yes, it
> can" violates "unordered".
> 
> [...]

The above is a sequence of one-key mappings.


This is a (not very useful) mapping:

>>> yaml.load("""
... qmp:
... - execute: eins
... hmp: zwei
... """)
{'qmp': [{'execute': 'eins'}], 'hmp': 'zwei'}


This is a sequence of one-key mappings:

>>> yaml.load("""
... - qmp:
...   - execute: eins
... - hmp: zwei
... - qmp:
...   - execute: drei
... """)
[{'qmp': [{'execute': 'eins'}]}, {'hmp': 'zwei'}, {'qmp': [{'execute': 
'drei'}]}]
>>>

-- 
Eduardo

Re: [Qemu-devel] [RFC 17/18] validator.py script

[Markus Armbruster]

Eduardo Habkost  writes:

> On Tue, Apr 17, 2018 at 02:01:53PM +0200, Markus Armbruster wrote:
>> Eduardo Habkost  writes:
[...]
>> > +command-line: '$QEMU -nodefaults -machine none'
>> > +monitor-commands:
>> > +- qmp:
>> > +  - execute: qom-list-types
>> > +arguments:
>> > +  implements: 'device'
>> > +  abstract: true
>> > +- hmp: 'device_add help'
[...]
>> > +monitor-commands
>> > +
>> > +
>> > +Mapping or list-of-mappings containing monitor commands to run.

http://yaml.org/spec/1.2/spec.html talks about mappings and sequences.
Recommend to say "sequence" rather than "list".

I can't see how sequences come into play.  Your example's
monitor-command is a mapping, not a sequence of mappings.  Am I
confused?

>> >   The key 
>> > on each
>> > +item can be ``hmp`` or ``qmp``.  The value on each entry can be a string,
>> > +mapping, or list.
>> > +
>> > +Default: None.
[...]
>> Can I do
>> 
>> monitor-commands:
>> - qmp:
>>   - execute: eins
>> - hmp: zwei
>> - qmp:
>>   - execute: drei
>> 
>> to execute eins, zwei, drei in this order?
>
> Yes, it can.  See the test specification examples.

I asked because the YAML spec I quoted above says mappings are unordered
and must be unique.  My example violates "unique", and your "yes, it
can" violates "unordered".

[...]

Re: [Qemu-devel] [RFC 17/18] validator.py script

[Eduardo Habkost]

On Tue, Apr 17, 2018 at 05:17:13PM +0200, Paolo Bonzini wrote:
> On 17/04/2018 16:42, Eduardo Habkost wrote:
> >> Should we introduce yet another markup language into QEMU?  (pardon the
> >> pun).
> > Fair question.  What are the existing markup languages in QEMU we
> > could use?
> > 
> > JSON is an option, but I believe YAML is more readable.
> 
> Isn't YAML actually a superset of JSON that adds "magic indent", bullet
> lists and a bunch of other stuff that you're not using?  You can still
> use JSON to express a complicated dictionary for example.

Things I like about YAML in comparison to JSON:
- Comments
- No need to count number of closing brackets or worry about
  trailing commas
- Convenient ways to write long strings without
  "\\"another layer of escaping syntax\\""

JSON is not a language for humans, YAML is.

-- 
Eduardo

Re: [Qemu-devel] [RFC 17/18] validator.py script

[Paolo Bonzini]

On 17/04/2018 16:42, Eduardo Habkost wrote:
>> Should we introduce yet another markup language into QEMU?  (pardon the
>> pun).
> Fair question.  What are the existing markup languages in QEMU we
> could use?
> 
> JSON is an option, but I believe YAML is more readable.

Isn't YAML actually a superset of JSON that adds "magic indent", bullet
lists and a bunch of other stuff that you're not using?  You can still
use JSON to express a complicated dictionary for example.

Paolo

Re: [Qemu-devel] [RFC 17/18] validator.py script

[Eduardo Habkost]

On Tue, Apr 17, 2018 at 02:01:53PM +0200, Markus Armbruster wrote:
> Eduardo Habkost  writes:
> 
> > See cover letter for a description of the new test system.
> >
> > TODO: code clean up
> > TODO: write description.
> >
> > Signed-off-by: Eduardo Habkost 
> > ---
> >  scripts/validator.py | 724 
> > +++
> >  1 file changed, 724 insertions(+)
> >  create mode 100755 scripts/validator.py
> >
> > diff --git a/scripts/validator.py b/scripts/validator.py
> > new file mode 100755
> > index 00..4312571feb
> > --- /dev/null
> > +++ b/scripts/validator.py
> > @@ -0,0 +1,724 @@
> > +#!/usr/bin/env python
> > +#
> > +#  Copyright (c) 2018 Red Hat Inc
> > +#
> > +# Author:
> > +#  Eduardo Habkost 
> > +#
> > +# This program is free software; you can redistribute it and/or modify
> > +# it under the terms of the GNU General Public License as published by
> > +# the Free Software Foundation; either version 2 of the License, or
> > +# (at your option) any later version.
> > +#
> > +# This program is distributed in the hope that it will be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License along
> > +# with this program; if not, write to the Free Software Foundation, Inc.,
> > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> > +
> > +"""
> > +QEMU validator script
> > +=
> > +
> > +This script will get test YAML test case specifications or Python
> > +modules as input, and generate/run test cases based on them.
> > +
> > +USAGE
> > +-
> > +
> > +validator.py ... -V VAR1=value1 VAR1=value2 VAR2=value3
> > +
> > +specification-file is a YAML file containing the test specification.
> 
> I can see the "test YAML test case specifications", but not the "Python
> modules".
> 
> Should we introduce yet another markup language into QEMU?  (pardon the
> pun).

Fair question.  What are the existing markup languages in QEMU we
could use?

JSON is an option, but I believe YAML is more readable.

> 
> > +
> > +Example::
> > +
> > +# this test specification is equivalent to the
> > +# "device/introspect/list" test case in device-introspect-test.c
> 
> If we merge this, not for long :)
> 
> > +command-line: '$QEMU -nodefaults -machine none'
> > +monitor-commands:
> > +- qmp:
> > +  - execute: qom-list-types
> > +arguments:
> > +  implements: 'device'
> > +  abstract: true
> > +- hmp: 'device_add help'
> > +
> > +
> > +VARIABLE EXPANSION
> > +--
> > +
> > +The test runner will try to run the test cases with all possible values
> > +for variables appearing in the test specification.
> > +
> > +Some built-in variables are automatically expanded:
> > +
> > +* `$MACHINE` - Expands to a machine-type name supported by $QEMU
> > +* `$ACCEL` - Expands to an accelerator name supported by $QEMU
> > +* `$DEVICE` - Expands to a (user-creatable) device type name supported by 
> > $QEMU
> > +* `$CPU` - Expands to a CPU model name supported by $QEMU
> > +
> > +Note that the $QEMU variable must be specified in th
> 
> Yes?

Heh.  It must be specified in the command-line.  I will fix that.

> 
> > +
> > +TEST SPECIFICATION FIELDS
> > +-
> > +
> > +command-line
> > +
> > +
> > +List or string, containing the QEMU command-line to be run.
> > +
> > +Default: '$QEMU'
> > +
> > +
> > +monitor-commands
> > +
> > +
> > +Mapping or list-of-mappings containing monitor commands to run.  The key 
> > on each
> > +item can be ``hmp`` or ``qmp``.  The value on each entry can be a string,
> > +mapping, or list.
> > +
> > +Default: None.
> > +
> > +TODO: not implemented yet.
> 
> The whole monitor-commands feature?

Outdated comment, sorry.

> 
> Can I do
> 
> monitor-commands:
> - qmp:
>   - execute: eins
> - hmp: zwei
> - qmp:
>   - execute: drei
> 
> to execute eins, zwei, drei in this order?

Yes, it can.  See the test specification examples.

> 
> > +
> > +
> > +qmp
> > +~~~
> > +
> > +Boolean.  If true (the default), a QMP monitor is configured on the 
> > command-line
> > +automatically.
> > +
> > +If true, the test runner will issue a ``quit`` command automatically when
> > +testing is finished.  If false, the test runner will wait until QEMU exits 
> > by
> > +itself.
> > +
> > +Example::
> > +
> > +# just run $QEMU -help and ensure it won't crash
> > +command-line: ['$QEMU', '-help']
> > +qmp: false
> > +
> > +
> > +TODO: whitelist
> > +TODO: validate output against reference output
> > +TODO: configure defaults for variables
> > +TODO: compatibility with Avocado multiplexer?
> > +"""
> 
> This is a DSL to write tests.  I 

Re: [Qemu-devel] [RFC 17/18] validator.py script

[Markus Armbruster]

Eduardo Habkost  writes:

> See cover letter for a description of the new test system.
>
> TODO: code clean up
> TODO: write description.
>
> Signed-off-by: Eduardo Habkost 
> ---
>  scripts/validator.py | 724 
> +++
>  1 file changed, 724 insertions(+)
>  create mode 100755 scripts/validator.py
>
> diff --git a/scripts/validator.py b/scripts/validator.py
> new file mode 100755
> index 00..4312571feb
> --- /dev/null
> +++ b/scripts/validator.py
> @@ -0,0 +1,724 @@
> +#!/usr/bin/env python
> +#
> +#  Copyright (c) 2018 Red Hat Inc
> +#
> +# Author:
> +#  Eduardo Habkost 
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +
> +"""
> +QEMU validator script
> +=
> +
> +This script will get test YAML test case specifications or Python
> +modules as input, and generate/run test cases based on them.
> +
> +USAGE
> +-
> +
> +validator.py ... -V VAR1=value1 VAR1=value2 VAR2=value3
> +
> +specification-file is a YAML file containing the test specification.

I can see the "test YAML test case specifications", but not the "Python
modules".

Should we introduce yet another markup language into QEMU?  (pardon the
pun).

> +
> +Example::
> +
> +# this test specification is equivalent to the
> +# "device/introspect/list" test case in device-introspect-test.c

If we merge this, not for long :)

> +command-line: '$QEMU -nodefaults -machine none'
> +monitor-commands:
> +- qmp:
> +  - execute: qom-list-types
> +arguments:
> +  implements: 'device'
> +  abstract: true
> +- hmp: 'device_add help'
> +
> +
> +VARIABLE EXPANSION
> +--
> +
> +The test runner will try to run the test cases with all possible values
> +for variables appearing in the test specification.
> +
> +Some built-in variables are automatically expanded:
> +
> +* `$MACHINE` - Expands to a machine-type name supported by $QEMU
> +* `$ACCEL` - Expands to an accelerator name supported by $QEMU
> +* `$DEVICE` - Expands to a (user-creatable) device type name supported by 
> $QEMU
> +* `$CPU` - Expands to a CPU model name supported by $QEMU
> +
> +Note that the $QEMU variable must be specified in th

Yes?

> +
> +TEST SPECIFICATION FIELDS
> +-
> +
> +command-line
> +
> +
> +List or string, containing the QEMU command-line to be run.
> +
> +Default: '$QEMU'
> +
> +
> +monitor-commands
> +
> +
> +Mapping or list-of-mappings containing monitor commands to run.  The key on 
> each
> +item can be ``hmp`` or ``qmp``.  The value on each entry can be a string,
> +mapping, or list.
> +
> +Default: None.
> +
> +TODO: not implemented yet.

The whole monitor-commands feature?

Can I do

monitor-commands:
- qmp:
  - execute: eins
- hmp: zwei
- qmp:
  - execute: drei

to execute eins, zwei, drei in this order?

> +
> +
> +qmp
> +~~~
> +
> +Boolean.  If true (the default), a QMP monitor is configured on the 
> command-line
> +automatically.
> +
> +If true, the test runner will issue a ``quit`` command automatically when
> +testing is finished.  If false, the test runner will wait until QEMU exits by
> +itself.
> +
> +Example::
> +
> +# just run $QEMU -help and ensure it won't crash
> +command-line: ['$QEMU', '-help']
> +qmp: false
> +
> +
> +TODO: whitelist
> +TODO: validate output against reference output
> +TODO: configure defaults for variables
> +TODO: compatibility with Avocado multiplexer?
> +"""

This is a DSL to write tests.  I applaud the idea to write more tests at
a higher level than C.  We just need to be careful not to create too
many different ways to write tests, or else readability will suffer.
Ideally, the way to use for a test should be fairly obvious.

Looking forward to your next iteration.

[...]