Re: [Qemu-devel] Improving QMP test coverage

2017-08-01 Thread Stefan Hajnoczi
On Thu, Jul 27, 2017 at 07:16:52PM +0800, Fam Zheng wrote: > On Thu, 07/27 11:09, Daniel P. Berrange wrote: > > On Thu, Jul 27, 2017 at 05:19:57PM +0800, Fam Zheng wrote: > > > On Thu, 07/27 10:14, Markus Armbruster wrote: > > > > This brings some advantages of "verify output with diff" to tests

Re: [Qemu-devel] Improving QMP test coverage

2017-07-27 Thread Fam Zheng
On Thu, 07/27 11:09, Daniel P. Berrange wrote: > On Thu, Jul 27, 2017 at 05:19:57PM +0800, Fam Zheng wrote: > > On Thu, 07/27 10:14, Markus Armbruster wrote: > > > This brings some advantages of "verify output with diff" to tests that > > > verify with code. Improvement if it simplifies the

Re: [Qemu-devel] Improving QMP test coverage

2017-07-27 Thread Daniel P. Berrange
On Thu, Jul 27, 2017 at 05:19:57PM +0800, Fam Zheng wrote: > On Thu, 07/27 10:14, Markus Armbruster wrote: > > This brings some advantages of "verify output with diff" to tests that > > verify with code. Improvement if it simplifies the verification code. > > > > I'd still prefer *no*

Re: [Qemu-devel] Improving QMP test coverage

2017-07-27 Thread Daniel P. Berrange
On Mon, Jul 24, 2017 at 08:56:50AM +0200, Markus Armbruster wrote: > Cleber Rosa writes: > > > On 07/21/2017 11:33 AM, Stefan Hajnoczi wrote: > >>> Output testing style delegates checking ouput to diff. I rather like it > >>> when text output is readily available. It is when

Re: [Qemu-devel] Improving QMP test coverage

2017-07-27 Thread Fam Zheng
On Thu, 07/27 17:19, Fam Zheng wrote: > On the other hand the iotests are more difficult to debug when things go wrong s/iotests/python tests/

Re: [Qemu-devel] Improving QMP test coverage

2017-07-27 Thread Fam Zheng
On Thu, 07/27 10:14, Markus Armbruster wrote: > This brings some advantages of "verify output with diff" to tests that > verify with code. Improvement if it simplifies the verification code. > > I'd still prefer *no* verification code (by delegating the job to diff) > for tests where I can get

Re: [Qemu-devel] Improving QMP test coverage

2017-07-27 Thread Markus Armbruster
Cleber Rosa writes: > On 07/24/2017 02:56 AM, Markus Armbruster wrote: >> Test code language is orthogonal to verification method (with code >> vs. with diff). Except verifying with shell code would be obviously >> nuts[*]. >> >> The existing iotests written in Python verify

Re: [Qemu-devel] Improving QMP test coverage

2017-07-25 Thread Cleber Rosa
On 07/24/2017 02:56 AM, Markus Armbruster wrote: > Test code language is orthogonal to verification method (with code > vs. with diff). Except verifying with shell code would be obviously > nuts[*]. > > The existing iotests written in Python verify with code, and the ones > written in shell

Re: [Qemu-devel] Improving QMP test coverage

2017-07-24 Thread Markus Armbruster
Cleber Rosa writes: > On 07/21/2017 11:33 AM, Stefan Hajnoczi wrote: >>> Output testing style delegates checking ouput to diff. I rather like it >>> when text output is readily available. It is when testing QMP. A >>> non-trivial example using this style could be useful, as

Re: [Qemu-devel] Improving QMP test coverage

2017-07-21 Thread Cleber Rosa
On 07/21/2017 11:33 AM, Stefan Hajnoczi wrote: >> Output testing style delegates checking ouput to diff. I rather like it >> when text output is readily available. It is when testing QMP. A >> non-trivial example using this style could be useful, as discussing >> ideas tends to be more

Re: [Qemu-devel] Improving QMP test coverage

2017-07-21 Thread Stefan Hajnoczi
On Tue, Jul 18, 2017 at 06:24:19PM +0200, Markus Armbruster wrote: > Stefan Hajnoczi writes: > > > On Thu, Jul 13, 2017 at 05:28:52PM +0200, Markus Armbruster wrote: > >> What can we do to improve QMP testing? Sadly, I don't have the master > >> plan ready. I can tell

Re: [Qemu-devel] Improving QMP test coverage

2017-07-18 Thread Markus Armbruster
Stefan Hajnoczi writes: > On Thu, Jul 13, 2017 at 05:28:52PM +0200, Markus Armbruster wrote: >> What can we do to improve QMP testing? Sadly, I don't have the master >> plan ready. I can tell people their new code needs to come with tests, >> but that won't help much unless

Re: [Qemu-devel] Improving QMP test coverage

2017-07-17 Thread Stefan Hajnoczi
On Thu, Jul 13, 2017 at 05:28:52PM +0200, Markus Armbruster wrote: > What can we do to improve QMP testing? Sadly, I don't have the master > plan ready. I can tell people their new code needs to come with tests, > but that won't help much unless subsystem maintainers pick up the habit, > too.

[Qemu-devel] Improving QMP test coverage

2017-07-13 Thread Markus Armbruster
I've seen enough QMP-related bugs that should've been caught in basic testing to know we need to become better at testing this stuff. That means investing into tests/. QMP is now almost eight years old, but its test coverage has always been spotty. We don't even really know where the holes are.