[PATCH v3 7/7] python, iotests: replace qmp with aqmp

2021-10-12 Thread John Snow
work concurrently. Signed-off-by: John Snow Reviewed-by: Hanna Reitz Tested-by: Hanna Reitz --- python/qemu/machine/machine.py | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/python/qemu/machine/machine.py b/python/qemu/machine/machine.py index a0cf69786b4..a487c397459

[PATCH v3 6/7] python/aqmp: Create sync QMP wrapper for iotests

2021-10-12 Thread John Snow
This is a wrapper around the async QMPClient that mimics the old, synchronous QEMUMonitorProtocol class. It is designed to be interchangeable with the old implementation. It does not, however, attempt to mimic Exception compatibility. Signed-off-by: John Snow --- python/qemu/aqmp/legacy.py

[PATCH v3 5/7] iotests: Conditionally silence certain AQMP errors

2021-10-12 Thread John Snow
can't amend the iotest output to just always expect the error message, either. Just suppress it for now. More rigorous log filtering can be investigated later if/when it is deemed safe to permanently replace the legacy QMP library.) Signed-off-by: John Snow --- tests/qemu-iotests/iotests.py

[PATCH v3 0/7] Switch iotests to using Async QMP

2021-10-12 Thread John Snow
ntroduce a helper in patch 14 to silence log output when it's unwanted. - Small addition of _get_greeting() helper in patch 15, coinciding with the new patch 05 here. - Contextual changes in 16. John Snow (7): python/machine: remove has_quit argument python/machine: Handle QMP errors on clo

[PATCH v3 3/7] python/aqmp: Remove scary message

2021-10-12 Thread John Snow
The scary message interferes with the iotests output. Coincidentally, if iotests works by removing this, then it's good evidence that we don't really need to scare people away from using it. Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 12 1 file changed, 12

[PATCH v3 4/7] iotests: Accommodate async QMP Exception classes

2021-10-12 Thread John Snow
. That'd be these: - QMPError - QMPConnectError - QMPCapabilitiesError - QMPTimeoutError - QMPProtocolError - QMPResponseError - QMPBadPortError Signed-off-by: John Snow --- scripts/simplebench/bench_block_job.py| 3 ++- tests/qemu-iotests/tests/mirror-top-perms | 5 +++-- 2 files changed, 5

[PATCH v3 1/7] python/machine: remove has_quit argument

2021-10-12 Thread John Snow
ut down. Signed-off-by: John Snow Reviewed-by: Hanna Reitz --- python/qemu/machine/machine.py | 34 +++--- tests/qemu-iotests/040 | 7 +-- tests/qemu-iotests/218 | 2 +- tests/qemu-iotests/255 | 2 +- 4 files changed, 22 insertions(+), 2

[PULL 10/10] python, iotests: remove socket_scm_helper

2021-10-12 Thread John Snow
It's not used anymore, now. Signed-off-by: John Snow Reviewed-by: Hanna Reitz Reviewed-by: Paolo Bonzini Message-id: 20210923004938.363-11-js...@redhat.com Signed-off-by: John Snow --- tests/qemu-iotests/socket_scm_helper.c | 136 - python/qemu/machine/machine.py

[PULL 07/10] python/aqmp: Disable logging messages by default

2021-10-12 Thread John Snow
that will suppress these messages until a client application opts into logging via logging.basicConfig or similar. Note that upon calling basicConfig(), this handler will *not* suppress these messages from being displayed by the client's configuration. Signed-off-by: John Snow Reviewed-by: Paolo Bonzini

[PULL 09/10] python/qmp: add send_fd_scm directly to QEMUMonitorProtocol

2021-10-12 Thread John Snow
implementation details. /that/ is helpful in turn because it allows me to write a compatible, alternative implementation. Signed-off-by: John Snow Reviewed-by: Hanna Reitz Reviewed-by: Paolo Bonzini Message-id: 20210923004938.363-10-js...@redhat.com Signed-off-by: John Snow --- python/qemu/machine

[PULL 08/10] python/qmp: clear events on get_events() call

2021-10-12 Thread John Snow
the retrieval also clear the queue is vastly simpler. Signed-off-by: John Snow Reviewed-by: Hanna Reitz Reviewed-by: Paolo Bonzini Message-id: 20210923004938.363-9-js...@redhat.com Signed-off-by: John Snow --- python/qemu/machine/machine.py | 1 - python/qemu/qmp/__init__.py| 6 -- python

[PULL 06/10] python/aqmp: Reduce severity of EOFError-caused loop terminations

2021-10-12 Thread John Snow
the normal operation of the library. Signed-off-by: John Snow Reviewed-by: Paolo Bonzini Reviewed-by: Eric Blake Message-id: 20210923004938.363-7-js...@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/protocol.py | 7 +-- 1 file changed, 5 insertions(+), 2 deletions(-) diff

[PULL 02/10] python/aqmp: add .empty() method to EventListener

2021-10-12 Thread John Snow
Synchronous clients may want to know if they're about to block waiting for an event or not. A method such as this is necessary to implement a compatible interface for the old QEMUMonitorProtocol using the new async internals. Signed-off-by: John Snow Reviewed-by: Hanna Reitz Reviewed-by: Paolo

[PULL 05/10] python/aqmp: Add dict conversion method to Greeting object

2021-10-12 Thread John Snow
The iotests interface expects to return the greeting as a dict; AQMP offers it as a rich object. Signed-off-by: John Snow Reviewed-by: Paolo Bonzini Reviewed-by: Eric Blake Message-id: 20210923004938.363-6-js...@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/models.py | 13

[PULL 04/10] python/aqmp: add send_fd_scm

2021-10-12 Thread John Snow
ps://bugs.python.org/issue43232 Signed-off-by: John Snow Reviewed-by: Paolo Bonzini Reviewed-by: Eric Blake Message-id: 20210923004938.363-5-js...@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/qmp_client.py | 22 ++ 1 file changed, 22 insertions(+) diff --gi

[PULL 03/10] python/aqmp: Return cleared events from EventListener.clear()

2021-10-12 Thread John Snow
This serves two purposes: (1) It is now possible to discern whether or not clear() removed any event(s) from the queue with absolute certainty, and (2) It is now very easy to get a List of all pending events in one chunk, which is useful for the sync bridge. Signed-off-by: John Snow Reviewed

[PULL 01/10] python/aqmp: add greeting property to QMPClient

2021-10-12 Thread John Snow
Expose the greeting as a read-only property of QMPClient so it can be retrieved at-will. Signed-off-by: John Snow Reviewed-by: Hanna Reitz Reviewed-by: Paolo Bonzini Message-id: 20210923004938.363-2-js...@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/qmp_client.py | 5 + 1

[PULL 00/10] Python patches

2021-10-12 Thread John Snow
for you to fetch changes up to c163c723ef92d0f629d015902396f2c67328b2e5: python, iotests: remove socket_scm_helper (2021-10-12 12:22:11 -0400) Pull request John

Re: [PATCH v2 13/17] iotests: Accommodate async QMP Exception classes

2021-10-12 Thread John Snow
On Tue, Oct 12, 2021 at 12:06 PM Hanna Reitz wrote: > On 23.09.21 02:49, John Snow wrote: > > (But continue to support the old ones for now, too.) > > > > There are very few cases of any user of QEMUMachine or a subclass > > thereof relying on a QMP Exception t

Re: [PATCH v2 12/17] python/machine: Handle QMP errors on close more meticulously

2021-10-07 Thread John Snow
On Wed, Sep 22, 2021 at 8:50 PM John Snow wrote: > To use the AQMP backend, Machine just needs to be a little more diligent > about what happens when closing a QMP connection. The operation is no > longer a freebie in the async world; it may return errors encountered in > the async

Re: [PATCH v2 12/17] python/machine: Handle QMP errors on close more meticulously

2021-10-07 Thread John Snow
On Thu, Oct 7, 2021 at 11:08 AM Eric Blake wrote: > On Wed, Sep 22, 2021 at 08:49:33PM -0400, John Snow wrote: > > To use the AQMP backend, Machine just needs to be a little more diligent > > about what happens when closing a QMP connection. The operation is no > > longer a

Re: [PATCH v2 04/17] python/aqmp: add send_fd_scm

2021-10-07 Thread John Snow
On Thu, Oct 7, 2021 at 10:52 AM Eric Blake wrote: > On Wed, Sep 22, 2021 at 08:49:25PM -0400, John Snow wrote: > > The single space is indeed required to successfully transmit the file > > descriptor to QEMU. > > Sending fds requires a payload of at least one byte, but I do

Re: [PATCH v2 15/17] python/aqmp: Create sync QMP wrapper for iotests

2021-10-06 Thread John Snow
On Wed, Oct 6, 2021 at 10:32 AM Paolo Bonzini wrote: > On 06/10/21 16:24, John Snow wrote: > > > > I had plans at one point to make a sync.py, but with an interface that > > matched async QMP itself more closely. I spent some time trying to > > research how to make a

Re: [PATCH v2 00/17] Switch iotests to using Async QMP

2021-10-06 Thread John Snow
On Wed, Oct 6, 2021 at 6:14 AM Paolo Bonzini wrote: > On 23/09/21 02:49, John Snow wrote: > > Based-on: <20210915162955.333025-1-js...@redhat.com> > >[PATCH v4 00/27] python: introduce Asynchronous QMP package > > GitLab: > https://gitlab.com/jsnow/qem

Re: [PATCH v2 15/17] python/aqmp: Create sync QMP wrapper for iotests

2021-10-06 Thread John Snow
On Wed, Oct 6, 2021 at 6:13 AM Paolo Bonzini wrote: > On 23/09/21 02:49, John Snow wrote: > > This is a wrapper around the async QMPClient that mimics the old, > > synchronous QEMUMonitorProtocol class. It is designed to be > > interchangeable with the old implementa

Re: [PATCH 12/15] iotests: Disable AQMP logging under non-debug modes

2021-10-04 Thread John Snow
On Mon, Oct 4, 2021 at 2:32 PM John Snow wrote: > > > On Mon, Oct 4, 2021 at 6:12 AM Hanna Reitz wrote: > >> On 18.09.21 04:14, John Snow wrote: >> > >> > >> > On Fri, Sep 17, 2021 at 8:58 PM John Snow > > <mailto:js...@redhat.com>>

[PATCH 12/13] python: Add iotest linters to test suite

2021-10-04 Thread John Snow
against a wide variety of python versions, which helps to find accidental cross-version python compatibility issues. Signed-off-by: John Snow --- python/tests/iotests-mypy.sh | 4 python/tests/iotests-pylint.sh | 4 2 files changed, 8 insertions(+) create mode 100755 python/tests/iotests

[PATCH 08/13] iotests/297: refactor run_[mypy|pylint] as generic execution shim

2021-10-04 Thread John Snow
There's virtually nothing special here anymore; we can combine these into a single, rather generic function. Signed-off-by: John Snow --- tests/qemu-iotests/297 | 46 +++--- 1 file changed, 25 insertions(+), 21 deletions(-) diff --git a/tests/qemu-iotests

[PATCH 06/13] iotests/297: Separate environment setup from test execution

2021-10-04 Thread John Snow
Move environment setup into main(), leaving pure test execution behind in run_linters(). Signed-off-by: John Snow --- tests/qemu-iotests/297 | 23 ++- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297 index

[PATCH 10/13] iotests/linters: Add entry point for linting via Python CI

2021-10-04 Thread John Snow
shebang.) Signed-off-by: John Snow --- (1) I think that the test file discovery logic and skip list belong together, and that those items belong in iotests/. I think they also belong in whichever directory pylintrc and mypy.ini are in, also in iotests/. (2) Moving this logic

[PATCH 13/13] iotests: [RFC] drop iotest 297

2021-10-04 Thread John Snow
running logic anymore is the skip list and the python-test-detection code. Without those, we could easily codify the tests as simply: [pylint|mypy] *.py tests/*.py ... and drop this entire file. We're not quite there yet, though. Signed-off-by: John Snow --- tests/qemu-iotests/297| 52

[PATCH 11/13] iotests/linters: Add workaround for mypy bug #9852

2021-10-04 Thread John Snow
es that 0.920 may bring with it.) See also: https://github.com/python/mypy/issues/11010 https://github.com/python/mypy/issues/9852 Signed-off-by: John Snow --- tests/qemu-iotests/linters.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/linters.py b/tests/qemu-i

[PATCH 05/13] iotests/297: Create main() function

2021-10-04 Thread John Snow
Instead of running "run_linters" directly, create a main() function that will be responsible for environment setup, leaving run_linters() responsible only for execution of the linters. (That environment setup will be moved over in forthcoming commits.) Signed-off-by: John Snow

[PATCH 07/13] iotests/297: Split run_linters apart into run_pylint and run_mypy

2021-10-04 Thread John Snow
Signed-off-by: John Snow --- Note, this patch really ought to be squashed with the next one, but I am performing a move known as "Hedging my bets." It's easier to squash than de-squash :) Signed-off-by: John Snow --- tests/qemu-iotests/297 | 19 --- 1 file c

[PATCH 09/13] iotests: split linters.py out from 297

2021-10-04 Thread John Snow
configurations both would be ... a hassle. Signed-off-by: John Snow --- tests/qemu-iotests/297| 72 +++--- tests/qemu-iotests/linters.py | 83 +++ 2 files changed, 88 insertions(+), 67 deletions(-) create mode 100644 tests/qemu-iotests

[PATCH 01/13] iotests/297: Move pylint config into pylintrc

2021-10-04 Thread John Snow
Move --score=n and --notes=XXX,FIXME into pylintrc. This pulls configuration out of code, which I think is probably a good thing in general. Signed-off-by: John Snow --- tests/qemu-iotests/297 | 4 +--- tests/qemu-iotests/pylintrc | 16 2 files changed, 17 insertions

[PATCH 00/13] python/iotests: Run iotest linters during Python CI

2021-10-04 Thread John Snow
inting issues. - Added a workaround for a rather nasty mypy bug ... >:( V2: - Added patches 1-5 which do some more delinting. - Added patch 8, which scans subdirs for tests to lint. - Added patch 17, which improves the speed of mypy analysis. - Patch 14 is different because of the new patch

[PATCH 04/13] iotests/297: Don't rely on distro-specific linter binaries

2021-10-04 Thread John Snow
test begins -- this check is now "overly strict", but shouldn't cause anything that was already running correctly to start failing. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Hanna Reitz --- tests/qemu-iotest

[PATCH 03/13] iotests/297: Add get_files() function

2021-10-04 Thread John Snow
Split out file discovery into its own method to begin separating out configuration/setup and test execution. Signed-off-by: John Snow --- tests/qemu-iotests/297 | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297 index

[PATCH 02/13] iotests/297: Split mypy configuration out into mypy.ini

2021-10-04 Thread John Snow
More separation of code and configuration. Signed-off-by: John Snow --- tests/qemu-iotests/297 | 14 +- tests/qemu-iotests/mypy.ini | 12 2 files changed, 13 insertions(+), 13 deletions(-) create mode 100644 tests/qemu-iotests/mypy.ini diff --git a/tests/qemu

Re: [PATCH v3 11/16] iotests/297: return error code from run_linters()

2021-10-04 Thread John Snow
On Mon, Oct 4, 2021 at 3:45 AM Hanna Reitz wrote: > On 22.09.21 22:18, John Snow wrote: > > > > > > On Fri, Sep 17, 2021 at 7:00 AM Hanna Reitz > <mailto:hre...@redhat.com>> wrote: > > [...] > > > > > As you say, run_linters() to

Re: [PATCH v3 07/16] iotests/297: Don't rely on distro-specific linter binaries

2021-10-04 Thread John Snow
On Mon, Oct 4, 2021 at 4:17 AM Hanna Reitz wrote: > On 22.09.21 21:53, John Snow wrote: > > (This email just explains python packaging stuff. No action items in > > here. Skim away.) > > > > On Fri, Sep 17, 2021 at 5:43 AM Hanna Reitz > <mailto:hre...@redhat.com

Re: [PATCH 12/15] iotests: Disable AQMP logging under non-debug modes

2021-10-04 Thread John Snow
On Mon, Oct 4, 2021 at 6:12 AM Hanna Reitz wrote: > On 18.09.21 04:14, John Snow wrote: > > > > > > On Fri, Sep 17, 2021 at 8:58 PM John Snow > <mailto:js...@redhat.com>> wrote: > > > > > > > > On Fri, Sep 17, 2021 at 10:3

Re: Running 297 from GitLab CI

2021-10-01 Thread John Snow
On Fri, Oct 1, 2021 at 4:21 AM Kevin Wolf wrote: > Am 30.09.2021 um 23:28 hat John Snow geschrieben: > > Hiya, I was talking this over with Hanna in review to '[PATCH v3 00/16] > > python/iotests: Run iotest linters during Python CI' [1] and I have some > > doubt about what

Running 297 from GitLab CI

2021-09-30 Thread John Snow
Hiya, I was talking this over with Hanna in review to '[PATCH v3 00/16] python/iotests: Run iotest linters during Python CI' [1] and I have some doubt about what you'd personally like to see happen, here. In a nutshell, I split out 'linters.py' from 297 and keep all of the iotest-bits in 297 and

Re: [PATCH 0/4] qemu-img compare --stat

2021-09-29 Thread John Snow
On Wed, Sep 29, 2021 at 9:34 AM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > Hi all! > > Recently we faced the following task: > > Customer comes and say: incremental backup images are too fat. Does you > incremental backup works correct? > > What to answer? We should check

Re: [PATCH v2 0/6] iotests: update environment and linting configuration

2021-09-24 Thread John Snow
On Thu, Sep 23, 2021 at 2:07 PM John Snow wrote: > GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-package-iotest-pt1 > CI: https://gitlab.com/jsnow/qemu/-/pipelines/376236687 > > This series partially supersedes: > [PATCH v3 00/16] python/iotests: Run iotest linters d

Re: [PATCH v2 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-24 Thread John Snow
On Thu, Sep 23, 2021 at 4:27 PM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > 23.09.2021 21:44, John Snow wrote: > > > > > > On Thu, Sep 23, 2021 at 2:32 PM Vladimir Sementsov-Ogievskiy < > vsement...@virtuozzo.com <mai

Re: [PATCH v2 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-23 Thread John Snow
On Thu, Sep 23, 2021 at 2:32 PM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > 23.09.2021 21:07, John Snow wrote: > > Add a warning for when 'iotests' runs against a qemu namespace that > > isn't the one in the source tree. This might occur if you h

[PATCH v2 5/6] iotests/migrate-bitmaps-test: delint

2021-09-23 Thread John Snow
Mostly uninteresting stuff. Move the test injections under a function named main() so that the variables used during that process aren't in the global scope. Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Hanna Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy

[PATCH v2 6/6] iotests: Update for pylint 2.11.1

2021-09-23 Thread John Snow
-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Kevin Wolf --- tests/qemu-iotests/pylintrc | 6 +- tests/qemu-iotests/testrunner.py | 7 --- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/tests/qemu-iotests/pylintrc b/tests/qemu-iotests

[PATCH v2 4/6] iotests/mirror-top-perms: Adjust imports

2021-09-23 Thread John Snow
We need to import subpackages from the qemu namespace package; importing the namespace package alone doesn't bring the subpackages with it -- unless someone else (like iotests.py) imports them too. Adjust the imports. Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé Reviewed

[PATCH v2 1/6] iotests: add 'qemu' package location to PYTHONPATH in testenv

2021-09-23 Thread John Snow
;help me find my python packages" goop is all in one place, right in one function. Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Kevin Wolf --- tests/qemu-iotests/235| 2 -- tests/qemu-i

[PATCH v2 3/6] iotests/linters: check mypy files all at once

2021-09-23 Thread John Snow
We can circumvent the '__main__' redefinition problem by passing --scripts-are-modules. Take mypy out of the loop per-filename and check everything in one go: it's quite a bit faster. Signed-off-by: John Snow Reviewed-by: Hanna Reitz Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Vladimir

[PATCH v2 0/6] iotests: update environment and linting configuration

2021-09-23 Thread John Snow
tion logic to not panic if it can't find 'qemu' at all (kwolf) - Updated commit messages for the first two patches. --js John Snow (6): iotests: add 'qemu' package location to PYTHONPATH in testenv iotests: add warning for rogue 'qemu' packages iotests/linters: check mypy files all at once iote

[PATCH v2 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-23 Thread John Snow
and executed `pip install .`, you will see output like this when running `./check`: WARNING: 'qemu' python packages will be imported from outside the source tree ('/home/jsnow/src/qemu/python') Importing instead from '/home/jsnow/.local/lib/python3.9/site-packages/qemu' Signed-off-by: John Snow

Re: [PATCH 1/6] iotests: add 'qemu' package location to PYTHONPATH in testenv

2021-09-23 Thread John Snow
On Thu, Sep 23, 2021 at 11:20 AM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > 23.09.2021 03:16, John Snow wrote: > > We can drop the sys.path hacking in various places by doing > > this. Additionally, by doing it in one place right up top, we can

Re: [PATCH 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-23 Thread John Snow
On Thu, Sep 23, 2021 at 7:09 AM Daniel P. Berrangé wrote: > On Wed, Sep 22, 2021 at 08:16:21PM -0400, John Snow wrote: > > Add a warning for when 'iotests' runs against a qemu namespace that > > isn't the one in the source tree. This might occur if you have > > (ac

Re: [PATCH 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-23 Thread John Snow
On Thu, Sep 23, 2021 at 7:17 AM Kevin Wolf wrote: > Am 23.09.2021 um 12:57 hat Kevin Wolf geschrieben: > > Am 23.09.2021 um 02:16 hat John Snow geschrieben: > > > Add a warning for when 'iotests' runs against a qemu namespace that > > > isn't the one in the source tr

[PATCH v2 12/17] python/machine: Handle QMP errors on close more meticulously

2021-09-22 Thread John Snow
' command. That error will ultimately be bubbled up when attempting to close the QMP connection. The manager class here then is free to discard it -- if it was expected. Signed-off-by: John Snow --- Yes, I regret that this class has become quite a dumping ground for complexity around the exit path

[PATCH v2 11/17] python/machine: remove has_quit argument

2021-09-22 Thread John Snow
ut down. Signed-off-by: John Snow --- python/qemu/machine/machine.py | 34 +++--- tests/qemu-iotests/040 | 7 +-- tests/qemu-iotests/218 | 2 +- tests/qemu-iotests/255 | 2 +- 4 files changed, 22 insertions(+), 23 deletions(-) diff --git a

[PATCH v2 15/17] python/aqmp: Create sync QMP wrapper for iotests

2021-09-22 Thread John Snow
This is a wrapper around the async QMPClient that mimics the old, synchronous QEMUMonitorProtocol class. It is designed to be interchangeable with the old implementation. It does not, however, attempt to mimic Exception compatibility. Signed-off-by: John Snow Acked-by: Hanna Reitz --- python

[PATCH v2 09/17] python/qmp: add send_fd_scm directly to QEMUMonitorProtocol

2021-09-22 Thread John Snow
implementation details. /that/ is helpful in turn because it allows me to write a compatible, alternative implementation. Signed-off-by: John Snow Reviewed-by: Hanna Reitz --- python/qemu/machine/machine.py | 44 +++--- python/qemu/qmp/__init__.py| 21 +++- 2

[PATCH v2 14/17] iotests: Conditionally silence certain AQMP errors

2021-09-22 Thread John Snow
can't amend the iotest output to just always expect the error message, either. Just suppress it for now. More rigorous log filtering can be investigated later if/when it is deemed safe to permanently replace the legacy QMP library.) Signed-off-by: John Snow --- tests/qemu-iotests/iotests.py

[PATCH v2 08/17] python/qmp: clear events on get_events() call

2021-09-22 Thread John Snow
the retrieval also clear the queue is vastly simpler. Signed-off-by: John Snow Reviewed-by: Hanna Reitz --- python/qemu/machine/machine.py | 1 - python/qemu/qmp/__init__.py| 6 -- python/qemu/qmp/qmp_shell.py | 1 - 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/python/qemu

[PATCH v2 07/17] python/aqmp: Disable logging messages by default

2021-09-22 Thread John Snow
that will suppress these messages until a client application opts into logging via logging.basicConfig or similar. Note that upon calling basicConfig(), this handler will *not* suppress these messages from being displayed by the client's configuration. Signed-off-by: John Snow --- python/qemu/aqmp/__init__

[PATCH v2 16/17] python/aqmp: Remove scary message

2021-09-22 Thread John Snow
The scary message interferes with the iotests output. Coincidentally, if iotests works by removing this, then it's good evidence that we don't really need to scare people away from using it. Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 12 1 file changed, 12

[PATCH v2 13/17] iotests: Accommodate async QMP Exception classes

2021-09-22 Thread John Snow
. That'd be these: - QMPError - QMPConnectError - QMPCapabilitiesError - QMPTimeoutError - QMPProtocolError - QMPResponseError - QMPBadPortError Signed-off-by: John Snow --- scripts/simplebench/bench_block_job.py| 3 ++- tests/qemu-iotests/tests/mirror-top-perms | 3 ++- 2 files changed, 4

[PATCH v2 17/17] python, iotests: replace qmp with aqmp

2021-09-22 Thread John Snow
work concurrently. Signed-off-by: John Snow Reviewed-by: Hanna Reitz Tested-by: Hanna Reitz --- python/qemu/machine/machine.py | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/python/qemu/machine/machine.py b/python/qemu/machine/machine.py index c33a78a2d9f..32879faeb40

[PATCH v2 10/17] python, iotests: remove socket_scm_helper

2021-09-22 Thread John Snow
It's not used anymore, now. Signed-off-by: John Snow Reviewed-by: Hanna Reitz --- tests/qemu-iotests/socket_scm_helper.c | 136 - python/qemu/machine/machine.py | 3 - python/qemu/machine/qtest.py | 2 - tests/Makefile.include | 1

[PATCH v2 04/17] python/aqmp: add send_fd_scm

2021-09-22 Thread John Snow
ust boldly void the warranty instead, while I try to follow up on https://bugs.python.org/issue43232 Signed-off-by: John Snow --- python/qemu/aqmp/qmp_client.py | 22 ++ 1 file changed, 22 insertions(+) diff --git a/python/qemu/aqmp/qmp_client.py b/python/qemu/aqmp/qmp_clien

[PATCH v2 01/17] python/aqmp: add greeting property to QMPClient

2021-09-22 Thread John Snow
Expose the greeting as a read-only property of QMPClient so it can be retrieved at-will. Signed-off-by: John Snow Reviewed-by: Hanna Reitz --- python/qemu/aqmp/qmp_client.py | 5 + 1 file changed, 5 insertions(+) diff --git a/python/qemu/aqmp/qmp_client.py b/python/qemu/aqmp/qmp_client.py

[PATCH v2 02/17] python/aqmp: add .empty() method to EventListener

2021-09-22 Thread John Snow
Synchronous clients may want to know if they're about to block waiting for an event or not. A method such as this is necessary to implement a compatible interface for the old QEMUMonitorProtocol using the new async internals. Signed-off-by: John Snow Reviewed-by: Hanna Reitz --- python/qemu

[PATCH v2 06/17] python/aqmp: Reduce severity of EOFError-caused loop terminations

2021-09-22 Thread John Snow
the normal operation of the library. Signed-off-by: John Snow --- python/qemu/aqmp/protocol.py | 7 +-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py index 32e78749c11..ae1df240260 100644 --- a/python/qemu/aqmp/prot

[PATCH v2 05/17] python/aqmp: Add dict conversion method to Greeting object

2021-09-22 Thread John Snow
The iotests interface expects to return the greeting as a dict; AQMP offers it as a rich object. Signed-off-by: John Snow --- python/qemu/aqmp/models.py | 13 + 1 file changed, 13 insertions(+) diff --git a/python/qemu/aqmp/models.py b/python/qemu/aqmp/models.py index 24c94123ac0

[PATCH v2 03/17] python/aqmp: Return cleared events from EventListener.clear()

2021-09-22 Thread John Snow
This serves two purposes: (1) It is now possible to discern whether or not clear() removed any event(s) from the queue with absolute certainty, and (2) It is now very easy to get a List of all pending events in one chunk, which is useful for the sync bridge. Signed-off-by: John Snow Reviewed

[PATCH v2 00/17] Switch iotests to using Async QMP

2021-09-22 Thread John Snow
ption handling logic in 13 (Hanna) - Introduce a helper in patch 14 to silence log output when it's unwanted. - Small addition of _get_greeting() helper in patch 15, coinciding with the new patch 05 here. - Contextual changes in 16. John Snow (17): python/aqmp: add greeting property t

[PATCH 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-22 Thread John Snow
in the blanks.) Signed-off-by: John Snow --- tests/qemu-iotests/testenv.py | 21 - 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/testenv.py b/tests/qemu-iotests/testenv.py index 88104dace90..8a43b193af5 100644 --- a/tests/qemu-iotests/testenv.py

[PATCH 5/6] iotests/migrate-bitmaps-test: delint

2021-09-22 Thread John Snow
Mostly uninteresting stuff. Move the test injections under a function named main() so that the variables used during that process aren't in the global scope. Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Hanna Reitz --- tests/qemu-iotests/tests/migrate-bitmaps-test

[PATCH 4/6] iotests/mirror-top-perms: Adjust imports

2021-09-22 Thread John Snow
We need to import things from the qemu namespace; importing the namespace alone doesn't bring the submodules with it -- unless someone else (like iotests.py) imports them too. Adjust the imports. Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Hanna Reitz --- tests

[PATCH 6/6] iotests: Update for pylint 2.11.1

2021-09-22 Thread John Snow
-off-by: John Snow --- tests/qemu-iotests/pylintrc | 6 +- tests/qemu-iotests/testrunner.py | 7 --- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/tests/qemu-iotests/pylintrc b/tests/qemu-iotests/pylintrc index f2c0b522ac0..8cb4e1d6a6d 100644 --- a/tests/qemu-iotests

[PATCH 3/6] iotests/linters: check mypy files all at once

2021-09-22 Thread John Snow
We can circumvent the '__main__' redefinition problem by passing --scripts-are-modules. Take mypy out of the loop per-filename and check everything in one go: it's quite a bit faster. Signed-off-by: John Snow Reviewed-by: Hanna Reitz --- tests/qemu-iotests/297 | 44

[PATCH 1/6] iotests: add 'qemu' package location to PYTHONPATH in testenv

2021-09-22 Thread John Snow
ython packages" goop is all in one place, right in one function. Signed-off-by: John Snow --- tests/qemu-iotests/235| 2 -- tests/qemu-iotests/297| 6 -- tests/qemu-iotests/300| 7 +++ tests/qemu-iotests/iotests.py

[PATCH 0/6] iotests: update environment and linting configuration

2021-09-22 Thread John Snow
nding some discussion on the other series. --js John Snow (6): iotests: add 'qemu' package location to PYTHONPATH in testenv iotests: add warning for rogue 'qemu' packages iotests/linters: check mypy files all at once iotests/mirror-top-perms: Adjust imports iotests/migrate-bitmaps-te

Re: [PATCH v3 16/16] iotests/linters: check mypy files all at once

2021-09-22 Thread John Snow
On Fri, Sep 17, 2021 at 7:23 AM Hanna Reitz wrote: > On 16.09.21 06:09, John Snow wrote: > > We can circumvent the '__main__' redefinition problem by passing > > --scripts-are-modules. Take mypy out of the loop per-filename and check > > everything in one go: it

Re: [PATCH v3 14/16] iotests/linters: Add workaround for mypy bug #9852

2021-09-22 Thread John Snow
On Wed, Sep 22, 2021 at 4:37 PM John Snow wrote: > > On Fri, Sep 17, 2021 at 7:16 AM Hanna Reitz wrote: > >> >> Question is, when “can we use” mypy >= 0.920? Should we check the >> version string and append this switch as required? >> >> > The ans

Re: [PATCH v3 14/16] iotests/linters: Add workaround for mypy bug #9852

2021-09-22 Thread John Snow
On Fri, Sep 17, 2021 at 7:16 AM Hanna Reitz wrote: > On 16.09.21 06:09, John Snow wrote: > > This one is insidious: if you use the invocation > > "from {namespace} import {subpackage}" as mirror-top-perms does, > > mypy will fail on every-other invocation *if*

Re: [PATCH v3 11/16] iotests/297: return error code from run_linters()

2021-09-22 Thread John Snow
On Fri, Sep 17, 2021 at 7:00 AM Hanna Reitz wrote: > On 16.09.21 06:09, John Snow wrote: > > This turns run_linters() into a bit of a hybrid test; returning non-zero > > on failed execution while also printing diffable information. This is > > done for the benefit of th

Re: [PATCH v3 07/16] iotests/297: Don't rely on distro-specific linter binaries

2021-09-22 Thread John Snow
(This email just explains python packaging stuff. No action items in here. Skim away.) On Fri, Sep 17, 2021 at 5:43 AM Hanna Reitz wrote: > On 16.09.21 06:09, John Snow wrote: > > 'pylint-3' is another Fedora-ism. Use "python3 -m pylint" or "python3 -m > > mypy&qu

Re: [PATCH v3 03/16] iotests/migrate-bitmaps-postcopy-test: declare instance variables

2021-09-22 Thread John Snow
On Fri, Sep 17, 2021 at 4:37 AM Hanna Reitz wrote: > On 16.09.21 06:09, John Snow wrote: > > Signed-off-by: John Snow > > --- > > tests/qemu-iotests/tests/migrate-bitmaps-postcopy-test | 3 +++ > > 1 file changed, 3 insertions(+) > > > > diff --git a/t

Re: [PATCH v3 06/16] iotests/297: Add get_files() function

2021-09-22 Thread John Snow
On Fri, Sep 17, 2021 at 5:24 AM Hanna Reitz wrote: > On 16.09.21 06:09, John Snow wrote: > > Split out file discovery into its own method to begin separating out the > > "environment setup" and "test execution" phases. > > > > Signed-off-by: John

Re: [PATCH 12/15] iotests: Disable AQMP logging under non-debug modes

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 8:58 PM John Snow wrote: > > > On Fri, Sep 17, 2021 at 10:30 AM Hanna Reitz wrote: > >> On 17.09.21 07:40, John Snow wrote: >> > Disable the aqmp logger, which likes to (at the moment) print out >> > intermediate warnings and e

Re: [PATCH 13/15] iotests: Accommodate async QMP Exception classes

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 10:35 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > (But continue to support the old ones for now, too.) > > > > There are very few cases of any user of QEMUMachine or a subclass > > thereof relying on a QMP Exception t

Re: [PATCH 12/15] iotests: Disable AQMP logging under non-debug modes

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 10:30 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > Disable the aqmp logger, which likes to (at the moment) print out > > intermediate warnings and errors that cause session termination; disable > > them so they don't interfer

Re: [PATCH 11/15] python/aqmp: Create sync QMP wrapper for iotests

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 10:23 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > This is a wrapper around the async QMPClient that mimics the old, > > synchronous QEMUMonitorProtocol class. It is designed to be > > interchangeable with the old implementa

Re: [PATCH 10/15] python/machine: Add support for AQMP backend

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 10:16 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > To use the AQMP backend, Machine just needs to be a little more diligent > > about what happens when closing a QMP connection. The operation is no > > longer a freeb

Re: [PATCH 09/15] python/machine: remove has_quit argument

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 9:59 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > If we spy on the QMP commands instead, we don't need callers to remember > > to pass it. Seems like a fair trade-off. > > > > The one slightly weird bit is overloading this ins

Re: [PATCH 08/15] python/aqmp: Create MessageModel and StandaloneModel classes

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021, 9:39 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > This allows 'Greeting' to be subclass of 'Message'. We need the adapter > > classes to avoid some typing problems that occur if we try to put too > > much into the 'Model' class itse

Re: [PATCH 07/15] python/aqmp: add send_fd_scm

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 9:34 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > The single space is indeed required to successfully transmit the file > > descriptor to QEMU. > > Yeah, socket_scm_helper.c said “Send a blank to notify qemu”. > >

Re: [PATCH 05/15] python/qmp: add send_fd_scm directly to QEMUMonitorProtocol

2021-09-17 Thread John Snow
On Fri, Sep 17, 2021 at 9:21 AM Hanna Reitz wrote: > On 17.09.21 07:40, John Snow wrote: > > It turns out you can do this directly from Python ... and because of > > this, you don't need to worry about setting the inheritability of the > > fds or spawning another p

<    5   6   7   8   9   10   11   12   13   14   >