On Wed, Sep 15, 2021 at 12:30 PM John Snow <js...@redhat.com> wrote: > GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-async-qmp-aqmp > CI: https://gitlab.com/jsnow/qemu/-/pipelines/371343890 > Docs: https://people.redhat.com/~jsnow/sphinx/html/qemu.aqmp.html > Based-on: <20210915154031.321592-2-js...@redhat.com> > > (This is a quick V4 to add a few tiny edits and get the proper mboxes > out on the list. My current plan is to send a PR for this series this > week. Don't panic, it isn't used by default anywhere yet, nothing should > break.) > > Hi! > > This patch series adds an Asynchronous QMP package to the Python > library. It offers a few improvements over the previous library: > > - out-of-band support > - true asynchronous event support > - avoids undocumented interfaces abusing non-blocking sockets > - unit tests! > - documentation! > > This library serves as the basis for a new qmp-shell program that will > offer improved reconnection support, true asynchronous display of > events, VM and job status update notifiers, and so on. > > My intent is to eventually publish this library directly to PyPI as a > standalone package. I would like to phase out our usage of the old QMP > library over time; eventually replacing it entirely with this > one. (Since v2 of this series, I have authored a compatibility shim not > included in this series that can be used to run all of iotests on this > new library successfully with very minimal churn.) > > This series looks big by line count, but it's *mostly* > docstrings. Seriously! > > This package has *no* external dependencies whatsoever. > > Notes & Design > ============== > > Here are some notes on the design of how the library works, to serve as > a primer for review; however I also **highly recommend** browsing the > generated Sphinx documentation for this series. > > Here's that link again: > https://people.redhat.com/~jsnow/sphinx/html/qemu.aqmp.html > > The core machinery is split between the AsyncProtocol and QMPClient > classes. AsyncProtocol provides the generic machinery, while QMPClient > provides the QMP-specific details. > > The design uses two independent coroutines that act as the "bottom > half", a writer task and a reader task. These tasks run for the duration > of the connection and independently send and receive messages, > respectively. > > A third task, disconnect, is scheduled asynchronously whenever an > unrecoverable error occurs and facilitates coalescing of the other two > tasks. > > This diagram for how execute() operates may be helpful for understanding > how AsyncProtocol is laid out. The arrows indicate the direction of a > QMP message; the long horizontal dash indicates the separation between > the upper and lower halves of the event loop. The queue mechanisms > between both dashes serve as the intermediaries between the upper and > lower halves. > > +---------+ > | caller | > +---------+ > ^ | > | v > +---------+ > +---------------> |execute()| -----------+ > | +---------+ | > | | > [-----------------------------------------------------------] > | | > | v > +----+------+ +----------------+ +------+-------+ > | ExecQueue | | EventListeners | |Outbound Queue| > +----+------+ +----+-----------+ +------+-------+ > ^ ^ | > | | | > [-----------------------------------------------------------] > | | | > | | v > +--+----------------+---+ +-----------+-----------+ > | Reader Task/Coroutine | | Writer Task/Coroutine | > +-----------+-----------+ +-----------+-----------+ > ^ | > | v > +-----+------+ +-----+------+ > |StreamReader| |StreamWriter| > +------------+ +------------+ > > The caller will invoke execute(), which in turn will deposit a message > in the outbound send queue. This will wake up the writer task, which > well send the message over the wire. > > The execute() method will then yield to wait for a reply delivered to an > execution queue created solely for that execute statement. > > When a message arrives, the Reader task will unblock and route the > message either to the EventListener subsystem, or place it in the > appropriate pending execution queue. > > Once a message is placed in the pending execution queue, execute() will > unblock and the execution will conclude, returning the result of the RPC > call to the caller. > > Patch Layout > ============ > > Patches 1-4 add tiny pre-requisites, utilities, etc. > Patches 5-12 add a generic async message-based protocol class, > AsyncProtocol. They are split fairly small and should > be reasonably self-contained. > Patches 13-15 check in more QMP-centric components. > Patches 16-21 add qmp_client.py, with a new 'QMPClient()' class. > They're split into reasonably tiny pieces here. > Patches 22-23 add a few finishing touches, they are small patches. > Patches 24-27 adds unit tests. They're a little messy still, but > they've been quite helpful to me so far. Coverage of > protocol.py is at about ~86%. > > Future Work > =========== > > These items are in progress: > > - A synchronous QMP wrapper that allows this library to be easily used > from non-async code; this will also allow me to prove it works well by > demoing its replacement throughout iotests. > > This work is feature-complete, but needs polish. All of iotests is now > passing with Async QMP and this Sync wrapper. This will be its own > follow-up series. > > - A QMP server class; to facilitate writing of unit tests. An early > version is done, but possibly not feature complete. More polish and > tests are warranted. This will be its own follow-up series. > > - More unit tests for qmp_client.py, qmp_server.py and other modules. > > Changelog > ========= > > V4: > > - (06) Minor typo fix in comment (Eric Blake) > - (25) Removed stale null_protocol.py file. > - (26, 27) New - increase test coverage and add coverage.py support. > > V3: > > - (02, 05) Typo fixes (Eric Blake) > - (04) Rewrote the "wait_closed" compatibility function for Python 3.6; > the older version raised unwanted exceptions in error pathways. > (Niteesh) > - (04, 05, 06, 08) Rewrote _bh_disconnect fairly substantially again; > the problem is that exceptions can surface during both flushing of the > stream and when waiting for the stream to close. These errors can be > new, primary causes of failure or secondary failures. Distinguishing > between them is tricky. The new disconnection method takes much > greater pains to ensure that even if Exceptions occur, disconnect > *will* complete. This adds robustness to cases exposed by iotests > where one or more endpoints might segfault or abort and cleanup can be > challenged. > - (11) Fixed logging hook names (Niteesh) > - (24, 25) Bumped avocado dependency to v90; It added support for async > test functions which made my prior workaround non-suitable. The > choices were to mandate <90 and keep the workarounds or mandate >=90 > and drop the workarounds. I went with the latter. > > V2: > > Renamed classes/methods: > > - Renamed qmp_protocol.py to qmp_client.py > - Renamed 'QMP' class to 'QMPClient' > - Renamed _begin_new_session() to _establish_session() > - Split _establish_connection() out from _new_session(). > - Removed _results() method > > Bugfixes: > > - Suppress duplicate Exceptions when attempting to drain the > StreamWriter > - Delay initialization of asyncio.Queue and asyncio.Event variables to > _new_session or later -- they must not be created outside of the loop, > even if they are not async functions. > - Rework runstate_changed events to guarantee visibility of events to > waiters > - Improve connect()/accept() cleanup to work with > asyncio.CancelledError, asyncio.TimeoutError > - No-argument form of Message() now succeeds properly. > - flush utility will correctly yield when data is below the "high water > mark", giving the stream a chance to actually flush. > - Increase read buffer size to accommodate query-qmp-schema (Thanks > Niteesh) > > Ugly bits from V1 removed: > > - Remove tertiary filtering from EventListener (for now), accompanying > documentation removed from events.py > - Use asyncio.wait() instead of custom wait_task_done() > - MultiException is removed in favor of just raising the first Exception > that occurs in the bottom half; other Exceptions if any are logged > instead. > > Improvements: > > - QMPClient now allows ID-less execution statements via the _raw() > interface. > - Add tests that grant ~86% coverage of protocol.py to the avocado test > suite. > - Removed 'force' parameter from _bh_disconnect; the disconnection > routine determines for itself if we are in the error pathway or not > instead now. This removes any chance of duplicate calls to > _schedule_disconnect accidentally dropping the 'force' setting. > > Debugging/Testing changes: > > - Add debug: bool parameter to asyncio_run utility wrapper > - Improve error messages for '@require' decorator > - Add debugging message for state change events > - Avoid flushing the StreamWriter if we don't have one (This > circumstance only arises in testing, but it's helpful.) > - Improved __repr__ method for AsyncProtocol, and removed __str__ > method. enforcing eval(__repr__(x)) == x does not make sense for > AsyncProtocol. > - Misc logging message changes > - Add a suite of fancy Task debugging utilities. > - Most tracebacks now log at the DEBUG level instead of > CRITICAL/ERROR/WARNING; In those error cases, a one-line summary is > logged instead. > > Misc. aesthetic changes: > > - Misc docstring fixes, whitespace, etc. > - Reordered the definition of some methods to try and keep similar > methods near each other (Moved _cleanup near _bh_disconnect in > QMPClient.) > > John Snow (27): > python/aqmp: add asynchronous QMP (AQMP) subpackage > python/aqmp: add error classes > python/pylint: Add exception for TypeVar names ('T') > python/aqmp: add asyncio compatibility wrappers > python/aqmp: add generic async message-based protocol support > python/aqmp: add runstate state machine to AsyncProtocol > python/aqmp: Add logging utility helpers > python/aqmp: add logging to AsyncProtocol > python/aqmp: add AsyncProtocol.accept() method > python/aqmp: add configurable read buffer limit > python/aqmp: add _cb_inbound and _cb_outbound logging hooks > python/aqmp: add AsyncProtocol._readline() method > python/aqmp: add QMP Message format > python/aqmp: add well-known QMP object models > python/aqmp: add QMP event support > python/pylint: disable too-many-function-args > python/aqmp: add QMP protocol support > python/pylint: disable no-member check > python/aqmp: Add message routing to QMP protocol > python/aqmp: add execute() interfaces > python/aqmp: add _raw() execution interface > python/aqmp: add asyncio_run compatibility wrapper > python/aqmp: add scary message > python: bump avocado to v90.0 > python/aqmp: add AsyncProtocol unit tests > python/aqmp: add LineProtocol tests > python/aqmp: Add Coverage.py support > > python/.gitignore | 5 + > python/Makefile | 9 + > python/Pipfile.lock | 8 +- > python/avocado.cfg | 3 + > python/qemu/aqmp/__init__.py | 59 +++ > python/qemu/aqmp/error.py | 50 ++ > python/qemu/aqmp/events.py | 706 ++++++++++++++++++++++++++ > python/qemu/aqmp/message.py | 209 ++++++++ > python/qemu/aqmp/models.py | 133 +++++ > python/qemu/aqmp/protocol.py | 902 +++++++++++++++++++++++++++++++++ > python/qemu/aqmp/py.typed | 0 > python/qemu/aqmp/qmp_client.py | 621 +++++++++++++++++++++++ > python/qemu/aqmp/util.py | 217 ++++++++ > python/setup.cfg | 17 +- > python/tests/protocol.py | 583 +++++++++++++++++++++ > 15 files changed, 3516 insertions(+), 6 deletions(-) > create mode 100644 python/qemu/aqmp/__init__.py > create mode 100644 python/qemu/aqmp/error.py > create mode 100644 python/qemu/aqmp/events.py > create mode 100644 python/qemu/aqmp/message.py > create mode 100644 python/qemu/aqmp/models.py > create mode 100644 python/qemu/aqmp/protocol.py > create mode 100644 python/qemu/aqmp/py.typed > create mode 100644 python/qemu/aqmp/qmp_client.py > create mode 100644 python/qemu/aqmp/util.py > create mode 100644 python/tests/protocol.py > > -- > 2.31.1 > > > Thanks, I have preliminarily staged this to my python branch: https://gitlab.com/jsnow/qemu/-/commits/python
--js
