Thank you, Philippe, those are great ideas. I have copied them to the Avocado+QEMU Trello board so we don't forget about them: https://trello.com/b/6Qi1pxVn/avocado-qemu
Additional comments below: On Mon, Nov 25, 2019 at 01:35:13PM +0100, Philippe Mathieu-Daudé wrote: > Hi Cleber, > > Here are my notes from talking about Avocado with various people during the > KVM forum in Lyon last month. > > All comments are QEMU oriented. > > > 1) Working offline > > Various people complained Avocado requires online access, and they would > like to use it offline. > > Maintainer workflow example is: > > - run avocado > - hack QEMU, build > - git pull > - build > - hack QEMU > (go offline) > - hack QEMU > - build > - run avocado <- FAILS > Ouch. This shouldn't happen even with no explicit --offline option. Failure to download artifacts shouldn't make tests report failure. > Failure is because mainstream added new tests, which got pulled in, and the > user only notice when running avocado again, but offline. > Test example is boot_linux_console.py, which added various tests from other > subsystems, so the maintainer has to disable the new tests manually to be > able to run his previous tests. > > Expected solution: skip tests when artifact is not available, eventually > when the --offline option is used > > > 2) Add artifacts manually to the cache > > Not all artifacts can be easily downloadable, some are public but require > the user to accept an End User License Agreement. > Users would like to share their tests with the documentation about where/how > to download the requisite files (accepting the EULA) to run the tests. > > > 2b) Add reference to artifact to the cache > > Group of users might share group of files (like NFS storage) and would like > to use directly their remote read-only files, instead of copying it to their > home directory. This sounds nice and useful, but I don't know how to make the interface for this usable. > > > 3) Provide qemu/avocado-qemu Python packages > > The mainstream project uses Avocado to test QEMU. Others projects use QEMU > to test their code, and would like to automatize that using Avocado. They > usually not rebuild QEMU but use a stable binary from distributions. The > python classes are not available, so they have to clone QEMU to use Avocado > (I guess they only need 5 python files). > When running on Continuous Integration, this is overkill, because when you > clone QEMU you also clone various other submodules. I only have one concern, here: I don't think we have the bandwidth to start maintaining a stable external Python API. Users of those packages will need to be aware that future versions of the modules might have incompatible APIs. > > > 4) Warn the user when Avocado is too old for the tests > > Some users tried Avocado following the examples on the mailing list and the > one in some commit's descriptions where we simply show "avocado run ...". Oops. > They installed the distribution Avocado package and tried and it fails for > few of them with no obvious reason (the .log file is hard to read when you > are not custom to). IIUC their distribution provides a older Avocado (69?) > while we use recent features (72). > > We never noticed it because we use 'make check-venv' and do not test the > distrib Avocado. While we can not test all distribs, we could add a version > test if the Avocado version is too old, display a friendly message to the > console (not the logfile). Sounds like a good idea. > > > That it for my notes. > > Eduardo/Wainer, are there other topics I forgot? I don't remember anything specific right now. Thanks again! -- Eduardo
