Re: [PULL 00/44] Python patches
On 6/1/21 6:36 AM, Peter Maydell wrote: On Sun, 30 May 2021 at 20:22, John Snow wrote: On 5/30/21 3:09 PM, Peter Maydell wrote: Fails to build on my machine that runs the BSD VMs, apparently before it gets to the point of launching the VM: When I have seen this error message in the past, it has been because of using a new Python version and it gets confused reading stale cached information generated from an older interpreter. This seems like a bug in Python... Yes. I don't know why it's like that either... Can you do me a favor and delete any __pycache__ folders and/or any *.pyc files that might be hiding in your tree and/or build folders and try running it again? I tried that, and it does seem to make more progress. It now fails with: One of the acceptance test patches causes this, I've squashed in a fixup and have re-sent the PR. Thanks, --js
Re: [PULL 00/44] Python patches
On Sun, 30 May 2021 at 20:22, John Snow wrote: > > On 5/30/21 3:09 PM, Peter Maydell wrote: > > Fails to build on my machine that runs the BSD VMs, apparently > > before it gets to the point of launching the VM: > When I have seen this error message in the past, it has been because of > using a new Python version and it gets confused reading stale cached > information generated from an older interpreter. This seems like a bug in Python... > Can you do me a favor and delete any __pycache__ folders and/or any > *.pyc files that might be hiding in your tree and/or build folders and > try running it again? I tried that, and it does seem to make more progress. It now fails with: ### Booting installer ... DEBUG:root:QEMU args: -nodefaults -m 4G -cpu max,sse4.2=off -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 18 -enable-kvm -drive file=/home/peter.maydell/.cache/qemu-vm/images/freebsd.img.tmp,if=none,id=drive0,cache=writeback -device virtio-blk,drive=drive0,bootindex=0 -bios pc-bios/bios-256k.bin -machine graphics=off -device VGA -cdrom /home/peter.maydell/.cache/qemu-vm/images/freebsd.img.install.iso DEBUG:qemu.machine.machine:VM launch command: 'qemu-system-x86_64 -display none -vga none -chardev socket,id=mon,path=/var/tmp/qemu-13084-monitor.sock -mon chardev=mon,mode=control -machine pc -chardev socket,id=console,path=/var/tmp/qemu-13084-console.sock,server=on,wait=off -serial chardev:console -nodefaults -m 4G -cpu max,sse4.2=off -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 18 -enable-kvm -drive file=/home/peter.maydell/.cache/qemu-vm/images/freebsd.img.tmp,if=none,id=drive0,cache=writeback -device virtio-blk,drive=drive0,bootindex=0 -bios pc-bios/bios-256k.bin -machine graphics=off -device VGA -cdrom /home/peter.maydell/.cache/qemu-vm/images/freebsd.img.install.iso' DEBUG:QMP.qemu-13084:>>> {'execute': 'qmp_capabilities'} DEBUG:QMP.qemu-13084:<<< {'return': {}} DEBUG:QMP.qemu-13084:>>> {'execute': 'human-monitor-command', 'arguments': {'command-line': 'info usernet'}} DEBUG:QMP.qemu-13084:<<< {'return': 'Hub -1 (vnet):\r\n Protocol[State]FD Source Address Port Dest. Address Port RecvQ SendQ\r\n TCP[HOST_FORWARD] 15 127.0.0.1 38577 10.0.2.1522 0 0\r\n'} Failed to prepare guest environment Traceback (most recent call last): File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 630, in main return vm.build_image(args.image) File "/home/peter.maydell/qemu-freebsd/tests/vm/freebsd", line 103, in build_image "-cdrom", iso File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 309, in boot self.ssh_port = get_info_usernet_hostfwd_port(usernet_info) File "/home/peter.maydell/qemu-freebsd/tests/vm/../../python/qemu/utils/__init__.py", line 40, in get_info_usernet_hostfwd_port for line in info_usernet_output.split('\r\n'): AttributeError: 'dict' object has no attribute 'split' DEBUG:QMP.qemu-13084:>>> {'execute': 'quit'} DEBUG:QMP.qemu-13084:<<< {'return': {}} /home/peter.maydell/qemu-freebsd/tests/vm/Makefile.include:79: recipe for target '/home/peter.maydell/.cache/qemu-vm/images/freebsd.img' failed make: *** [/home/peter.maydell/.cache/qemu-vm/images/freebsd.img] Error 2 make: Leaving directory '/home/peter.maydell/qemu-freebsd/build' thanks -- PMM
Re: [PULL 00/44] Python patches
On 5/30/21 9:22 PM, John Snow wrote: > On 5/30/21 3:09 PM, Peter Maydell wrote: >> Fails to build on my machine that runs the BSD VMs, apparently >> before it gets to the point of launching the VM: >> >> make: Entering directory '/home/peter.maydell/qemu-freebsd/build' >> /usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/meson/meson.py >> introspect --targets --tests --benchmarks | /usr/bin/python3 -B >> scripts/mtest2make.py > Makefile.mtest >> { \ >> echo 'ninja-targets = \'; \ >> /usr/bin/ninja -t targets all | sed 's/:.*//; $!s/$/ \\/'; \ >> echo 'build-files = \'; \ >> /usr/bin/ninja -t query build.ninja | sed -n '1,/^ input:/d; /^ >> outputs:/q; s/$/ \\/p'; \ >> } > Makefile.ninja.tmp && mv Makefile.ninja.tmp Makefile.ninja >> (GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh" >> update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3 >> tests/fp/berkeley-softfloat-3 dtc capstone slirp) >> (GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh" >> update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3 >> tests/fp/berkeley-softfloat-3 dtc capstone slirp) >> /usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/tests/vm/freebsd >> --debug --genisoimage /usr/bin/genisoimage --image >> "/home/peter.maydell/.cache/qemu-vm/images/freebsd.img" --force >> --build-image /home/peter.maydell/.cache/qemu-vm/images/freebsd.img >> Traceback (most recent call last): >> File "/home/peter.maydell/qemu-freebsd/tests/vm/freebsd", line 21, >> in >> import basevm >> File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 22, >> in >> from qemu.machine import QEMUMachine >> ImportError: bad magic number in 'qemu': b'\x03\xf3\r\n' >> /home/peter.maydell/qemu-freebsd/tests/vm/Makefile.include:79: recipe >> for target '/home/peter.maydell/.cache/qemu-vm/images/freebsd.img' >> failed >> make: *** [/home/peter.maydell/.cache/qemu-vm/images/freebsd.img] Error 1 >> make: Leaving directory '/home/peter.maydell/qemu-freebsd/build' > In case that doesn't fix it, can you tell me what version of Python your > BSD setup uses? (And what version of FreeBSD you use?) -- I'll try to > set up a VM and see if I can reproduce the problem. IIUC correctly this is on the host (likely Linux); no BSD VM could be launched yet.
Re: [PULL 00/44] Python patches
On 5/30/21 3:09 PM, Peter Maydell wrote: On Fri, 28 May 2021 at 03:32, John Snow wrote: The following changes since commit c8616fc7670b884de5f74d2767aade224c1c5c3a: Merge remote-tracking branch 'remotes/philmd/tags/gitlab-ci-20210527' into staging (2021-05-27 16:32:57 +0100) are available in the Git repository at: https://gitlab.com/jsnow/qemu.git tags/python-pull-request for you to fetch changes up to faa40e297406445ab8814844bd9aa532a7538cde: gitlab: add python linters to CI (2021-05-27 18:11:17 -0400) Python pull request Python packaging & CI implementation Acceptance tests sent w/ Cleber's blessing. New CI tests send w/ stsquad's RB. --js Fails to build on my machine that runs the BSD VMs, apparently before it gets to the point of launching the VM: make: Entering directory '/home/peter.maydell/qemu-freebsd/build' /usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/meson/meson.py introspect --targets --tests --benchmarks | /usr/bin/python3 -B scripts/mtest2make.py > Makefile.mtest { \ echo 'ninja-targets = \'; \ /usr/bin/ninja -t targets all | sed 's/:.*//; $!s/$/ \\/'; \ echo 'build-files = \'; \ /usr/bin/ninja -t query build.ninja | sed -n '1,/^ input:/d; /^ outputs:/q; s/$/ \\/p'; \ } > Makefile.ninja.tmp && mv Makefile.ninja.tmp Makefile.ninja (GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh" update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc capstone slirp) (GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh" update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc capstone slirp) /usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/tests/vm/freebsd --debug --genisoimage /usr/bin/genisoimage--image "/home/peter.maydell/.cache/qemu-vm/images/freebsd.img" --force --build-image /home/peter.maydell/.cache/qemu-vm/images/freebsd.img Traceback (most recent call last): File "/home/peter.maydell/qemu-freebsd/tests/vm/freebsd", line 21, in import basevm File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 22, in from qemu.machine import QEMUMachine ImportError: bad magic number in 'qemu': b'\x03\xf3\r\n' /home/peter.maydell/qemu-freebsd/tests/vm/Makefile.include:79: recipe for target '/home/peter.maydell/.cache/qemu-vm/images/freebsd.img' failed make: *** [/home/peter.maydell/.cache/qemu-vm/images/freebsd.img] Error 1 make: Leaving directory '/home/peter.maydell/qemu-freebsd/build' thanks -- PMM Hmm ... When I have seen this error message in the past, it has been because of using a new Python version and it gets confused reading stale cached information generated from an older interpreter. None of these patches *should* have affected what version of python is used during the build, only the locations of some of the files. I don't immediately know why this PR would affect the QEMU build at all, actually. Can you do me a favor and delete any __pycache__ folders and/or any *.pyc files that might be hiding in your tree and/or build folders and try running it again? (If it helps, I can see about amending 'make clean' to try and clean up stale Python cache, which is the best solution I can think of for right now.) In case that doesn't fix it, can you tell me what version of Python your BSD setup uses? (And what version of FreeBSD you use?) -- I'll try to set up a VM and see if I can reproduce the problem. --js
Re: [PULL 00/44] Python patches
On Fri, 28 May 2021 at 03:32, John Snow wrote: > > The following changes since commit c8616fc7670b884de5f74d2767aade224c1c5c3a: > > Merge remote-tracking branch 'remotes/philmd/tags/gitlab-ci-20210527' into > staging (2021-05-27 16:32:57 +0100) > > are available in the Git repository at: > > https://gitlab.com/jsnow/qemu.git tags/python-pull-request > > for you to fetch changes up to faa40e297406445ab8814844bd9aa532a7538cde: > > gitlab: add python linters to CI (2021-05-27 18:11:17 -0400) > > > Python pull request > > Python packaging & CI implementation > Acceptance tests sent w/ Cleber's blessing. > New CI tests send w/ stsquad's RB. > > --js > > Fails to build on my machine that runs the BSD VMs, apparently before it gets to the point of launching the VM: make: Entering directory '/home/peter.maydell/qemu-freebsd/build' /usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/meson/meson.py introspect --targets --tests --benchmarks | /usr/bin/python3 -B scripts/mtest2make.py > Makefile.mtest { \ echo 'ninja-targets = \'; \ /usr/bin/ninja -t targets all | sed 's/:.*//; $!s/$/ \\/'; \ echo 'build-files = \'; \ /usr/bin/ninja -t query build.ninja | sed -n '1,/^ input:/d; /^ outputs:/q; s/$/ \\/p'; \ } > Makefile.ninja.tmp && mv Makefile.ninja.tmp Makefile.ninja (GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh" update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc capstone slirp) (GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh" update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc capstone slirp) /usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/tests/vm/freebsd --debug --genisoimage /usr/bin/genisoimage--image "/home/peter.maydell/.cache/qemu-vm/images/freebsd.img" --force --build-image /home/peter.maydell/.cache/qemu-vm/images/freebsd.img Traceback (most recent call last): File "/home/peter.maydell/qemu-freebsd/tests/vm/freebsd", line 21, in import basevm File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 22, in from qemu.machine import QEMUMachine ImportError: bad magic number in 'qemu': b'\x03\xf3\r\n' /home/peter.maydell/qemu-freebsd/tests/vm/Makefile.include:79: recipe for target '/home/peter.maydell/.cache/qemu-vm/images/freebsd.img' failed make: *** [/home/peter.maydell/.cache/qemu-vm/images/freebsd.img] Error 1 make: Leaving directory '/home/peter.maydell/qemu-freebsd/build' thanks -- PMM
[PULL 00/44] Python patches
The following changes since commit c8616fc7670b884de5f74d2767aade224c1c5c3a: Merge remote-tracking branch 'remotes/philmd/tags/gitlab-ci-20210527' into staging (2021-05-27 16:32:57 +0100) are available in the Git repository at: https://gitlab.com/jsnow/qemu.git tags/python-pull-request for you to fetch changes up to faa40e297406445ab8814844bd9aa532a7538cde: gitlab: add python linters to CI (2021-05-27 18:11:17 -0400) Python pull request Python packaging & CI implementation Acceptance tests sent w/ Cleber's blessing. New CI tests send w/ stsquad's RB. --js Cleber Rosa (12): Python: expose QEMUMachine's temporary directory tests/acceptance/virtiofs_submounts.py: add missing accel tag tests/acceptance/virtiofs_submounts.py: evaluate string not length Python: add utility function for retrieving port redirection Acceptance Tests: move useful ssh methods to base class Acceptance Tests: add port redirection for ssh by default Acceptance Tests: make username/password configurable Acceptance Tests: set up SSH connection by default after boot for LinuxTest tests/acceptance/virtiofs_submounts.py: remove launch_vm() Acceptance Tests: add basic documentation on LinuxTest base class Acceptance Tests: introduce CPU hotplug test tests/acceptance/virtiofs_submounts.py: fix setup of SSH pubkey John Snow (31): python/console_socket: avoid one-letter variable python/machine: use subprocess.DEVNULL instead of open(os.path.devnull) python/machine: use subprocess.run instead of subprocess.Popen python/console_socket: Add a pylint ignore python/machine: Disable pylint warning for open() in _pre_launch python/machine: disable warning for Popen in _launch() python/machine: Trim line length to below 80 chars iotests/297: add --namespace-packages to mypy arguments python: create qemu packages python: add qemu package installer python: add VERSION file python: add directory structure README.rst files python: add MANIFEST.in python: Add pipenv support python: add pylint import exceptions python: move pylintrc into setup.cfg python: add pylint to pipenv python: move flake8 config to setup.cfg python: add excluded dirs to flake8 config python: Add flake8 to pipenv python: move mypy.ini into setup.cfg python: add mypy to pipenv python: move .isort.cfg into setup.cfg python/qemu: add isort to pipenv python/qemu: add qemu package itself to pipenv python: add devel package requirements to setuptools python: add avocado-framework and tests python: add Makefile for some common tasks python: add .gitignore python: add tox support gitlab: add python linters to CI Willian Rampazzo (1): acceptance tests: bump Avocado version to 88.1 docs/devel/testing.rst | 26 +++ python/PACKAGE.rst | 43 python/README.rst | 58 + python/qemu/README.rst | 8 + python/qemu/machine/README.rst | 9 + python/qemu/qmp/README.rst | 9 + python/qemu/utils/README.rst| 7 + .gitlab-ci.d/containers.yml | 5 + .gitlab-ci.d/static_checks.yml | 21 ++ python/.gitignore | 16 ++ python/MANIFEST.in | 3 + python/Makefile | 48 python/Pipfile | 13 ++ python/Pipfile.lock | 231 python/VERSION | 1 + python/avocado.cfg | 10 + python/mypy.ini | 4 - python/qemu/.flake8 | 2 - python/qemu/.isort.cfg | 7 - python/qemu/__init__.py | 11 - python/qemu/machine/__init__.py | 36 +++ python/qemu/{ => machine}/console_socket.py | 11 +- python/qemu/{ => machine}/machine.py| 68 -- python/qemu/{ => machine}/qtest.py | 9 +- python/qemu/pylintrc| 58 - python/qemu/{qmp.py => qmp/__init__.py} | 12 +- python/qemu/utils/__init__.py | 45 python/qemu/{ => utils}/accel.py| 0 python/setup.cfg| 102 + python/setup.py | 23 ++ python/tests/flake8.sh | 2 + python/tests/isort.sh | 2 + python/tests/mypy.sh| 2 + python/tests/pylint.sh | 2 + tests/acceptance/avocado_qemu/__init__.py | 69 +- tests/acceptance/boot_linux.py | 18 +- tests/acceptance/hotplug_cpu.py | 37 tests/acceptance/info_usernet.py| 29 +++