Re: [PULL 00/44] Python patches

2021-06-01 Thread John Snow

On 6/1/21 6:36 AM, Peter Maydell wrote:

On Sun, 30 May 2021 at 20:22, John Snow  wrote:


On 5/30/21 3:09 PM, Peter Maydell wrote:

Fails to build on my machine that runs the BSD VMs, apparently
before it gets to the point of launching the VM:



When I have seen this error message in the past, it has been because of
using a new Python version and it gets confused reading stale cached
information generated from an older interpreter.


This seems like a bug in Python...



Yes. I don't know why it's like that either...


Can you do me a favor and delete any __pycache__ folders and/or any
*.pyc files that might be hiding in your tree and/or build folders and
try running it again?


I tried that, and it does seem to make more progress. It now
fails with:



One of the acceptance test patches causes this, I've squashed in a fixup 
and have re-sent the PR.


Thanks,
--js




Re: [PULL 00/44] Python patches

2021-06-01 Thread Peter Maydell
On Sun, 30 May 2021 at 20:22, John Snow  wrote:
>
> On 5/30/21 3:09 PM, Peter Maydell wrote:
> > Fails to build on my machine that runs the BSD VMs, apparently
> > before it gets to the point of launching the VM:

> When I have seen this error message in the past, it has been because of
> using a new Python version and it gets confused reading stale cached
> information generated from an older interpreter.

This seems like a bug in Python...

> Can you do me a favor and delete any __pycache__ folders and/or any
> *.pyc files that might be hiding in your tree and/or build folders and
> try running it again?

I tried that, and it does seem to make more progress. It now
fails with:

### Booting installer ...
DEBUG:root:QEMU args: -nodefaults -m 4G -cpu max,sse4.2=off -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device
virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 18 -enable-kvm
-drive 
file=/home/peter.maydell/.cache/qemu-vm/images/freebsd.img.tmp,if=none,id=drive0,cache=writeback
-device virtio-blk,drive=drive0,bootindex=0 -bios
pc-bios/bios-256k.bin -machine graphics=off -device VGA -cdrom
/home/peter.maydell/.cache/qemu-vm/images/freebsd.img.install.iso
DEBUG:qemu.machine.machine:VM launch command: 'qemu-system-x86_64
-display none -vga none -chardev
socket,id=mon,path=/var/tmp/qemu-13084-monitor.sock -mon
chardev=mon,mode=control -machine pc -chardev
socket,id=console,path=/var/tmp/qemu-13084-console.sock,server=on,wait=off
-serial chardev:console -nodefaults -m 4G -cpu max,sse4.2=off -netdev
user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device
virtio-net-pci,netdev=vnet -vnc 127.0.0.1:0,to=20 -smp 18 -enable-kvm
-drive 
file=/home/peter.maydell/.cache/qemu-vm/images/freebsd.img.tmp,if=none,id=drive0,cache=writeback
-device virtio-blk,drive=drive0,bootindex=0 -bios
pc-bios/bios-256k.bin -machine graphics=off -device VGA -cdrom
/home/peter.maydell/.cache/qemu-vm/images/freebsd.img.install.iso'
DEBUG:QMP.qemu-13084:>>> {'execute': 'qmp_capabilities'}
DEBUG:QMP.qemu-13084:<<< {'return': {}}
DEBUG:QMP.qemu-13084:>>> {'execute': 'human-monitor-command',
'arguments': {'command-line': 'info usernet'}}
DEBUG:QMP.qemu-13084:<<< {'return': 'Hub -1 (vnet):\r\n
Protocol[State]FD  Source Address  Port   Dest. Address  Port
RecvQ SendQ\r\n  TCP[HOST_FORWARD]  15   127.0.0.1 38577
10.0.2.1522 0 0\r\n'}
Failed to prepare guest environment
Traceback (most recent call last):
  File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 630, in main
return vm.build_image(args.image)
  File "/home/peter.maydell/qemu-freebsd/tests/vm/freebsd", line 103,
in build_image
"-cdrom", iso
  File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 309, in boot
self.ssh_port = get_info_usernet_hostfwd_port(usernet_info)
  File 
"/home/peter.maydell/qemu-freebsd/tests/vm/../../python/qemu/utils/__init__.py",
line 40, in get_info_usernet_hostfwd_port
for line in info_usernet_output.split('\r\n'):
AttributeError: 'dict' object has no attribute 'split'
DEBUG:QMP.qemu-13084:>>> {'execute': 'quit'}
DEBUG:QMP.qemu-13084:<<< {'return': {}}
/home/peter.maydell/qemu-freebsd/tests/vm/Makefile.include:79: recipe
for target '/home/peter.maydell/.cache/qemu-vm/images/freebsd.img'
failed
make: *** [/home/peter.maydell/.cache/qemu-vm/images/freebsd.img] Error 2
make: Leaving directory '/home/peter.maydell/qemu-freebsd/build'

thanks
-- PMM



Re: [PULL 00/44] Python patches

2021-05-31 Thread Philippe Mathieu-Daudé
On 5/30/21 9:22 PM, John Snow wrote:
> On 5/30/21 3:09 PM, Peter Maydell wrote:
>> Fails to build on my machine that runs the BSD VMs, apparently
>> before it gets to the point of launching the VM:
>>
>> make: Entering directory '/home/peter.maydell/qemu-freebsd/build'
>> /usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/meson/meson.py
>> introspect --targets --tests --benchmarks | /usr/bin/python3 -B
>> scripts/mtest2make.py > Makefile.mtest
>> { \
>>    echo 'ninja-targets = \'; \
>>    /usr/bin/ninja -t targets all | sed 's/:.*//; $!s/$/ \\/'; \
>>    echo 'build-files = \'; \
>>    /usr/bin/ninja -t query build.ninja | sed -n '1,/^  input:/d; /^
>> outputs:/q; s/$/ \\/p'; \
>> } > Makefile.ninja.tmp && mv Makefile.ninja.tmp Makefile.ninja
>> (GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh"
>> update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
>> tests/fp/berkeley-softfloat-3 dtc capstone slirp)
>> (GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh"
>> update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
>> tests/fp/berkeley-softfloat-3 dtc capstone slirp)
>> /usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/tests/vm/freebsd
>> --debug --genisoimage /usr/bin/genisoimage    --image
>> "/home/peter.maydell/.cache/qemu-vm/images/freebsd.img" --force
>> --build-image /home/peter.maydell/.cache/qemu-vm/images/freebsd.img
>> Traceback (most recent call last):
>>    File "/home/peter.maydell/qemu-freebsd/tests/vm/freebsd", line 21,
>> in 
>>  import basevm
>>    File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 22,
>> in 
>>  from qemu.machine import QEMUMachine
>> ImportError: bad magic number in 'qemu': b'\x03\xf3\r\n'
>> /home/peter.maydell/qemu-freebsd/tests/vm/Makefile.include:79: recipe
>> for target '/home/peter.maydell/.cache/qemu-vm/images/freebsd.img'
>> failed
>> make: *** [/home/peter.maydell/.cache/qemu-vm/images/freebsd.img] Error 1
>> make: Leaving directory '/home/peter.maydell/qemu-freebsd/build'

> In case that doesn't fix it, can you tell me what version of Python your
> BSD setup uses? (And what version of FreeBSD you use?) -- I'll try to
> set up a VM and see if I can reproduce the problem.

IIUC correctly this is on the host (likely Linux); no BSD VM could be
launched yet.




Re: [PULL 00/44] Python patches

2021-05-30 Thread John Snow

On 5/30/21 3:09 PM, Peter Maydell wrote:

On Fri, 28 May 2021 at 03:32, John Snow  wrote:


The following changes since commit c8616fc7670b884de5f74d2767aade224c1c5c3a:

   Merge remote-tracking branch 'remotes/philmd/tags/gitlab-ci-20210527' into 
staging (2021-05-27 16:32:57 +0100)

are available in the Git repository at:

   https://gitlab.com/jsnow/qemu.git tags/python-pull-request

for you to fetch changes up to faa40e297406445ab8814844bd9aa532a7538cde:

   gitlab: add python linters to CI (2021-05-27 18:11:17 -0400)


Python pull request

Python packaging & CI implementation
Acceptance tests sent w/ Cleber's blessing.
New CI tests send w/ stsquad's RB.

--js




Fails to build on my machine that runs the BSD VMs, apparently
before it gets to the point of launching the VM:

make: Entering directory '/home/peter.maydell/qemu-freebsd/build'
/usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/meson/meson.py
introspect --targets --tests --benchmarks | /usr/bin/python3 -B
scripts/mtest2make.py > Makefile.mtest
{ \
   echo 'ninja-targets = \'; \
   /usr/bin/ninja -t targets all | sed 's/:.*//; $!s/$/ \\/'; \
   echo 'build-files = \'; \
   /usr/bin/ninja -t query build.ninja | sed -n '1,/^  input:/d; /^
outputs:/q; s/$/ \\/p'; \
} > Makefile.ninja.tmp && mv Makefile.ninja.tmp Makefile.ninja
(GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh"
update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 dtc capstone slirp)
(GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh"
update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 dtc capstone slirp)
/usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/tests/vm/freebsd
--debug --genisoimage /usr/bin/genisoimage--image
"/home/peter.maydell/.cache/qemu-vm/images/freebsd.img" --force
--build-image /home/peter.maydell/.cache/qemu-vm/images/freebsd.img
Traceback (most recent call last):
   File "/home/peter.maydell/qemu-freebsd/tests/vm/freebsd", line 21, in 

 import basevm
   File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 22,
in 
 from qemu.machine import QEMUMachine
ImportError: bad magic number in 'qemu': b'\x03\xf3\r\n'
/home/peter.maydell/qemu-freebsd/tests/vm/Makefile.include:79: recipe
for target '/home/peter.maydell/.cache/qemu-vm/images/freebsd.img'
failed
make: *** [/home/peter.maydell/.cache/qemu-vm/images/freebsd.img] Error 1
make: Leaving directory '/home/peter.maydell/qemu-freebsd/build'

thanks
-- PMM



Hmm ...

When I have seen this error message in the past, it has been because of 
using a new Python version and it gets confused reading stale cached 
information generated from an older interpreter.


None of these patches *should* have affected what version of python is 
used during the build, only the locations of some of the files. I don't 
immediately know why this PR would affect the QEMU build at all, actually.


Can you do me a favor and delete any __pycache__ folders and/or any 
*.pyc files that might be hiding in your tree and/or build folders and 
try running it again?


(If it helps, I can see about amending 'make clean' to try and clean up 
stale Python cache, which is the best solution I can think of for right 
now.)


In case that doesn't fix it, can you tell me what version of Python your 
BSD setup uses? (And what version of FreeBSD you use?) -- I'll try to 
set up a VM and see if I can reproduce the problem.


--js




Re: [PULL 00/44] Python patches

2021-05-30 Thread Peter Maydell
On Fri, 28 May 2021 at 03:32, John Snow  wrote:
>
> The following changes since commit c8616fc7670b884de5f74d2767aade224c1c5c3a:
>
>   Merge remote-tracking branch 'remotes/philmd/tags/gitlab-ci-20210527' into 
> staging (2021-05-27 16:32:57 +0100)
>
> are available in the Git repository at:
>
>   https://gitlab.com/jsnow/qemu.git tags/python-pull-request
>
> for you to fetch changes up to faa40e297406445ab8814844bd9aa532a7538cde:
>
>   gitlab: add python linters to CI (2021-05-27 18:11:17 -0400)
>
> 
> Python pull request
>
> Python packaging & CI implementation
> Acceptance tests sent w/ Cleber's blessing.
> New CI tests send w/ stsquad's RB.
>
> --js
>
> 

Fails to build on my machine that runs the BSD VMs, apparently
before it gets to the point of launching the VM:

make: Entering directory '/home/peter.maydell/qemu-freebsd/build'
/usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/meson/meson.py
introspect --targets --tests --benchmarks | /usr/bin/python3 -B
scripts/mtest2make.py > Makefile.mtest
{ \
  echo 'ninja-targets = \'; \
  /usr/bin/ninja -t targets all | sed 's/:.*//; $!s/$/ \\/'; \
  echo 'build-files = \'; \
  /usr/bin/ninja -t query build.ninja | sed -n '1,/^  input:/d; /^
outputs:/q; s/$/ \\/p'; \
} > Makefile.ninja.tmp && mv Makefile.ninja.tmp Makefile.ninja
(GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh"
update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 dtc capstone slirp)
(GIT="git" "/home/peter.maydell/qemu-freebsd/scripts/git-submodule.sh"
update ui/keycodemapdb meson tests/fp/berkeley-testfloat-3
tests/fp/berkeley-softfloat-3 dtc capstone slirp)
/usr/bin/python3 -B /home/peter.maydell/qemu-freebsd/tests/vm/freebsd
--debug --genisoimage /usr/bin/genisoimage--image
"/home/peter.maydell/.cache/qemu-vm/images/freebsd.img" --force
--build-image /home/peter.maydell/.cache/qemu-vm/images/freebsd.img
Traceback (most recent call last):
  File "/home/peter.maydell/qemu-freebsd/tests/vm/freebsd", line 21, in 
import basevm
  File "/home/peter.maydell/qemu-freebsd/tests/vm/basevm.py", line 22,
in 
from qemu.machine import QEMUMachine
ImportError: bad magic number in 'qemu': b'\x03\xf3\r\n'
/home/peter.maydell/qemu-freebsd/tests/vm/Makefile.include:79: recipe
for target '/home/peter.maydell/.cache/qemu-vm/images/freebsd.img'
failed
make: *** [/home/peter.maydell/.cache/qemu-vm/images/freebsd.img] Error 1
make: Leaving directory '/home/peter.maydell/qemu-freebsd/build'

thanks
-- PMM



[PULL 00/44] Python patches

2021-05-27 Thread John Snow
The following changes since commit c8616fc7670b884de5f74d2767aade224c1c5c3a:

  Merge remote-tracking branch 'remotes/philmd/tags/gitlab-ci-20210527' into 
staging (2021-05-27 16:32:57 +0100)

are available in the Git repository at:

  https://gitlab.com/jsnow/qemu.git tags/python-pull-request

for you to fetch changes up to faa40e297406445ab8814844bd9aa532a7538cde:

  gitlab: add python linters to CI (2021-05-27 18:11:17 -0400)


Python pull request

Python packaging & CI implementation
Acceptance tests sent w/ Cleber's blessing.
New CI tests send w/ stsquad's RB.

--js



Cleber Rosa (12):
  Python: expose QEMUMachine's temporary directory
  tests/acceptance/virtiofs_submounts.py: add missing accel tag
  tests/acceptance/virtiofs_submounts.py: evaluate string not length
  Python: add utility function for retrieving port redirection
  Acceptance Tests: move useful ssh methods to base class
  Acceptance Tests: add port redirection for ssh by default
  Acceptance Tests: make username/password configurable
  Acceptance Tests: set up SSH connection by default after boot for
LinuxTest
  tests/acceptance/virtiofs_submounts.py: remove launch_vm()
  Acceptance Tests: add basic documentation on LinuxTest base class
  Acceptance Tests: introduce CPU hotplug test
  tests/acceptance/virtiofs_submounts.py: fix setup of SSH pubkey

John Snow (31):
  python/console_socket: avoid one-letter variable
  python/machine: use subprocess.DEVNULL instead of
open(os.path.devnull)
  python/machine: use subprocess.run instead of subprocess.Popen
  python/console_socket: Add a pylint ignore
  python/machine: Disable pylint warning for open() in _pre_launch
  python/machine: disable warning for Popen in _launch()
  python/machine: Trim line length to below 80 chars
  iotests/297: add --namespace-packages to mypy arguments
  python: create qemu packages
  python: add qemu package installer
  python: add VERSION file
  python: add directory structure README.rst files
  python: add MANIFEST.in
  python: Add pipenv support
  python: add pylint import exceptions
  python: move pylintrc into setup.cfg
  python: add pylint to pipenv
  python: move flake8 config to setup.cfg
  python: add excluded dirs to flake8 config
  python: Add flake8 to pipenv
  python: move mypy.ini into setup.cfg
  python: add mypy to pipenv
  python: move .isort.cfg into setup.cfg
  python/qemu: add isort to pipenv
  python/qemu: add qemu package itself to pipenv
  python: add devel package requirements to setuptools
  python: add avocado-framework and tests
  python: add Makefile for some common tasks
  python: add .gitignore
  python: add tox support
  gitlab: add python linters to CI

Willian Rampazzo (1):
  acceptance tests: bump Avocado version to 88.1

 docs/devel/testing.rst  |  26 +++
 python/PACKAGE.rst  |  43 
 python/README.rst   |  58 +
 python/qemu/README.rst  |   8 +
 python/qemu/machine/README.rst  |   9 +
 python/qemu/qmp/README.rst  |   9 +
 python/qemu/utils/README.rst|   7 +
 .gitlab-ci.d/containers.yml |   5 +
 .gitlab-ci.d/static_checks.yml  |  21 ++
 python/.gitignore   |  16 ++
 python/MANIFEST.in  |   3 +
 python/Makefile |  48 
 python/Pipfile  |  13 ++
 python/Pipfile.lock | 231 
 python/VERSION  |   1 +
 python/avocado.cfg  |  10 +
 python/mypy.ini |   4 -
 python/qemu/.flake8 |   2 -
 python/qemu/.isort.cfg  |   7 -
 python/qemu/__init__.py |  11 -
 python/qemu/machine/__init__.py |  36 +++
 python/qemu/{ => machine}/console_socket.py |  11 +-
 python/qemu/{ => machine}/machine.py|  68 --
 python/qemu/{ => machine}/qtest.py  |   9 +-
 python/qemu/pylintrc|  58 -
 python/qemu/{qmp.py => qmp/__init__.py} |  12 +-
 python/qemu/utils/__init__.py   |  45 
 python/qemu/{ => utils}/accel.py|   0
 python/setup.cfg| 102 +
 python/setup.py |  23 ++
 python/tests/flake8.sh  |   2 +
 python/tests/isort.sh   |   2 +
 python/tests/mypy.sh|   2 +
 python/tests/pylint.sh  |   2 +
 tests/acceptance/avocado_qemu/__init__.py   |  69 +-
 tests/acceptance/boot_linux.py  |  18 +-
 tests/acceptance/hotplug_cpu.py |  37 
 tests/acceptance/info_usernet.py|  29 +++