Re: [Qemu-block] [PULL 11/15] Bootstrap Python venv for tests

[Peter Maydell]

On 31 October 2018 at 00:31, Eduardo Habkost  wrote:
> From: Cleber Rosa 
>
> A number of QEMU tests are written in Python, and may benefit
> from an untainted Python venv.
>
> By using make rules, tests that depend on specific Python libs
> can set that rule as a requirement, along with rules that require
> the presence or installation of specific libraries.
>
> The tests/requirements.txt is supposed to contain the Python
> requirements that should be added to the venv created by check-venv.
>
> Signed-off-by: Cleber Rosa 
> Tested-by: Philippe Mathieu-Daudé 
> Acked-by: Stefan Hajnoczi 
> Acked-by: Wainer dos Santos Moschetta 
> Reviewed-by: Caio Carrara 
> Message-Id: <20181018153134.8493-2-cr...@redhat.com>
> Signed-off-by: Eduardo Habkost 

> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -12,6 +12,7 @@ check-help:
> @echo " $(MAKE) check-block  Run block tests"
> @echo " $(MAKE) check-tcgRun TCG tests"
> @echo " $(MAKE) check-report.htmlGenerates an HTML test report"
> +   @echo " $(MAKE) check-venv   Creates a Python venv for tests"
> @echo " $(MAKE) check-clean  Clean the tests"
> @echo
> @echo "Please note that HTML reports do not regenerate if the unit 
> tests"
> @@ -899,6 +900,30 @@ check-decodetree:
>./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>TEST, decodetree.py)
>
> +# Python venv for running tests
> +
> +.PHONY: check-venv
> +
> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> +TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
> +
> +$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' 
> >/dev/null 2>&1)
> +ifeq ($(.SHELLSTATUS),0)
> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> +   $(call quiet-command, \
> +$(PYTHON) -m venv --system-site-packages $@, \
> +VENV, $@)
> +   $(call quiet-command, \
> +$(TESTS_VENV_DIR)/bin/python -m pip -q install -r 
> $(TESTS_VENV_REQ), \
> +PIP, $(TESTS_VENV_REQ))
> +   $(call quiet-command, touch $@)
> +else
> +$(TESTS_VENV_DIR):
> +   $(error "venv directory for tests requires Python 3")
> +endif
> +
> +check-venv: $(TESTS_VENV_DIR)

Hi -- this seems to be causing one of the travis configs to fail:

https://travis-ci.org/qemu/qemu/jobs/451311466

The config includes "--python=/usr/bin/python3", but the build
fails with
 CHK version_gen.h
/home/travis/build/qemu/qemu/tests/Makefile.include:928: *** "venv
directory for tests requires Python 3". Stop.

Would you mind having a look at what's happening there ?

thanks
-- PMM

[Qemu-block] [PULL 11/15] Bootstrap Python venv for tests

[Eduardo Habkost]

From: Cleber Rosa 

A number of QEMU tests are written in Python, and may benefit
from an untainted Python venv.

By using make rules, tests that depend on specific Python libs
can set that rule as a requirement, along with rules that require
the presence or installation of specific libraries.

The tests/requirements.txt is supposed to contain the Python
requirements that should be added to the venv created by check-venv.

Signed-off-by: Cleber Rosa 
Tested-by: Philippe Mathieu-Daudé 
Acked-by: Stefan Hajnoczi 
Acked-by: Wainer dos Santos Moschetta 
Reviewed-by: Caio Carrara 
Message-Id: <20181018153134.8493-2-cr...@redhat.com>
Signed-off-by: Eduardo Habkost 
---
 tests/requirements.txt |  3 +++
 tests/Makefile.include | 26 ++
 2 files changed, 29 insertions(+)
 create mode 100644 tests/requirements.txt

diff --git a/tests/requirements.txt b/tests/requirements.txt
new file mode 100644
index 00..d39f9d1576
--- /dev/null
+++ b/tests/requirements.txt
@@ -0,0 +1,3 @@
+# Add Python module requirements, one per line, to be installed
+# in the tests/venv Python virtual environment. For more info,
+# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
diff --git a/tests/Makefile.include b/tests/Makefile.include
index f77a495109..eabc1da2f3 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -12,6 +12,7 @@ check-help:
@echo " $(MAKE) check-block  Run block tests"
@echo " $(MAKE) check-tcgRun TCG tests"
@echo " $(MAKE) check-report.htmlGenerates an HTML test report"
+   @echo " $(MAKE) check-venv   Creates a Python venv for tests"
@echo " $(MAKE) check-clean  Clean the tests"
@echo
@echo "Please note that HTML reports do not regenerate if the unit 
tests"
@@ -899,6 +900,30 @@ check-decodetree:
   ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
   TEST, decodetree.py)
 
+# Python venv for running tests
+
+.PHONY: check-venv
+
+TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
+TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
+
+$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 
2>&1)
+ifeq ($(.SHELLSTATUS),0)
+$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
+   $(call quiet-command, \
+$(PYTHON) -m venv --system-site-packages $@, \
+VENV, $@)
+   $(call quiet-command, \
+$(TESTS_VENV_DIR)/bin/python -m pip -q install -r 
$(TESTS_VENV_REQ), \
+PIP, $(TESTS_VENV_REQ))
+   $(call quiet-command, touch $@)
+else
+$(TESTS_VENV_DIR):
+   $(error "venv directory for tests requires Python 3")
+endif
+
+check-venv: $(TESTS_VENV_DIR)
+
 # Consolidated targets
 
 .PHONY: check-qapi-schema check-qtest check-unit check check-clean
@@ -912,6 +937,7 @@ check-clean:
rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), 
$(check-qtest-$(target)-y)) $(check-qtest-generic-y))
rm -f tests/test-qapi-gen-timestamp
+   rm -rf $(TESTS_VENV_DIR)
 
 clean: check-clean
 
-- 
2.18.0.rc1.1.g3f1ff2140