Re: [PATCH v1 08/14] tests/vm: Added configuration file support

2020-02-14 Thread Robert Foley
On Fri, 14 Feb 2020 at 11:54, Alex Bennée  wrote:
> >  from socket_thread import SocketThread
> > +import yaml
>
> So this throws my setup on my Gentoo SynQuacer. Is this meant to be in
> the standard library or is this a separate dependency?
>
This is a separate dependency.  On Ubuntu the package is python3-yaml.
This brings up an interesting point.
If the yaml dependency is not there, should we expect to gracefully
handle this in the python script, and
simply not allow yaml files?  I suppose we could error out if a yaml
file is supplied
and ask for the dependency to get installed.
Or is there something we should do earlier, maybe in the configure, to
warn or error about the missing
dependency?

Thanks & Regards,
-Rob


> --
> Alex Bennée



Re: [PATCH v1 08/14] tests/vm: Added configuration file support

2020-02-14 Thread Alex Bennée


Robert Foley  writes:

> Changes to tests/vm/basevm.py to allow accepting a configuration file
> as a parameter. Allows for specifying VM options such as
> cpu, machine, memory, and arbitrary qemu arguments for specifying options
> such as NUMA configuration.
> Also added an example conf_example_aarch64.yml and conf_example_x86.yml.
>
> Signed-off-by: Robert Foley 
> ---
>  tests/vm/Makefile.include |  2 ++
>  tests/vm/basevm.py| 29 +-
>  tests/vm/conf_example_aarch64.yml | 51 +++
>  tests/vm/conf_example_x86.yml | 50 ++
>  4 files changed, 131 insertions(+), 1 deletion(-)
>  create mode 100644 tests/vm/conf_example_aarch64.yml
>  create mode 100644 tests/vm/conf_example_x86.yml
>
> diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
> index 778e506755..e9ed33226d 100644
> --- a/tests/vm/Makefile.include
> +++ b/tests/vm/Makefile.include
> @@ -35,6 +35,8 @@ vm-help vm-test:
>   @echo "V=1   - Enable verbose ouput on host 
> and guest commands"
>   @echo "QEMU=/path/to/qemu- Change path to QEMU binary"
>   @echo "QEMU_IMG=/path/to/qemu-img- Change path to qemu-img tool"
> + @echo "QEMU_CONFIG=/path/conf.yml   - Change path to VM 
> configuration .yml file."
> + @echo "   See conf_example_*.yml for 
> file format details."
>  
>  vm-build-all: $(addprefix vm-build-, $(IMAGES))
>  
> diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
> index 33004934af..f488d4103c 100755
> --- a/tests/vm/basevm.py
> +++ b/tests/vm/basevm.py
> @@ -32,6 +32,7 @@ import shutil
>  import multiprocessing
>  import traceback
>  from socket_thread import SocketThread
> +import yaml

So this throws my setup on my Gentoo SynQuacer. Is this meant to be in
the standard library or is this a separate dependency?

-- 
Alex Bennée



[PATCH v1 08/14] tests/vm: Added configuration file support

2020-02-05 Thread Robert Foley
Changes to tests/vm/basevm.py to allow accepting a configuration file
as a parameter. Allows for specifying VM options such as
cpu, machine, memory, and arbitrary qemu arguments for specifying options
such as NUMA configuration.
Also added an example conf_example_aarch64.yml and conf_example_x86.yml.

Signed-off-by: Robert Foley 
---
 tests/vm/Makefile.include |  2 ++
 tests/vm/basevm.py| 29 +-
 tests/vm/conf_example_aarch64.yml | 51 +++
 tests/vm/conf_example_x86.yml | 50 ++
 4 files changed, 131 insertions(+), 1 deletion(-)
 create mode 100644 tests/vm/conf_example_aarch64.yml
 create mode 100644 tests/vm/conf_example_x86.yml

diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index 778e506755..e9ed33226d 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -35,6 +35,8 @@ vm-help vm-test:
@echo "V=1   - Enable verbose ouput on host 
and guest commands"
@echo "QEMU=/path/to/qemu- Change path to QEMU binary"
@echo "QEMU_IMG=/path/to/qemu-img- Change path to qemu-img tool"
+   @echo "QEMU_CONFIG=/path/conf.yml   - Change path to VM 
configuration .yml file."
+   @echo "   See conf_example_*.yml for 
file format details."
 
 vm-build-all: $(addprefix vm-build-, $(IMAGES))
 
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 33004934af..f488d4103c 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -32,6 +32,7 @@ import shutil
 import multiprocessing
 import traceback
 from socket_thread import SocketThread
+import yaml
 
 SSH_KEY_FILE = os.path.join(os.path.dirname(__file__),
"..", "keys", "id_rsa")
@@ -500,9 +501,31 @@ class BaseVM(object):
cwd=cidir,
stdin=self._devnull, stdout=self._stdout,
stderr=self._stdout)
-
 return os.path.join(cidir, "cloud-init.iso")
 
+def parse_config(config, args):
+""" Parse yaml config and populate our config structure.
+The yaml config allows the user to override the
+defaults for VM parameters.  In many cases these
+defaults can be overridden without rebuilding the VM."""
+if args.config:
+config_file = args.config
+elif 'QEMU_CONFIG' in os.environ:
+config_file = os.environ['QEMU_CONFIG']
+else:
+return config
+if not os.path.exists(config_file):
+raise Exception("config file {} does not exist".format(config_file))
+with open(config_file) as f:
+yaml_dict = yaml.safe_load(f)
+
+if 'qemu-conf' in yaml_dict:
+config.update(yaml_dict['qemu-conf'])
+else:
+raise Exception("config file {} is not valid"\
+" missing qemu-conf".format(config_file))
+return config
+
 def parse_args(vmcls):
 
 def get_default_jobs():
@@ -537,6 +560,9 @@ def parse_args(vmcls):
   help="Interactively run command")
 parser.add_option("--snapshot", "-s", action="store_true",
   help="run tests with a snapshot")
+parser.add_option("--config", "-c", default=None,
+  help="Provide config yaml for configuration. "\
+   "See config_example.yaml for example.")
 parser.disable_interspersed_args()
 return parser.parse_args()
 
@@ -548,6 +574,7 @@ def main(vmcls, config=None):
 if not argv and not args.build_qemu and not args.build_image:
 print("Nothing to do?")
 return 1
+config = parse_config(config, args)
 logging.basicConfig(level=(logging.DEBUG if args.debug
else logging.WARN))
 vm = vmcls(debug=args.debug, vcpus=args.jobs, config=config)
diff --git a/tests/vm/conf_example_aarch64.yml 
b/tests/vm/conf_example_aarch64.yml
new file mode 100644
index 00..9d44ae356f
--- /dev/null
+++ b/tests/vm/conf_example_aarch64.yml
@@ -0,0 +1,51 @@
+#
+# Example yaml for use by any of the scripts in tests/vm.
+# Can be provided as an environment variable QEMU_CONFIG
+#
+qemu-conf:
+
+# If any of the below are not provided, we will just use the qemu defaults.
+
+# Login username and password(has to be sudo enabled)
+guest_user: qemu
+guest_pass: "qemupass"
+
+# Password for root user can be different from guest.
+root_pass: "qemupass"
+
+# If one key is provided, both must be provided.
+#ssh_key: /complete/path/of/your/keyfile/id_rsa
+#ssh_pub_key: /complete/path/of/your/keyfile/id_rsa.pub
+
+cpu: max
+machine: virt,gic-version=max
+memory: 16G
+
+# The below is a example for how to configure NUMA topology with
+# 4 NUMA nodes and 2 different NUMA distances.
+qemu_args: "-smp cpus=16,sockets=2,cores=8
+-numa