Re: [Users] Python SDK list vms slow, excessive cpu usage
Hi, just curious, with the upcoming 3.4 release. was the binding updated? And did somebody test if the XML performance has gotten better with it? Am 17.12.2013 15:31, schrieb Michael Pasternak: This is a known issue caused by generateDS python bindings we use, it's extremely slow in python-xml marshalling, and unable to recognize cyclic referencing in the objects, i'm planning to upgrade in 3.4 from 2.9a to 2.12, if it won't help, we may consider other options. -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Python SDK list vms slow, excessive cpu usage
Hi, thanks to Sander for disclosing this issue. We were also looking into using python to query the vms. But this looks not very promising. We have the need to query for much more vms, and we can't wait for just the possibility of a fix in 3.4. Maybe this can be circumvented by using the REST-API? I didn't test it for performance yet, but it seems to be reasonable fast without high load. Am 17.12.2013 15:31, schrieb Michael Pasternak: Hi Sander, This is a known issue caused by generateDS python bindings we use, it's extremely slow in python-xml marshalling, and unable to recognize cyclic referencing in the objects, i'm planning to upgrade in 3.4 from 2.9a to 2.12, if it won't help, we may consider other options. -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] Python SDK list vms slow, excessive cpu usage
Hi Sander, This is a known issue caused by generateDS python bindings we use, it's extremely slow in python-xml marshalling, and unable to recognize cyclic referencing in the objects, i'm planning to upgrade in 3.4 from 2.9a to 2.12, if it won't help, we may consider other options. On 12/17/2013 04:04 PM, Sander Grendelman wrote: Fetching the list of vms through the API with the python SDK takes several seconds[1] with 100% cpu usage in the python script. When I look at the engine log there is only one (fast) fetch from the API. The rest of the time is spent in the SDK processing a relatively small bit of XML data (only 26 VMs in my environment). This seems an excessive amount of CPU for processing ~6KB of XML. I've included some sample code [2] and output [3]. Attached is the cProfile output for this call and a visualization. [1] ~6,5 seconds on an oVirt VM with 1 vcpu on older hardware, ~3,5 seconds on a fast physical machine with an i5 cpu. [2] Sample code, add your own url/credentials/certificate: #!/usr/bin/python Get ovirt VM names import time from ovirtsdk.api import API from ovirtsdk.xml import params def print_elapsed(logString): elapsed = time.time() - startTime print %fs %s % (elapsed, logString) startTime = time.time() print_elapsed(before connect) api = API( url=https://my.engine.url/api;, username=user@domain, password=password, ca_file=./ca.crt, ) print_elapsed(after connect) print_elapsed(before fetch list) vmList = api.vms.list() print_elapsed(after fetch list) print_elapsed(before use list) count = 0 for vm in vmList: vmName = vm.get_name() count += 1 print \t number of VMs %d % count [3] Example of output: [grendelmans@bungee src]$ /usr/bin/time ./vm_list_test.py 0.03s before connect 0.338248s after connect 0.338299s before fetch list 6.795967s after fetch list 6.796041s before use list number of VMs 26 6.796181s after use list 7.28user 0.05system 0:07.64elapsed 96%CPU (0avgtext+0avgdata 84624maxresident)k 0inputs+0outputs (0major+5605minor)pagefaults 0swaps [grendelmans@bungee src]$ ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- Michael Pasternak RedHat, ENG-Virtualization RD ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
