Re: [Engine-devel] Polling vs Pushing engine events
Hi, the following seems to be killing our plans to use a pythonscript to fetch the list of vms: This comes from the users list: Am 17.12.2013 15:04, schrieb Sander Grendelman: Fetching the list of vms through the API with the python SDK takes several seconds[1] with 100% cpu usage in the python script. When I look at the engine log there is only one (fast) fetch from the API. The rest of the time is spent in the SDK processing a relatively small bit of XML data (only 26 VMs in my environment). This seems an excessive amount of CPU for processing ~6KB of XML. I've included some sample code [2] and output [3]. Attached is the cProfile output for this call and a visualization. [1] ~6,5 seconds on an oVirt VM with 1 vcpu on older hardware, ~3,5 seconds on a fast physical machine with an i5 cpu. Am 17.12.2013 15:31, schrieb Michael Pasternak: Hi Sander, This is a known issue caused by generateDS python bindings we use, it's extremely slow in python-xml marshalling, and unable to recognize cyclic referencing in the objects, i'm planning to upgrade in 3.4 from 2.9a to 2.12, if it won't help, we may consider other options. This is no option for us at all: We can't wait for a possible fix in 3.4 (which maybe even does not fix it). Additionally we will have much higher amounts of vms to query. -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Polling vs Pushing engine events
Note that you can usually get all the information you want using 1 API call, which should still scale. For instance, /ovirt-engine/api/vms will give you a list of all VMs and their statuses, so you can just run an XPath and get the status of all of them. On Tue, Dec 17, 2013 at 10:18 AM, Itamar Heim ih...@redhat.com wrote: On 12/17/2013 03:08 AM, Sven Kieske wrote: Hi, we got the following problem: we create / start / stop hole vms /data centers / storage etc (basically: everything ovirt can handle via REST-API) But if you want to know e.g. the status of a vm (or anything) you need to constantly poll the API. This is not what we desire to do, as it does not scale very well (e.g. polling 100 vms). well, you can search events since last event you searched for, only for the specific type of event you are interested in, then check which vm its for. Is there a standardized way of pushing information from the engine? well, the notification service which sends emails on these actually polls for them every minute in order to send the emails. we are discussing snmptraps here[1] one of the options this could be implemented with is via log4j getting all the audit log events, then you could use any log4j appender (db table, jms queue, etc.) [1] Bug 1032661 - Add SNMP trap as notification method to to ovirt-engine-notification ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Polling vs Pushing engine events
Hi, thanks for your fast replies. We are aware that we can poll this via API, but we don't want to poll it, as it's not a good solution, from a design point of view. We will maybe write a little python script, which runs on engine-node via SDK which polls the API and let this script push the information forward to external systems over network when a certain event occurs. Something similar to this in oVirt would be useful. You don't want to poll every second, or more often, for simple events like is the vm already up?. This just creates unnecessary network traffic and load on the polling system. With this solution we can at least avoid the network traffic. This is, why I asked for a pushing logic. Maybe it's now more clear? But thanks again for your help :-) Am 17.12.2013 09:24, schrieb Liran Zelkha: Note that you can usually get all the information you want using 1 API call, which should still scale. For instance, /ovirt-engine/api/vms will give you a list of all VMs and their statuses, so you can just run an XPath and get the status of all of them. -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Polling vs Pushing engine events
You can use the DWH API to check these things. The status is sampled and stored for most entities every 1 minute by default (and can be set to less than that). Yaniv - Original Message - From: Sven Kieske s.kie...@mittwald.de To: us...@ovirt.org, engine-devel@ovirt.org Sent: Tuesday, December 17, 2013 10:08:11 AM Subject: [Engine-devel] Polling vs Pushing engine events Hi, we got the following problem: we create / start / stop hole vms /data centers / storage etc (basically: everything ovirt can handle via REST-API) But if you want to know e.g. the status of a vm (or anything) you need to constantly poll the API. This is not what we desire to do, as it does not scale very well (e.g. polling 100 vms). Is there a standardized way of pushing information from the engine? -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
