Re: [ovirt-devel] [vdsm] Infrastructure design for node (host) devices
On Jun 29, 2014, at 16:55 , Saggi Mizrahi wrote: > > > - Original Message - >> From: "Martin Polednik" >> To: devel@ovirt.org >> Sent: Tuesday, June 24, 2014 1:26:17 PM >> Subject: [ovirt-devel] [vdsm] Infrastructure design for node (host) devices >> >> Hello, >> >> I'm actively working on getting host device passthrough (pci, usb and scsi) >> exposed in VDSM, but I've encountered growing complexity of this feature. >> >> The devices are currently created in the same manner as virtual devices and >> their reporting is done via hostDevices list in getCaps. As I implemented >> usb and scsi devices, the size of this list grew almost twice - and that is >> on a laptop. > There should be a separate verb with ability to filter by type. +1 >> >> Similar problem is with the devices themselves, they are closely tied to host >> and currently, engine would have to keep their mapping to VMs, reattach back >> loose devices and handle all of this in case of migration. > Migration sound very complicated, especially at the phase where the VM > actually > starts running on the target host. The hardware state is completely different > but the guest OS wouldn't have any idea that happened. > So detaching before migration and than reattaching on the destination is a > must > but that could cause issues in the guest. I'd imaging that this would be an > issue > when hibernating on one host and waking up on another. If qemu actually supports this at all it would need to be very specific for each device, restoring/setting a concrete HW state is a challenging task. I would also see it as pin to host and then on specific cases detach&attach (or that sr-iov's fancy temporary emulated device) >> >> I would like to hear your opinion on building something like host device pool >> in VDSM. The pool would be populated and periodically updated (to handle >> hot(un)plugs) and VMs/engine could query it for free/assigned/possibly >> problematic >> devices (which could be reattached by the pool). This has added benefit of >> requiring fewer libvirt calls, but a bit more complexity and possibly one >> thread. >> The persistence of the pool on VDSM restart could be kept in config or >> constructed >> from XML. > I'd much rather VDSM not cache state unless this is absolutely necessary. > This sounds like something that doesn't need to be queried every 3 seconds > so it's best if we just get to ask libvirt. well, unless we try to persist it a cache doesn't hurt I don't see a particular problem in reconstructing the structures on startup > > I do wonder how that kind of thing can be configured in the VM creation > phase as you would sometimes want to just specify a type of device and > sometimes specify a specific one. Also, I'd assume there will be a > fallback policy stating if the VM should run if said resource is unavailable. >> >> I'd need new API verbs to allow engine to communicate with the pool, >> possibly leaving caps as they are and engine could detect the presence of >> newer >> vdsm by presence of these API verbs. > Again, I think that getting a list of devices filterable by kind\type might > be best than a real pool. We might want to return if a device is in use > (could also be in use by the host operating system and not just VMs) >> The vmCreate call would remain almost >> the >> same, only with the addition of new device for VMs (where the detach and >> tracking >> routine would be communicated with the pool). >> ___ >> Devel mailing list >> Devel@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > ___ > Devel mailing list > Devel@ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [ovirt-users] [ANN] oVirt 3.5 Test Day - Tomorrow Jul 1st
(Sorry - fixed links inline) - Original Message - > From: "Yedidyah Bar David" > To: "users" , devel@ovirt.org > Sent: Monday, June 30, 2014 11:12:20 PM > Subject: [ovirt-users] [ANN] oVirt 3.5 Test Day - Tomorrow Jul 1st > > Hi all, > > Tomorrow Jul 1st we'll have oVirt 3.5.0 test day. > > On this day all relevant engineers will be online ready to support > any issues you find during install / operating this new release. > > Just make sure you have 1 host or more to test drive the new release. > If you're curious to see how it works, this is your chance. > > Thanks again for everyone who will join us tomorrow! > > Location > #ovirt irc channel > Please communicate here to allow others to see any issues > > What > In this test day you have a license to kill ;) > Follow the documentation to setup your environment, and test drive the > new features. > Please remember we expect to see some issues, and anything you come up > with will save you when you'll install final release > Remember to try daily tasks you'd usually do in the engine, to see there > are no regressions. > Write down the configuration you used (HW, console, etc) in the report > etherpad[1]. > > Documentation > Release notes: http://www.ovirt.org/OVirt_3.4.0_release_notes http://www.ovirt.org/OVirt_3.5_Release_Notes > Features pages links: http://bit.ly/17qBn6F http://bit.ly/1k7n6fv > If you find errors in the wiki please annotate it as well in report > etherpad [1] > > Prerequisites / recommendations > Use CentOS or RHEL 6.5 only. 6.4 is unsupported due to various issues > (sanlock, libvirt, etc). > Use Fedora 19 or 20. > > Latest RPMs > repository to be enabled for testing the release are listed in the > release notes page [2]. > > NEW issues / reports > For any new issue, please update the reports etherpad [1] > > Feature owners, please make sure: > your feature is updated and referenced on release page [2]. > you have testing instruction for your feature either on test day page [3] > or in your feature page. > your team regression testing section is organized and up to date on test > day page [3]. > > > [1] http://etherpad.ovirt.org/p/3.5-testday-1 > [2] http://www.ovirt.org/OVirt_3.5_Release_Notes > [3] http://www.ovirt.org/OVirt_3.5_TestDay > > > Thanks. > -- > Yedidyah Bar David, > RHEV Integration team > ___ > Users mailing list > us...@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > -- Didi ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] [ANN] oVirt 3.5 Test Day - Tomorrow Jul 1st
Hi all, Tomorrow Jul 1st we'll have oVirt 3.5.0 test day. On this day all relevant engineers will be online ready to support any issues you find during install / operating this new release. Just make sure you have 1 host or more to test drive the new release. If you're curious to see how it works, this is your chance. Thanks again for everyone who will join us tomorrow! Location #ovirt irc channel Please communicate here to allow others to see any issues What In this test day you have a license to kill ;) Follow the documentation to setup your environment, and test drive the new features. Please remember we expect to see some issues, and anything you come up with will save you when you'll install final release Remember to try daily tasks you'd usually do in the engine, to see there are no regressions. Write down the configuration you used (HW, console, etc) in the report etherpad[1]. Documentation Release notes: http://www.ovirt.org/OVirt_3.4.0_release_notes Features pages links: http://bit.ly/17qBn6F If you find errors in the wiki please annotate it as well in report etherpad [1] Prerequisites / recommendations Use CentOS or RHEL 6.5 only. 6.4 is unsupported due to various issues (sanlock, libvirt, etc). Use Fedora 19 or 20. Latest RPMs repository to be enabled for testing the release are listed in the release notes page [2]. NEW issues / reports For any new issue, please update the reports etherpad [1] Feature owners, please make sure: your feature is updated and referenced on release page [2]. you have testing instruction for your feature either on test day page [3] or in your feature page. your team regression testing section is organized and up to date on test day page [3]. [1] http://etherpad.ovirt.org/p/3.5-testday-1 [2] http://www.ovirt.org/OVirt_3.5_Release_Notes [3] http://www.ovirt.org/OVirt_3.5_TestDay Thanks. -- Yedidyah Bar David, RHEV Integration team ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [vdsm] Infrastructure design for node (host) devices
- Original Message - > From: "Francesco Romani" > To: devel@ovirt.org > Sent: Monday, June 30, 2014 7:16:20 PM > Subject: Re: [ovirt-devel] [vdsm] Infrastructure design for node (host) > devices > I can be convinced, anyway, if you can outline the design win of the pool > concept, and especially the pain points this approach is supposed to solve. I mean with specific and more focused examples and use cases (something like "we are going to save X KiB of traffic and/or Y libvirt calls" for example), because of course you already introduced the problem in general. My problem is just the terms are a bit too vague to really appreciate the possible gains; on the other hand, I'm quite concerned by the cons of the approach. -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [vdsm] Infrastructure design for node (host) devices
- Original Message - > From: "Martin Polednik" > To: devel@ovirt.org > Sent: Tuesday, June 24, 2014 12:26:17 PM > Subject: [ovirt-devel] [vdsm] Infrastructure design for node (host) devices > > Hello, Sorry for the late reply, > Similar problem is with the devices themselves, they are closely tied to host > and currently, engine would have to keep their mapping to VMs, reattach back > loose devices and handle all of this in case of migration. This is the simplest and the most consistent solution, so I'd start from here as first option. Your concern is about the sheer number of (host) devices and the added traffic to/from Engine or there is something else? I was also trying to guess which devices should we expect in the common case, and if it is safe to filter out by default in VDSM some devices classes; or, to flip the side, to have whitelists. This could possibly help us to migration work smoothly and without surprises like an host device available on host A but not into host B. For example, and just to my understanding, which one is the intended use case of an host SCSI device? This on top of any filter verbs (which someone else, probably Saggi, already mentioned, and I like the concept). > I would like to hear your opinion on building something like host device pool > in VDSM. The pool would be populated and periodically updated (to handle > hot(un)plugs) and VMs/engine could query it for free/assigned/possibly > problematic > devices (which could be reattached by the pool). This has added benefit of > requiring fewer libvirt calls, but a bit more complexity and possibly one > thread. > The persistence of the pool on VDSM restart could be kept in config or > constructed > from XML. The drawbacks of the added complexity seems to hedge the gains, and I'm concerned about migrations in particular, so I tend to dislike the pool concept. I can be convinced, anyway, if you can outline the design win of the pool concept, and especially the pain points this approach is supposed to solve. -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] oVirt.js GWT wrapper prototype
Hi Vojtech,
just one small remark to the oVirt.GWT wrapper.
Do you think it would be possible to replace
> Sdk.get().api().getDataCenters() with
> Sdk.api().getDataCenters() or even better yet with
> Sdk.getDataCenters()
In general I think that "Flat is better than nested" and I don't think
the collection names would introduce any collisions with the .services()
namespace so this should be a safe change.
Other than that I really like the current prototypes for both oVirt.js
and oVirt.GWT and I'm looking forward to future development.
Best regards
Martin
- Original Message -
> From: "Vojtech Szocs"
> To: devel@ovirt.org
> Sent: Tuesday, June 24, 2014 1:29:26 PM
> Subject: [ovirt-devel] oVirt.js GWT wrapper prototype
>
> Hello everyone,
>
> following the announcement of oVirt.js prototype, I've developed a sample
> GWT wrapper that provides Java API to oVirt.js for all GWT applications.
>
> Please find the GWT wrapper patch attached. It bundles oVirt.js & Lo-Dash
> libraries via GWT module (SdkGwtWrapper) providing Java-like API based on
> oVirt.js.
>
> In order for WebAdmin to use oVirt.js GWT wrapper, all we have to do is
> add following into WebAdmin.gwt.xml (GWT module descriptor):
>
>
>
> and add following into pom.xml (Maven project descriptor):
>
>
> ${engine.groupId}
> ovirt-js-gwt-wrapper
> ${engine.version}
> provided
>
>
> and that's it.
>
> The Java API takes inspiration from oVirt.js API. For example, to add
> new DataCenter:
>
> // Create data object template, 'name' and 'local' are required.
> DataCenterTemplate dcTemplate = DataCenterTemplate.create(
> "test-dc", // name
> false // local
> );
> // Set optional parameters such as 'description', if necessary.
> dcTemplate.setDescription("my-desc");
>
> // Obtain DataCenter resource collection.
> ResourceCollection dcColl = Sdk.get().api().getDataCenters();
>
> // Add new DataCenter by running 'add' operation on 'dcColl'.
> dcColl.add(dcTemplate).success(new SuccessCallback() {
> @Override
> public void onSuccess(DataCenter dc) {
> Window.alert("Added: " + dc.getName());
> }
> }).run();
>
> The concept of resource, resource collection and operation is the same
> as presented in oVirt.js.
>
> dc.setName("test-dc-updated");
> dc.setDescription("test")
> dc.update().run(); // we could register 'success' callback here
>
> You can see the full example by looking at SdkGwtWrapperTest class,
> located in WebAdmin codebase (org.ovirt.engine.ui.webadmin.sdk_test).
>
> Note that 'DataCenter' and 'DataCenterTemplate' will probably be
> auto-generated from oVirt Engine REST API definition (XSD/RSDL).
>
> As mentioned in my previous email, oVirt.js GWT wrapper ("overlay")
> can be initially part of ovirt-engine repo, while oVirt.js project
> deserves (in my opinion) a separate repo on its own.
>
> Regards,
> Vojtech
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] oVirt 3.5.0 Beta is now available for testing
The oVirt team is pleased to announce that the 3.5.0 Beta is now available for testing. Feel free to join us testing it! You'll find all needed info for installing it on the release notes page, already available on the wiki [1]. A new oVirt Live iso is already available for testing[2] including all available updates from CentOS. An oVirt Guest Tools iso is now available too[3]. A new oVirt Node build will be available soon as well. Please note that mirrors will need a couple of days before being synchronized. If you want to be sure to use latest rpms and don't want to wait for the mirrors, you can edit /etc/yum.repos.d/ovirt-3.5.repo commenting the mirror line and removing the comment on baseurl line. Known Issues * if you're using f19 host, you'll need libvirt >= 1.0.1 to use cluster level 3.5 (pkg will be provided from ovirt repo, but if it not there, you can update from f20 repo: : yum update --releasever=20 libvirt\* ) * You can't refresh iso file list after adding a host, see https://bugzilla.redhat.com/show_bug.cgi?id=1114499 for a workarond. [1] http://www.ovirt.org/OVirt_3.5_Release_Notes [2] http://resources.ovirt.org/pub/ovirt-3.5-pre/iso/ovirt-live-el6-3.5.0-alpha2.iso (not updated from last alpha yet) [3] http://resources.ovirt.org/pub/ovirt-3.5-pre/iso/ovirt-guest-tools/ovirt-guest-tools-3.5-1.iso (no change from last alpha) Eyal Edri oVirt infra Team. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] ovirt-3.5 test day: Package oVirt windows guest tools ISO and put it in ovirt repositories
Hi Lev, I am assigned to test the feature in $subject as part of the ovirt-3.5 test day. it says in [1] to "Talk to Lev" (attached). So I'm talking :) - in the BZ [2] description, it mentions that it would be nice to (a) have the windows guest tools ISO pre-populated in the ISO domain, but as a first step - (b) virtio-win should be RPM packaged and put it in ovirt (yum?) repositories. IIUC - neither (a) nor (b) were done - the ISO was simply put in resources.ovirt.org (not RPM packaged or anything like that) - correct? - which ISO should I use for the test-day? the one that appears in comment #5 in [2] as "master"? "3.5.0 pre release"? something else? - any particular windows versions that I should test (or, should NOT test)? - anything else that I should know about this feature? anything in particular that I should focus on? Regards, Einav [1] https://docs.google.com/spreadsheet/ccc?key=0AuAtmJW_VMCRdHJ6N1M3d1F1UTJTS1dSMnZwMF9XWVE&usp=drive_web&pli=1#gid=5 [2] Bug 1026930 - OVIRT35 - [RFE] Package oVirt windows guest tools ISO and put it in ovirt repositories [https://bugzilla.redhat.com/show_bug.cgi?id=1026930]___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] scale & hooks
Am 30.06.2014 14:48, schrieb Michal Skrivanek: > I think doing the scan only on startup is easier and may even make the code > simpler > (yes, ignoring the use case of adding a new hook on the fly, but I don't think it matters > much and in some cases I'd even prefer to control the deployment of my new fancy hook(s) and trigger them only on vdsmd restart) While I agree with this, keep in mind you are breaking the default behavior from old installs. -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] scale & hooks
On Jun 30, 2014, at 14:18 , Dan Kenigsberg wrote: > On Mon, Jun 30, 2014 at 01:04:09PM +0200, Michal Skrivanek wrote: >> Hi, >> just thinking about the recent stats hook[1] by Dima and scale, >> It is a very nice hook and quite useful for the scale testing - ( though >> when focus is on engine I'd suggest to use FakeVDSM instead to eliminate all >> the bottlenecks on VDSM side.) >> >> One issue I see - since this is a very high-profile call, every 15s time >> number of VMs - is with the hook invocation. Currently the hook mechanism >> always go through the hooks dit and does stat() call to see script is there >> or not… >> it certainly doesn't matter on small scale but when the system is on the >> edge every IO on local FS cost something…. > >> Would be nice if the hook mechanism is enhanced in general, figure out which >> hooks are relevant only at the startup and don't waste time passing >> domainXML every single time when the hook is not there > > But a caching mechanism for hook scripts costs "something", too (code > size, bugs, simplicty of testing new script). Before complicating the > mechanism, I'd rather see numbers: how much would be saved? I would > guess that filesystem caching already shaves costs of stat()s. Writing > domxml to a memory-map file before starting a vm may be insignificant, > too. I don't expect it's a big impact, but I'm worried about effects of filesystem access issues (say, very stressed environment with high IO), we used to have issues with vdsm persistent files; I think doing the scan only on startup is easier and may even make the code simpler (yes, ignoring the use case of adding a new hook on the fly, but I don't think it matters much and in some cases I'd even prefer to control the deployment of my new fancy hook(s) and trigger them only on vdsmd restart) high level comparison is always tricky. On one hand you won't see anything like "it makes a CPU usage drop by 1/2", however every single thing adds up; and as I said, it may only be ever relevant for things which are executed very often, like's the case of *_get_vm_stats. Of course to eliminate the number of calls from engine would be much much more important > > Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] scale & hooks
On Mon, Jun 30, 2014 at 01:04:09PM +0200, Michal Skrivanek wrote: > Hi, > just thinking about the recent stats hook[1] by Dima and scale, > It is a very nice hook and quite useful for the scale testing - ( though when > focus is on engine I'd suggest to use FakeVDSM instead to eliminate all the > bottlenecks on VDSM side.) > > One issue I see - since this is a very high-profile call, every 15s time > number of VMs - is with the hook invocation. Currently the hook mechanism > always go through the hooks dit and does stat() call to see script is there > or not… > it certainly doesn't matter on small scale but when the system is on the edge > every IO on local FS cost something…. > Would be nice if the hook mechanism is enhanced in general, figure out which > hooks are relevant only at the startup and don't waste time passing domainXML > every single time when the hook is not there But a caching mechanism for hook scripts costs "something", too (code size, bugs, simplicty of testing new script). Before complicating the mechanism, I'd rather see numbers: how much would be saved? I would guess that filesystem caching already shaves costs of stat()s. Writing domxml to a memory-map file before starting a vm may be insignificant, too. Dan. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] scale & hooks
- Original Message - > From: "Michal Skrivanek" > To: devel@ovirt.org > Sent: Monday, June 30, 2014 1:04:09 PM > Subject: [ovirt-devel] scale & hooks > Would be nice if the hook mechanism is enhanced in general, figure out which > hooks are relevant only at the startup and don't waste time passing > domainXML every single time when the hook is not there I'd like to tackle this (and hooks in general) Count me on! -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
[ovirt-devel] scale & hooks
Hi, just thinking about the recent stats hook[1] by Dima and scale, It is a very nice hook and quite useful for the scale testing - ( though when focus is on engine I'd suggest to use FakeVDSM instead to eliminate all the bottlenecks on VDSM side.) One issue I see - since this is a very high-profile call, every 15s time number of VMs - is with the hook invocation. Currently the hook mechanism always go through the hooks dit and does stat() call to see script is there or not… it certainly doesn't matter on small scale but when the system is on the edge every IO on local FS cost something…. Would be nice if the hook mechanism is enhanced in general, figure out which hooks are relevant only at the startup and don't waste time passing domainXML every single time when the hook is not there Thanks, michal [1] http://gerrit.ovirt.org/#/c/25927/ ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] Failed to run "make rpm" on VDSM latest master
On Sun, Jun 29, 2014 at 05:29:35AM -0400, Saggi Mizrahi wrote: > We have ioprocess in EPEL and brew builds for el6\7, and builds for f19\f20 > in bodhi. > > There is no such thing as "pushing to stable". In order to be deemed stable > a package has to sit there for 30 days without negative karma or get enough > (3) > positive karma. If I recall correctly, https://admin.fedoraproject.org/updates/search/ioprocess has a column titled "request" which the maintainer may click if he wants to push it to stable. A F20 ioprocess-0.5.0 build seems to be completely missing from there, btw. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] XML benchmarks
- Original Message - > From: "Francesco Romani" > To: devel@ovirt.org > Cc: "Nir Soffer" , "Martin Sivak" > Sent: Monday, June 30, 2014 12:14:34 PM > Subject: Re: [ovirt-devel] XML benchmarks > > - Original Message - > > From: "Francesco Romani" > > To: "Nir Soffer" > > Cc: devel@ovirt.org > > Sent: Monday, June 30, 2014 8:47:15 AM > > Subject: Re: [ovirt-devel] XML benchmarks > > > > - Original Message - > > > From: "Nir Soffer" > > > To: "Francesco Romani" > > > Cc: devel@ovirt.org, "Martin Sivak" > > > Sent: Sunday, June 29, 2014 10:34:08 AM > > > Subject: Re: [ovirt-devel] XML benchmarks > > > > > > CPU measurement: just opened a terminal and run 'htop' on it. > > > > CPU profile: clustered around the sampling interval. Usage negligible > > > > most > > > > of > > > > time, peak on sampling as shown below > > > > > > > > 300 VMs > > > > minidom: ~38% CPU > > > > cElementTree: ~5% CPU > > > > > > What is 38% - (38% of one core? how may cores are on the machine?) > > > > 4 cores: 2 physical, 2 logical. I'm prepping a more precise test > > using a better and less ambiguous indicator. > > Here. Attached un updated script (xmlbench2.py) which uses 'psutil' > (https://pypi.python.org/pypi/psutil) to gather the samples. > > CPU sampled each 500ms (half a second). 100% is one core. > My laptop reports 4 core (dualcore with hyperthreading). > > See attached some graphs for easier comsumption and their gnuplot recipe. > > cpu_300t_3m.png: load using the test script with 300 threads, each thread > runs ~3 minutes > cpu_500t_3m.png: load using the test script with 500 threads, each thread > runs ~3 minutes > > sampling is not really accurate but it is more than enough to get an idea. Nice! ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] XML benchmarks
- Original Message -
> From: "Francesco Romani"
> To: "Nir Soffer"
> Cc: devel@ovirt.org
> Sent: Monday, June 30, 2014 8:47:15 AM
> Subject: Re: [ovirt-devel] XML benchmarks
>
> - Original Message -
> > From: "Nir Soffer"
> > To: "Francesco Romani"
> > Cc: devel@ovirt.org, "Martin Sivak"
> > Sent: Sunday, June 29, 2014 10:34:08 AM
> > Subject: Re: [ovirt-devel] XML benchmarks
>
> > > CPU measurement: just opened a terminal and run 'htop' on it.
> > > CPU profile: clustered around the sampling interval. Usage negligible
> > > most
> > > of
> > > time, peak on sampling as shown below
> > >
> > > 300 VMs
> > > minidom: ~38% CPU
> > > cElementTree: ~5% CPU
> >
> > What is 38% - (38% of one core? how may cores are on the machine?)
>
> 4 cores: 2 physical, 2 logical. I'm prepping a more precise test
> using a better and less ambiguous indicator.
Here. Attached un updated script (xmlbench2.py) which uses 'psutil'
(https://pypi.python.org/pypi/psutil) to gather the samples.
CPU sampled each 500ms (half a second). 100% is one core.
My laptop reports 4 core (dualcore with hyperthreading).
See attached some graphs for easier comsumption and their gnuplot recipe.
cpu_300t_3m.png: load using the test script with 300 threads, each thread runs
~3 minutes
cpu_500t_3m.png: load using the test script with 500 threads, each thread runs
~3 minutes
sampling is not really accurate but it is more than enough to get an idea.
--
Francesco Romani
RedHat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani
#!/usr/bin/env python
import sys
import threading
import time
import xml.dom.minidom
import xml.etree.cElementTree
import xml.etree.ElementTree
import psutil
def eprint(s):
sys.stderr.write('%s\n' % s)
class Worker(threading.Thread):
def __init__(self, func, xml, delay, numruns):
super(Worker, self).__init__()
self.daemon = True
self.func = func
self.xml = xml
self.delay = delay
self.numruns = numruns
def mustgo(self):
if self.numruns is not None:
self.numruns -= 1
if self.numruns <= 0:
return False
return True
def run(self):
while self.mustgo():
time.sleep(self.delay)
self.func(self.xml)
PARSERS = {
'md': xml.dom.minidom.parseString,
'et': xml.etree.ElementTree.fromstring,
'cet': xml.etree.cElementTree.fromstring
}
def runner(xml, mode, nthreads, delay, numruns):
workers = []
for i in range(nthreads):
w = Worker(PARSERS[mode], xml, delay, numruns)
w.start()
workers.append(w)
p = psutil.Process()
p.cpu_percent() # see psutil docs. Discard the first one
samples = []
ts = 0.0
while any(w.is_alive() for w in workers):
time.sleep(0.5)
ts += 0.5
samples.append((ts, p.cpu_percent()))
return samples
def _usage():
eprint("usage: xmlbench xmlpath mode nthreads [delay [numruns]]")
eprint("available modes: %s" % ' '.join(PARSERS.keys()))
def _main(args):
if len(args) < 3:
_usage()
sys.exit(1)
else:
xmlpath = args[0]
mode = args[1]
nthreads = int(args[2])
delay = int(args[3]) if len(args) > 3 else 15
numruns = int(args[4]) if len(args) > 4 else None
if mode not in PARSERS:
_usage()
sys.exit(2)
with open(xmlpath, 'rt') as xml:
samples = runner(xml.read(), mode, nthreads, delay, numruns)
for (ts, value) in samples:
print '%f,%f' % (ts, value)
if __name__ == "__main__":
_main(sys.argv[1:])
plot.sh
Description: application/shellscript
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
