Re: [ovirt-devel] performance improvements and gwt-rpc switch

[Vojtech Szocs]

Thanks Greg for the excellent write-up!

oVirt WebAdmin UI is quite complex, with lots of abstractions (the GWT
technology being a Java-on-top-of-Web-APIs abstraction itself) as well
as many features (and lots of dialogs..) so this task wasn't easy, but
I guess we managed to get UI into a stabilized state now.

Dialogs (non-singleton by design) were a major source of memory leaks,
so we added UI infra to ensure automatic dialog cleanup.

In 4.2, we'd like to upgrade GWT SDK version to latest stable (2.8.x)
which will hopefully improve GWT compiler performance and, importantly,
allow us to use Java 8 language features in our GWT UI code.

There's still potential for improvement, we're tracking that through:

  [tracker] oVirt UI performance improvements
  https://bugzilla.redhat.com/1378935

(some IE-specific improvements are already submitted as part of ^^)

Regards,
Vojtech


- Original Message -
> From: "Greg Sheremeta" 
> To: "devel" 
> Sent: Monday, November 28, 2016 7:43:27 PM
> Subject: [ovirt-devel] performance improvements and gwt-rpc switch
> 
> Hi everyone,
> 
> For the upcoming oVirt 4.1, the UX team has been focused hard on webadmin
> performance improvements. There have been some reports [1] [2] of UI
> sluggishness in both 3.6 and 4.0, usually after the browser had been open
> some time, and usually in scale environments.
> 
> After some research, we determined that the primary cause of this
> sluggishness was memory leaks.
> 
> We embarked on several weeks of hunting down memory leak bugs and squashing
> them. Alexander Wels and Vojtech Szocs led this work, and I helped test the
> performance of each patch as they created them. As they created patches to
> squash leaks, performance kept getting better and better. Today we've merged
> the last of our patches [*], and I'm happy to announce that we are now
> seeing much better UI performance on 4.1-master and 4.0.6.
> 
> Over the course of several hours with the browser window open, users should
> see no sluggishness at all.
> 
> [*] This last patch switches oVirt from using de-rpc to gwt-rpc in the
> frontend. This improves performance, but it also allows us to upgrade to GWT
> 2.8. We'd been previously blocked on that.
> 
> If you're interested in UI performance testing, continue reading. If not, you
> can stop here :)
> 
> .
> 
> To verify our performance improvements, we took some simple measurements
> using selenium webdriver. The tests were unscientific, but very helpful. We
> ran a webdriver flow through oVirt that clicked some buttons and tabs and
> refreshed some grids. We did it a few hundred or thousand times. The tests
> were run using stubbed hosts (ovirt-vdsmfake) so that only the engine and UI
> were under test.
> 
> Below are the important takeaways. The x axis is time, and each point on a
> graph is a loop through the same webdriver flow. The (ms) y axes are
> response times, and memory is in MB.
> 
> In this graph, we compare oVirt 4.1 with and without our most impactful patch
> applied. As you can see, with the patch applied, response time stays flat
> for 200 loops of my test script over the course of 18 and 43 minutes.
> Without the patch applied, response time quickly degraded such that 200
> loops of my test script took 1 hr 2 minutes vs. 18 minutes with the patch
> applied -- a 66% improvement!
> 
> In this graph [ignore the spike], we tested oVirt hard for 6 hours 25 minutes
> (2000 loops). As you can see, the response times stay relatively flat over 6
> hours! This is a great improvement. Do note that the memory is still
> growing, albeit much more slowly now. You can see towards the end of this
> run, maybe around hour 5, that the deviation starts to go up (the line
> thickens). Takeaway: maybe refresh your browser after many hours of having
> webadmin open. But, this is a stress test -- I'm betting users won't notice
> this slowdown after even 6 hours of regular webadmin use or idling.
> 
> ​
> Last, here is a graph that shows gwt-rpc performing slightly better than
> de-rpc. Memory consumption is about the same -- gwt-rpc is just a faster rpc
> implementation.
> 
> ​
> Reply with any questions or concerns. Thanks!
> 
> Best wishes,
> Greg
> 
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1368101
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1388462
> 
> --
> Greg Sheremeta, MBA
> Red Hat, Inc.
> Sr. Software Engineer
> gsher...@redhat.com
> 
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Gerrit parallel patch handling and CI (Or, why did my code fail post-merge)

[Vojtech Szocs]

- Original Message -
> From: "Barak Korren" 
> To: "Vojtech Szocs" 
> Cc: "devel" 
> Sent: Wednesday, November 30, 2016 9:20:27 AM
> Subject: Re: [ovirt-devel] Gerrit parallel patch handling and CI (Or, why did 
> my code fail post-merge)
> 
> On 29 November 2016 at 19:34, Vojtech Szocs  wrote:
> >
> > Question: after the patch is submitted in Gerrit (it's fully acked
> > and maintainer hits the "Submit" button), does Gerrit allow us to
> > run CI (e.g. `check-merged` script) *before* doing the actual merge?
> >
> > [In other words, does Gerrit support gating merge based on CI script?]
> 
> No.
> See separate thread about merge-gateing:
> http://lists.ovirt.org/pipermail/devel/2016-November/014192.html

Thanks, I've missed that thread.

So before we adopt auto-submit-after-ci-pass (e.g. using Zuul, as Eyal
mentioned in that thread), we should be able to manually trigger heavy
CI (check-merged) job from Gerrit web interface, is that correct?

If so, it would be the patch owner's responsibility to submit (merge)
only after heavy CI (check-merged) pass?

As for Gerrit's "Submit" button hook, there's "pre-merge validation":

  
https://gerrit-review.googlesource.com/Documentation/config-validation.html#pre-merge-validation
  
https://github.com/gerrit-review/gerrit/blob/master/gerrit-server/src/main/java/com/google/gerrit/server/git/validators/MergeValidationListener.java

This validation API (MergeValidationListener) doesn't seem async as it
relies on exception (MergeValidationException) to block the operation.

So we could actually write Gerrit plugin using MergeValidationListener
that would operate in the following way:

1, on first "Submit" button click, tell Jenkins CI to run check-merged
   and block merge (e.g. providing reason like "wait till CI finishes")
2, Jenkins CI is expected to set some flag on patch once it finishes
3, on next "Submit" button click, block unless flag has "ok" value

> 
> > That said, +1 on "Fast Forward Only" submit type.
> 
> So far only VDSM stepped forward to try it out - and it proved to be
> too cumbersome for VDSM maintainers - so switched back to "Merge if
> Necessary".
> 
> --
> Barak Korren
> bkor...@redhat.com
> RHCE, RHCi, RHV-DevOps Team
> https://ifireball.wordpress.com/
> 
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] aarch64 / Raspberry Pi3

[Martin Polednik]

On 28/10/16 09:06 +0200, Sandro Bonazzola wrote:

Hi,
some time ago we've been asked to provide aarch64 build of qemu-kvm-ev for
CentOS Cloud SIG consumption.
We did it, and while at it we also built oVirt 4.0 VDSM dependencies for
aarch64 in CentOS Virt SIG.

Testing repositories have been created and are now publicly available:

[centos-qemu-ev-test]
name=CentOS-$releasever - QEMU EV Testing
baseurl=
http://buildlogs.centos.org/centos/$releasever/virt/$basearch/kvm-common/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Virtualization

[centos-ovirt40-test]
name=CentOS-$releasever - oVirt 4.0 Testing
baseurl=
http://buildlogs.centos.org/centos/$releasever/virt/$basearch/ovirt-4.0/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Virtualization

We don't officially support aarch64 and this is highly experimental so
expect bugs. But if you've a Raspberry Pi 3 and you decide to give VDSM a
run, please share your feedback.


I'd like to revive this thread as I've experimented with master VDSM
on rpi 3. First, there are close to no aarch64 builds for raspberry
except from images by Kraxel https://www.kraxel.org/repos/rpi2/images/.

Even with these images, it's doesn't seem to be easy to get KVM up and
running due to various HW quirks. Additionally, there is no network
bridge/tunnel support built into the kernel.

If you manage to bypass all this, VDSM depends on wide range of
packages such as python-blivet that don't work on rasperry's fedora 24
properly and require additional hacking (arch detection).

I'll probably blog about my adventure some time in the future, but
maybe those findings could be helpful.

mpolednik


Thanks,

--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com




___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel


___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] VDSM changes Linux memory dirty ratios - why?

[Ben England]

I uploaded the article to:

https://s3.amazonaws.com/ben.england/rhev-vm-rsptime.pdf

I had just gone to work for Red Hat back in 2011 so I made the mistake of 
including a company-internal URL in a public-facing document.We can update 
the bz with the above link if you want.  This article is ANCIENT (2011), so 
while the basic approach should still be relevant, some of the specifics may 
have changed.  This was a RHEV-on-NFS configuration.

As for why dirty ratio lowering is not part of a tuned profile, there is a 
debate in the performance team about dirty ratio, with some advocating for 
higher dirty ratios that help some application workloads (example: writing /tmp 
files).   I'd stand by this article for the particular configuration that was 
being studied though, and I think the basic approach of controlling latency and 
improving fairness by managing queue depths is very relevant today.

cc'ed Sanjay Rao, who continues to be a RHEV expert in the perf team (I no 
longer am working on it).

-ben


- Original Message -
> From: "Martin Polednik" 
> To: "Sven Kieske" 
> Cc: devel@ovirt.org, "Ben" 
> Sent: Wednesday, November 30, 2016 7:22:05 AM
> Subject: Re: VDSM changes Linux memory dirty ratios - why?
> 
> On 30/11/16 13:10 +0100, Sven Kieske wrote:
> >On 30/11/16 08:48, Martin Polednik wrote:
> >> It's not really irrelevant, the host still uses disk cache. Anyway,
> >> there is BZ[1] with a presentation[2] that (imho reasonably) states:
> >>
> >> "Reduce dirty page limits in KVM host to allow
> >> direct I/O writer VMs to compete successfully
> >> with buffered writer processes for storage
> >> access"
> >>
> >> I wonder why virtual-host tuned profile doesn't contain these values:
> >>
> >> $ grep vm.dirty /usr/lib/tuned/virtual-host/tuned.conf
> >> vm.dirty_background_ratio = 5
> >>
> >> [1]https://bugzilla.redhat.com/show_bug.cgi?id=740887
> >> [2]http://perf1.lab.bos.redhat.com/bengland/laptop/rhev/rhev-vm-rsptime.pdf
> >
> >Could you share [2] with the wider community? This would be awesome!
> 
> Sorry, I've totally missed the fact it's internal link (unfortunately
> publicly visible on the BZ). I believe it's slightly outdated, but
> let's ask the author.
> 
> Ben, is the document[2] somehow still valid and could it be made publicly
> available?
> 
> Re-referencing for completeness:
> [2]http://perf1.lab.bos.redhat.com/bengland/laptop/rhev/rhev-vm-rsptime.pdf
> 
> >--
> >Mit freundlichen Grüßen / Regards
> >
> >Sven Kieske
> >
> >Systemadministrator
> >Mittwald CM Service GmbH & Co. KG
> >Königsberger Straße 6
> >32339 Espelkamp
> >T: +495772 293100
> >F: +495772 29
> >https://www.mittwald.de
> >Geschäftsführer: Robert Meyer
> >St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
> >Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
> >
> 
> 
> 
> 
> >___
> >Devel mailing list
> >Devel@ovirt.org
> >http://lists.ovirt.org/mailman/listinfo/devel
> 
> 
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] python path conflicts issues while running python scripts from vdsm verbs

[Aravinda]

If vdsm setting PYTHONPATH as below

export PYTHONPATH=/usr/share/vdsm:$PYTHONPATH

change it as

export PYTHONPATH=$PYTHONPATH:/usr/share/vdsm

With this change, /usr/share/vdsm will not get first preference compared 
to /usr/lib*/python2.7/site-packages


regards
Aravinda

On 11/30/2016 10:25 AM, Ramesh Nachimuthu wrote:




- Original Message -

From: "Nir Soffer" 
To: "Ramesh Nachimuthu" 
Cc: "devel" , "Dan Kenigsberg" , "Piotr Kliczewski" 
, "Yaniv
Bronheim" , "Aravinda Vishwanathapura Krishna Murthy" 
, "Sahina Bose"

Sent: Tuesday, November 29, 2016 7:20:57 PM
Subject: Re: python path conflicts issues while running python scripts from 
vdsm verbs

On Tue, Nov 29, 2016 at 3:26 PM, Ramesh Nachimuthu 
wrote:

Hi,

  I am trying to run a python script '/sbin/gluster-eventsapi' in vdsm verb
  which internally imports some python modules from
  /usr/lib/python2.7/site-packages/gluster/cliutils/. But it fails with the
  import error. Following error is seen in the supervdsm log.


MainProcess|Thread-1::DEBUG::2016-11-28
16:54:35,130::commands::93::root::(execCmd) FAILED:  = 'Traceback
(most recent call last):\n  File "/sbin/gluster-eventsapi", line 25, in
\nfrom gluster.cliutils.cliutils import (Cmd, execute,
node_output_ok, node_output_notok,\nImportError: No module named
cliutils.cliutils\n';  = 1


I think the import statement "from gluster.cliutils.cliutils import (Cmd,
execute, node_output_ok, node_output_notok)" in the python script resolves
to '/usr/share/vdsm/gluster' instead of
/usr/lib/python2.7/site-packages/gluster/cliutils.

I see the following in python sys.path while executing a python script from
vdsm.

['/usr/libexec/glusterfs', '/usr/share/vdsm', '/usr/lib64/python27.zip',
'/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2',
'/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old',
'/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages',
'/usr/lib64/python2.7/site-packages/gtk-2.0',
'/usr/lib/python2.7/site-packages']

Looks like '/usr/share/vdsm' take precedence over
'/usr/lib64/python2.7/site-packages'.

Can someone suggests a way to fix this issue?

Note: '/sbin/gluster-eventsapi' works perfectly while running directly from
CLI.

Related vdsm patch:
https://gerrit.ovirt.org/#/c/67168/2/vdsm/gluster/events.py

If an import fails because /usr/share/vdsm/foo hides a package in
/usr/lib/python2.7/site-packages/vdsm it means that you have a wrong
import, you need to import anything like this:

 from vdsm.foo import bar

So names in /usr/share/vdsm/* cannot hide names in vdsm package.


Here its a different case. Some external python script (which I am trying to 
run from vdsm verb) is trying to import some gluster package from 
/usr/lib/python2.7/site-packages/gluster. But it is conflicting with gluster 
modules under /usr/share/vdsm/gluster

Python Script which I am trying to run: 
https://github.com/gluster/glusterfs/blob/master/events/src/peer_eventsapi.py

Regards,
Ramesh


Nir



___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] VDSM changes Linux memory dirty ratios - why?

[Yaniv Kaul]

On Wed, Nov 30, 2016 at 2:11 PM, Sven Kieske  wrote:

> On 30/11/16 09:13, Yaniv Kaul wrote:
> > Thanks, but it really makes no sense to me. The direct IO by the VMs is
> > going to a different storage than what the host is writing to, in most
> > cases.
> > The host would write to the local disk, the VMs - to a shared storage,
> > across NFS or block layer or so. Moreover, their IO is not buffered.
> > There is very little IO coming from the host itself, generally (I hope
> > so!).
> >
> > Partially unrelated - the trend today is actually to put NOOP on the VMs
> -
> > the deadline is quite meaningless, as the host scheduler will reschedule
> > anyway as it see fits.
> > Most likely it is also a deadline scheduler (but could be NOOP as well if
> > it's an all flash array, for example). Therefore there is no reason for
> > anything but simple NOOP on the VMs themselves.
> >
> > In short, I think it's an outdated decision that perhaps should be
> > revisited. Not urgent, though.
> > Y.
>
> I know it's not the most cared about usecase
> but I'd like to add that this might affect local storage
> domains, which I happen to use a lot, and maybe others too.
>

In which case we should change the default for local storage.
BTW, when using Gluster, they have returned these values to the original
values...
Y.


> --
> Mit freundlichen Grüßen / Regards
>
> Sven Kieske
>
> Systemadministrator
> Mittwald CM Service GmbH & Co. KG
> Königsberger Straße 6
> 32339 Espelkamp
> T: +495772 293100
> F: +495772 29
> https://www.mittwald.de
> Geschäftsführer: Robert Meyer
> St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
> Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
>
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] VDSM changes Linux memory dirty ratios - why?

[Martin Polednik]

On 30/11/16 13:10 +0100, Sven Kieske wrote:

On 30/11/16 08:48, Martin Polednik wrote:

It's not really irrelevant, the host still uses disk cache. Anyway,
there is BZ[1] with a presentation[2] that (imho reasonably) states:

"Reduce dirty page limits in KVM host to allow
direct I/O writer VMs to compete successfully
with buffered writer processes for storage
access"

I wonder why virtual-host tuned profile doesn't contain these values:

$ grep vm.dirty /usr/lib/tuned/virtual-host/tuned.conf
vm.dirty_background_ratio = 5

[1]https://bugzilla.redhat.com/show_bug.cgi?id=740887
[2]http://perf1.lab.bos.redhat.com/bengland/laptop/rhev/rhev-vm-rsptime.pdf


Could you share [2] with the wider community? This would be awesome!


Sorry, I've totally missed the fact it's internal link (unfortunately
publicly visible on the BZ). I believe it's slightly outdated, but
let's ask the author.

Ben, is the document[2] somehow still valid and could it be made publicly
available?

Re-referencing for completeness:
[2]http://perf1.lab.bos.redhat.com/bengland/laptop/rhev/rhev-vm-rsptime.pdf


--
Mit freundlichen Grüßen / Regards

Sven Kieske

Systemadministrator
Mittwald CM Service GmbH & Co. KG
Königsberger Straße 6
32339 Espelkamp
T: +495772 293100
F: +495772 29
https://www.mittwald.de
Geschäftsführer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen







___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel


___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] VDSM changes Linux memory dirty ratios - why?

[Sven Kieske]

On 30/11/16 09:13, Yaniv Kaul wrote:
> Thanks, but it really makes no sense to me. The direct IO by the VMs is
> going to a different storage than what the host is writing to, in most
> cases.
> The host would write to the local disk, the VMs - to a shared storage,
> across NFS or block layer or so. Moreover, their IO is not buffered.
> There is very little IO coming from the host itself, generally (I hope
> so!).
> 
> Partially unrelated - the trend today is actually to put NOOP on the VMs -
> the deadline is quite meaningless, as the host scheduler will reschedule
> anyway as it see fits.
> Most likely it is also a deadline scheduler (but could be NOOP as well if
> it's an all flash array, for example). Therefore there is no reason for
> anything but simple NOOP on the VMs themselves.
> 
> In short, I think it's an outdated decision that perhaps should be
> revisited. Not urgent, though.
> Y.

I know it's not the most cared about usecase
but I'd like to add that this might affect local storage
domains, which I happen to use a lot, and maybe others too.

-- 
Mit freundlichen Grüßen / Regards

Sven Kieske

Systemadministrator
Mittwald CM Service GmbH & Co. KG
Königsberger Straße 6
32339 Espelkamp
T: +495772 293100
F: +495772 29
https://www.mittwald.de
Geschäftsführer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen



signature.asc
Description: OpenPGP digital signature
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] VDSM changes Linux memory dirty ratios - why?

[Sven Kieske]

On 30/11/16 08:48, Martin Polednik wrote:
> It's not really irrelevant, the host still uses disk cache. Anyway,
> there is BZ[1] with a presentation[2] that (imho reasonably) states:
> 
> "Reduce dirty page limits in KVM host to allow
> direct I/O writer VMs to compete successfully
> with buffered writer processes for storage
> access"
> 
> I wonder why virtual-host tuned profile doesn't contain these values:
> 
> $ grep vm.dirty /usr/lib/tuned/virtual-host/tuned.conf
> vm.dirty_background_ratio = 5
> 
> [1]https://bugzilla.redhat.com/show_bug.cgi?id=740887
> [2]http://perf1.lab.bos.redhat.com/bengland/laptop/rhev/rhev-vm-rsptime.pdf

Could you share [2] with the wider community? This would be awesome!
-- 
Mit freundlichen Grüßen / Regards

Sven Kieske

Systemadministrator
Mittwald CM Service GmbH & Co. KG
Königsberger Straße 6
32339 Espelkamp
T: +495772 293100
F: +495772 29
https://www.mittwald.de
Geschäftsführer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen



signature.asc
Description: OpenPGP digital signature
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] python path conflicts issues while running python scripts from vdsm verbs

[Dan Kenigsberg]

On Wed, Nov 30, 2016 at 03:49:02AM -0500, Ramesh Nachimuthu wrote:
> 
> 
> 
> 
> - Original Message -
> > From: "Dan Kenigsberg" 
> > To: "Aravinda" , igoih...@redhat.com
> > Cc: "Ramesh Nachimuthu" , "Nir Soffer" 
> > , "devel" , "Piotr
> > Kliczewski" , "Yaniv Bronheim" , 
> > "Sahina Bose" 
> > Sent: Wednesday, November 30, 2016 2:02:43 PM
> > Subject: Re: python path conflicts issues while running python scripts from 
> > vdsm verbs
> > 
> > 
> > Regardless of the solution you plan to take, let it serve as a reminder
> > about moving gluster bits out of /usr/share/vdsm/ and into
> > site-packages. I am willing to pay the price of an ugly
> > exception-handling in xmlrpc, if this is the cost.
> 
> Yes. I think this is the right time to revive the patch 
> https://gerrit.ovirt.org/#/c/57530/ and https://gerrit.ovirt.org/#/c/57524/ 
> to move all the gluster code to site-packages. Let me re-post these packages 
> for review.

Please do.

IIRC https://gerrit.ovirt.org/#/c/57530/9 is broken in xmlrpc.

Since xmlrpc is going to be used only by the command line in 4.1, I'd be
pleased to see the problem solved via a simplistic hack.
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] python path conflicts issues while running python scripts from vdsm verbs

[Ramesh Nachimuthu]

- Original Message -
> From: "Dan Kenigsberg" 
> To: "Aravinda" , igoih...@redhat.com
> Cc: "Ramesh Nachimuthu" , "Nir Soffer" 
> , "devel" , "Piotr
> Kliczewski" , "Yaniv Bronheim" , 
> "Sahina Bose" 
> Sent: Wednesday, November 30, 2016 2:02:43 PM
> Subject: Re: python path conflicts issues while running python scripts from 
> vdsm verbs
> 
> 
> Regardless of the solution you plan to take, let it serve as a reminder
> about moving gluster bits out of /usr/share/vdsm/ and into
> site-packages. I am willing to pay the price of an ugly
> exception-handling in xmlrpc, if this is the cost.

Yes. I think this is the right time to revive the patch 
https://gerrit.ovirt.org/#/c/57530/ and https://gerrit.ovirt.org/#/c/57524/ to 
move all the gluster code to site-packages. Let me re-post these packages for 
review.

Regards,
Ramesh

> 
> On Wed, Nov 30, 2016 at 11:27:06AM +0530, Aravinda wrote:
> > If vdsm setting PYTHONPATH as below
> > 
> > export PYTHONPATH=/usr/share/vdsm:$PYTHONPATH
> > 
> > change it as
> > 
> > export PYTHONPATH=$PYTHONPATH:/usr/share/vdsm
> > 
> > With this change, /usr/share/vdsm will not get first preference compared to
> > /usr/lib*/python2.7/site-packages
> > 
> > regards
> > Aravinda
> > 
> > On 11/30/2016 10:25 AM, Ramesh Nachimuthu wrote:
> > > 
> > > 
> > > 
> > > - Original Message -
> > > > From: "Nir Soffer" 
> > > > To: "Ramesh Nachimuthu" 
> > > > Cc: "devel" , "Dan Kenigsberg" ,
> > > > "Piotr Kliczewski" , "Yaniv
> > > > Bronheim" , "Aravinda Vishwanathapura Krishna
> > > > Murthy" , "Sahina Bose"
> > > > 
> > > > Sent: Tuesday, November 29, 2016 7:20:57 PM
> > > > Subject: Re: python path conflicts issues while running python scripts
> > > > from vdsm verbs
> > > > 
> > > > On Tue, Nov 29, 2016 at 3:26 PM, Ramesh Nachimuthu
> > > > 
> > > > wrote:
> > > > > Hi,
> > > > > 
> > > > >   I am trying to run a python script '/sbin/gluster-eventsapi' in
> > > > >   vdsm verb
> > > > >   which internally imports some python modules from
> > > > >   /usr/lib/python2.7/site-packages/gluster/cliutils/. But it fails
> > > > >   with the
> > > > >   import error. Following error is seen in the supervdsm log.
> > > > > 
> > > > > 
> > > > > MainProcess|Thread-1::DEBUG::2016-11-28
> > > > > 16:54:35,130::commands::93::root::(execCmd) FAILED:  =
> > > > > 'Traceback
> > > > > (most recent call last):\n  File "/sbin/gluster-eventsapi", line 25,
> > > > > in
> > > > > \nfrom gluster.cliutils.cliutils import (Cmd, execute,
> > > > > node_output_ok, node_output_notok,\nImportError: No module named
> > > > > cliutils.cliutils\n';  = 1
> > > > > 
> > > > > 
> > > > > I think the import statement "from gluster.cliutils.cliutils import
> > > > > (Cmd,
> > > > > execute, node_output_ok, node_output_notok)" in the python script
> > > > > resolves
> > > > > to '/usr/share/vdsm/gluster' instead of
> > > > > /usr/lib/python2.7/site-packages/gluster/cliutils.
> > > > > 
> > > > > I see the following in python sys.path while executing a python
> > > > > script from
> > > > > vdsm.
> > > > > 
> > > > > ['/usr/libexec/glusterfs', '/usr/share/vdsm',
> > > > > '/usr/lib64/python27.zip',
> > > > > '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2',
> > > > > '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old',
> > > > > '/usr/lib64/python2.7/lib-dynload',
> > > > > '/usr/lib64/python2.7/site-packages',
> > > > > '/usr/lib64/python2.7/site-packages/gtk-2.0',
> > > > > '/usr/lib/python2.7/site-packages']
> > > > > 
> > > > > Looks like '/usr/share/vdsm' take precedence over
> > > > > '/usr/lib64/python2.7/site-packages'.
> > > > > 
> > > > > Can someone suggests a way to fix this issue?
> > > > > 
> > > > > Note: '/sbin/gluster-eventsapi' works perfectly while running
> > > > > directly from
> > > > > CLI.
> > > > > 
> > > > > Related vdsm patch:
> > > > > https://gerrit.ovirt.org/#/c/67168/2/vdsm/gluster/events.py
> > > > If an import fails because /usr/share/vdsm/foo hides a package in
> > > > /usr/lib/python2.7/site-packages/vdsm it means that you have a wrong
> > > > import, you need to import anything like this:
> > > > 
> > > >  from vdsm.foo import bar
> > > > 
> > > > So names in /usr/share/vdsm/* cannot hide names in vdsm package.
> > > > 
> > > Here its a different case. Some external python script (which I am trying
> > > to run from vdsm verb) is trying to import some gluster package from
> > > /usr/lib/python2.7/site-packages/gluster. But it is conflicting with
> > > gluster modules under /usr/share/vdsm/gluster
> > > 
> > > Python Script which I am trying to run:
> > > 

Re: [ovirt-devel] python path conflicts issues while running python scripts from vdsm verbs

[Dan Kenigsberg]

Regardless of the solution you plan to take, let it serve as a reminder
about moving gluster bits out of /usr/share/vdsm/ and into
site-packages. I am willing to pay the price of an ugly
exception-handling in xmlrpc, if this is the cost.

On Wed, Nov 30, 2016 at 11:27:06AM +0530, Aravinda wrote:
> If vdsm setting PYTHONPATH as below
> 
> export PYTHONPATH=/usr/share/vdsm:$PYTHONPATH
> 
> change it as
> 
> export PYTHONPATH=$PYTHONPATH:/usr/share/vdsm
> 
> With this change, /usr/share/vdsm will not get first preference compared to
> /usr/lib*/python2.7/site-packages
> 
> regards
> Aravinda
> 
> On 11/30/2016 10:25 AM, Ramesh Nachimuthu wrote:
> > 
> > 
> > 
> > - Original Message -
> > > From: "Nir Soffer" 
> > > To: "Ramesh Nachimuthu" 
> > > Cc: "devel" , "Dan Kenigsberg" , 
> > > "Piotr Kliczewski" , "Yaniv
> > > Bronheim" , "Aravinda Vishwanathapura Krishna 
> > > Murthy" , "Sahina Bose"
> > > 
> > > Sent: Tuesday, November 29, 2016 7:20:57 PM
> > > Subject: Re: python path conflicts issues while running python scripts 
> > > from vdsm verbs
> > > 
> > > On Tue, Nov 29, 2016 at 3:26 PM, Ramesh Nachimuthu 
> > > wrote:
> > > > Hi,
> > > > 
> > > >   I am trying to run a python script '/sbin/gluster-eventsapi' in vdsm 
> > > > verb
> > > >   which internally imports some python modules from
> > > >   /usr/lib/python2.7/site-packages/gluster/cliutils/. But it fails with 
> > > > the
> > > >   import error. Following error is seen in the supervdsm log.
> > > > 
> > > > 
> > > > MainProcess|Thread-1::DEBUG::2016-11-28
> > > > 16:54:35,130::commands::93::root::(execCmd) FAILED:  = 'Traceback
> > > > (most recent call last):\n  File "/sbin/gluster-eventsapi", line 25, in
> > > > \nfrom gluster.cliutils.cliutils import (Cmd, execute,
> > > > node_output_ok, node_output_notok,\nImportError: No module named
> > > > cliutils.cliutils\n';  = 1
> > > > 
> > > > 
> > > > I think the import statement "from gluster.cliutils.cliutils import 
> > > > (Cmd,
> > > > execute, node_output_ok, node_output_notok)" in the python script 
> > > > resolves
> > > > to '/usr/share/vdsm/gluster' instead of
> > > > /usr/lib/python2.7/site-packages/gluster/cliutils.
> > > > 
> > > > I see the following in python sys.path while executing a python script 
> > > > from
> > > > vdsm.
> > > > 
> > > > ['/usr/libexec/glusterfs', '/usr/share/vdsm', '/usr/lib64/python27.zip',
> > > > '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2',
> > > > '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old',
> > > > '/usr/lib64/python2.7/lib-dynload', 
> > > > '/usr/lib64/python2.7/site-packages',
> > > > '/usr/lib64/python2.7/site-packages/gtk-2.0',
> > > > '/usr/lib/python2.7/site-packages']
> > > > 
> > > > Looks like '/usr/share/vdsm' take precedence over
> > > > '/usr/lib64/python2.7/site-packages'.
> > > > 
> > > > Can someone suggests a way to fix this issue?
> > > > 
> > > > Note: '/sbin/gluster-eventsapi' works perfectly while running directly 
> > > > from
> > > > CLI.
> > > > 
> > > > Related vdsm patch:
> > > > https://gerrit.ovirt.org/#/c/67168/2/vdsm/gluster/events.py
> > > If an import fails because /usr/share/vdsm/foo hides a package in
> > > /usr/lib/python2.7/site-packages/vdsm it means that you have a wrong
> > > import, you need to import anything like this:
> > > 
> > >  from vdsm.foo import bar
> > > 
> > > So names in /usr/share/vdsm/* cannot hide names in vdsm package.
> > > 
> > Here its a different case. Some external python script (which I am trying 
> > to run from vdsm verb) is trying to import some gluster package from 
> > /usr/lib/python2.7/site-packages/gluster. But it is conflicting with 
> > gluster modules under /usr/share/vdsm/gluster
> > 
> > Python Script which I am trying to run: 
> > https://github.com/gluster/glusterfs/blob/master/events/src/peer_eventsapi.py
> > 
> > Regards,
> > Ramesh
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Gerrit parallel patch handling and CI (Or, why did my code fail post-merge)

[Barak Korren]

On 29 November 2016 at 19:34, Vojtech Szocs  wrote:
>
> Question: after the patch is submitted in Gerrit (it's fully acked
> and maintainer hits the "Submit" button), does Gerrit allow us to
> run CI (e.g. `check-merged` script) *before* doing the actual merge?
>
> [In other words, does Gerrit support gating merge based on CI script?]

No.
See separate thread about merge-gateing:
http://lists.ovirt.org/pipermail/devel/2016-November/014192.html

> That said, +1 on "Fast Forward Only" submit type.

So far only VDSM stepped forward to try it out - and it proved to be
too cumbersome for VDSM maintainers - so switched back to "Merge if
Necessary".

-- 
Barak Korren
bkor...@redhat.com
RHCE, RHCi, RHV-DevOps Team
https://ifireball.wordpress.com/
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] VDSM changes Linux memory dirty ratios - why?

[Yaniv Kaul]

On Wed, Nov 30, 2016 at 9:48 AM, Martin Polednik 
wrote:

> On 29/11/16 22:01 +0200, Yaniv Kaul wrote:
>
>> It appears that VDSM changes the following params:
>> vm.dirty_ratio = 5
>> vm.dirty_background_ratio = 2
>>
>> Any idea why? Because we use cache=none it's irrelevant anyway?
>>
>
> It's not really irrelevant, the host still uses disk cache. Anyway,
> there is BZ[1] with a presentation[2] that (imho reasonably) states:
>
> "Reduce dirty page limits in KVM host to allow
> direct I/O writer VMs to compete successfully
> with buffered writer processes for storage
> access"
>

Thanks, but it really makes no sense to me. The direct IO by the VMs is
going to a different storage than what the host is writing to, in most
cases.
The host would write to the local disk, the VMs - to a shared storage,
across NFS or block layer or so. Moreover, their IO is not buffered.
There is very little IO coming from the host itself, generally (I hope
so!).

Partially unrelated - the trend today is actually to put NOOP on the VMs -
the deadline is quite meaningless, as the host scheduler will reschedule
anyway as it see fits.
Most likely it is also a deadline scheduler (but could be NOOP as well if
it's an all flash array, for example). Therefore there is no reason for
anything but simple NOOP on the VMs themselves.

In short, I think it's an outdated decision that perhaps should be
revisited. Not urgent, though.
Y.


> I wonder why virtual-host tuned profile doesn't contain these values:
>
> $ grep vm.dirty /usr/lib/tuned/virtual-host/tuned.conf
> vm.dirty_background_ratio = 5
>
> [1]https://bugzilla.redhat.com/show_bug.cgi?id=740887
> [2]http://perf1.lab.bos.redhat.com/bengland/laptop/rhev/
> rhev-vm-rsptime.pdf
>
> TIA,
>> Y.
>>
>
> ___
>> Devel mailing list
>> Devel@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel