OK, I suggest to test using a VM with local disk (preferably on a host with SSD configured), if its working, lets expedite moving all VMs or at least a large amount of VMs to it until we see network load reduced.
e. On Wed, May 25, 2016 at 12:38 PM, Evgheni Dereveanchin <[email protected]> wrote: > Hi, > > I checked SAR data on storage servers and compared loads > yesterday and three weeks ago (May 3). Load values are in > pretty much the same range, yet now most of the time they > are around the "high" mark so we may be nearing a bottleneck, > specifically on I/O where we mostly do writes to the NAS, > not reads and there's quite a bit of overhead: > > VM -> QCOW -> file -> network -> NFS -> DRBD -> disk > > Surely, using local scratch disks stored on SSDs should greatly > improve performance as at least half of the above steps will be gone. > We don't really need to centrally store (NFS) or mirror (DRBD) > data that slaves write to their disks all the time anyways. > > For VMs where we do need redundancy, I'd suggest using > iSCSI storage domains in the long run. > > Regards, > Evgheni Dereveanchin > > ----- Original Message ----- > From: "Eyal Edri" <[email protected]> > To: "Sandro Bonazzola" <[email protected]>, "Evgheni Dereveanchin" < > [email protected]>, "Anton Marchukov" <[email protected]> > Cc: "Fabian Deutsch" <[email protected]>, "infra" <[email protected]> > Sent: Wednesday, 25 May, 2016 9:31:43 AM > Subject: Re: ngn build jobs take more than twice (x) as long as in the > last days > > It might be more load on the storage servers with now running much more > jobs. > Evgheni - can you check if the load on the storage servers has changed > significantly to justify this degradation of service? > > We need to expedite the enablement of SSDs in the hypervisors and move to > local hooks. > Anton - do we have a test VM that uses a local DISK we can use to test if > it improves the runtime? > > On Tue, May 24, 2016 at 11:19 PM, Sandro Bonazzola <[email protected]> > wrote: > > > > > Il 24/Mag/2016 17:57, "Fabian Deutsch" <[email protected]> ha scritto: > > > > > > Hey, > > > > > > $subj says it all. > > > > > > Affected jobs are: > > > http://jenkins.ovirt.org/user/fabiand/my-views/view/ovirt-node-ng/ > > > > > > I.e. 3.6 - before: ~46min, now 1:23hrs > > > > > > In master it's even worse: >1:30hrs > > > > > > Can someone help to idnetify the reason? > > > > I have no numbers but I have the feeling that all jobs are getting slower > > since a couple of weeks ago. Yum install phase takes ages. I thoughtit > was > > some temporary storage i/o peak but looks like it's not temporary. > > > > > > > > - fabian > > > > > > -- > > > Fabian Deutsch <[email protected]> > > > RHEV Hypervisor > > > Red Hat > > > _______________________________________________ > > > Infra mailing list > > > [email protected] > > > http://lists.ovirt.org/mailman/listinfo/infra > > > > _______________________________________________ > > Infra mailing list > > [email protected] > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > > > -- > Eyal Edri > Associate Manager > RHEV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) > -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/infra
