Just another thought, from looking at the vdsm logs it looks like there are too many calls to getVolumeSize that are eating up all handlers and eventually a timeout occurs. Adam, do you have any idea about this?
----- Original Message ----- > From: "Yeela Kaplan" <[email protected]> > To: "Tiemen Ruiten" <[email protected]> > Cc: "[email protected]" <[email protected]> > Sent: Wednesday, December 17, 2014 1:30:52 PM > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > > > ----- Original Message ----- > > From: "Tiemen Ruiten" <[email protected]> > > To: "Yeela Kaplan" <[email protected]> > > Cc: "[email protected]" <[email protected]> > > Sent: Wednesday, December 17, 2014 1:22:59 PM > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > creating VM from template > > > > Thank you, I will try to increase to 20 and see what happens. Bug is filed: > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255 > > Thank you > and I forgot to mention that you have to restart vdsmd so the change will > apply... > > > > > > On 17 December 2014 at 11:48, Yeela Kaplan <[email protected]> wrote: > > > > > > Hi, > > > You are right, the problem is with the file handlers. > > > You can increase the number of handlers in pool using the vdsm config > > > file, which is supposed to be under the following directory in your > > > system: > > > > > > /usr/lib/python2.6/site-packages/vdsm/config.py > > > > > > The default value for 'process_pool_max_slots_per_domain' is 10, so you > > > can increase it by a bit, but not too much. > > > > > > But I suspect the problem is in a larger scale, and this is only a > > > temporary relief for your system and this needs much more attention and a > > > proper fix. > > > could you please open a bug on RHEV/vdsm in bugzilla stating all of the > > > details of your setup and logs? > > > > > > thanks, > > > Yeela > > > > > > ----- Original Message ----- > > > > From: "Tiemen Ruiten" <[email protected]> > > > > To: "[email protected]" <[email protected]> > > > > Sent: Wednesday, December 17, 2014 10:53:39 AM > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > > creating VM from template > > > > > > > > Would this be limits for the vdsm process? Then what is the proper way > > > > to > > > > change ulimits for VDSM? > > > > > > > > On 16 December 2014 at 20:45, Donny Davis < [email protected] > wrote: > > > > > > > > > > > > > > > > > > > > The only thing I can think of would be file hard and soft limits, but I > > > am no > > > > oVirt pro. > > > > > > > > > > > > > > > > 'no free file handlers in pool' that would make sense to me… > > > > > > > > Donny > > > > > > > > > > > > > > > > > > > > > > > > From: Tiemen Ruiten [mailto: [email protected] ] > > > > Sent: Tuesday, December 16, 2014 12:40 PM > > > > To: Donny Davis > > > > Cc: [email protected] > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > > > creating VM from template > > > > > > > > > > > > > > > > > > > > > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 > > > > hyperthreaded cores and 64 GB of RAM each. At the moment I created the > > > VM, > > > > processor load on both nodes was less than 1. > > > > > > > > > > > > > > > > > > > > > > > > On 16 December 2014 at 20:29, Donny Davis < [email protected] > wrote: > > > > > > > > What is the VM load you are running on your servers? > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: [email protected] [mailto: [email protected] ] On > > > Behalf Of > > > > Tiemen Ruiten > > > > Sent: Tuesday, December 16, 2014 12:27 PM > > > > To: [email protected] > > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while > > > creating > > > > VM from template > > > > > > > > Hello, > > > > > > > > I ran into a nasty problem today when creating a new, cloned VM from a > > > > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the > > > node > > > > where I started a VM creation job, load skyrocketed and some VMs > > > > stopped > > > > responding until and after the job failed. Everything recovered without > > > > intervention, but this obviously shouldn't happen. I have attached the > > > > relevant vdsm log file. The button to create the VM was pressed around > > > > 11:17, the first error in the vdsm log is at 11:23:58. > > > > > > > > The ISO domain is a gluster volume exposed via NFS, the storage domain > > > for > > > > the VM's is also a gluster volume. The underlying filesystem is ZFS. > > > > The hypervisor nodes are full CentOS 6 installs. > > > > > > > > I'm guessing the 'no free file handlers in pool' in the vdsm log file > > > > is > > > key > > > > here. What can I do to prevent this from happening again? Apart from > > > > not > > > > creating new VMs of course :) > > > > > > > > Tiemen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Tiemen Ruiten > > > > Systems Engineer > > > > R&D Media > > > > > > > > > > > > -- > > > > Tiemen Ruiten > > > > Systems Engineer > > > > R&D Media > > > > > > > > _______________________________________________ > > > > Users mailing list > > > > [email protected] > > > > http://lists.ovirt.org/mailman/listinfo/users > > > > > > > > > > > > > -- > > Tiemen Ruiten > > Systems Engineer > > R&D Media > > > _______________________________________________ > Users mailing list > [email protected] > http://lists.ovirt.org/mailman/listinfo/users > _______________________________________________ Users mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/users
