Yeela, any chance you could shed any light on this? On 5 January 2015 at 15:46, Tiemen Ruiten <[email protected]> wrote:
> Sorry, to be clear: no default disk policy exists for the 3.4 cluster > storage domain. > > On 5 January 2015 at 15:45, Tiemen Ruiten <[email protected]> wrote: > >> Hello, >> >> I just tried creating a thin-provisioned disk and the same problem (load >> spike, non responsive VMs) occurred. I may have an idea what is going wrong >> now. The following is an excerpt from the engine log after starting the >> disk create job: >> >> 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery] >> (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is >> StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM >> vds_groups_view WHERE ( vds_group_id IN (SELECT distinct >> vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain >> LEFT OUTER JOIN storage_pool_with_storage_domain ON >> vds_groups_storage_domain.storage_pool_id= >> storage_pool_with_storage_domain.id WHERE >> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY >> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is >> org.postgresql.util.PSQLException: The column name disk_storage_type was >> not found in this ResultSet. : >> org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL >> grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id >> IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM >> vds_groups_storage_domain LEFT OUTER JOIN >> storage_pool_with_storage_domain ON >> vds_groups_storage_domain.storage_pool_id= >> storage_pool_with_storage_domain.id WHERE >> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY >> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is >> org.postgresql.util.PSQLException: The column name disk_storage_type was >> not found in this ResultSet.: >> org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL >> grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id >> IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM >> vds_groups_storage_domain LEFT OUTER JOIN >> storage_pool_with_storage_domain ON >> vds_groups_storage_domain.storage_pool_id= >> storage_pool_with_storage_domain.id WHERE >> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY >> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is >> org.postgresql.util.PSQLException: The column name disk_storage_type was >> not found in this ResultSet. >> >> This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5. >> There's also a 3.5 cluster managed by the same engine. Could this be >> related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>? >> >> >> >> On 17 December 2014 at 12:34, Yeela Kaplan <[email protected]> wrote: >> >>> Just another thought, >>> from looking at the vdsm logs it looks like there are too many calls to >>> getVolumeSize that are eating up all handlers >>> and eventually a timeout occurs. >>> Adam, do you have any idea about this? >>> >>> ----- Original Message ----- >>> > From: "Yeela Kaplan" <[email protected]> >>> > To: "Tiemen Ruiten" <[email protected]> >>> > Cc: "[email protected]" <[email protected]> >>> > Sent: Wednesday, December 17, 2014 1:30:52 PM >>> > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while >>> creating VM from template >>> > >>> > >>> > >>> > ----- Original Message ----- >>> > > From: "Tiemen Ruiten" <[email protected]> >>> > > To: "Yeela Kaplan" <[email protected]> >>> > > Cc: "[email protected]" <[email protected]> >>> > > Sent: Wednesday, December 17, 2014 1:22:59 PM >>> > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' >>> while >>> > > creating VM from template >>> > > >>> > > Thank you, I will try to increase to 20 and see what happens. Bug is >>> filed: >>> > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255 >>> > >>> > Thank you >>> > and I forgot to mention that you have to restart vdsmd so the change >>> will >>> > apply... >>> > >>> > >>> > > >>> > > On 17 December 2014 at 11:48, Yeela Kaplan <[email protected]> >>> wrote: >>> > > > >>> > > > Hi, >>> > > > You are right, the problem is with the file handlers. >>> > > > You can increase the number of handlers in pool using the vdsm >>> config >>> > > > file, which is supposed to be under the following directory in your >>> > > > system: >>> > > > >>> > > > /usr/lib/python2.6/site-packages/vdsm/config.py >>> > > > >>> > > > The default value for 'process_pool_max_slots_per_domain' is 10, >>> so you >>> > > > can increase it by a bit, but not too much. >>> > > > >>> > > > But I suspect the problem is in a larger scale, and this is only a >>> > > > temporary relief for your system and this needs much more >>> attention and a >>> > > > proper fix. >>> > > > could you please open a bug on RHEV/vdsm in bugzilla stating all >>> of the >>> > > > details of your setup and logs? >>> > > > >>> > > > thanks, >>> > > > Yeela >>> > > > >>> > > > ----- Original Message ----- >>> > > > > From: "Tiemen Ruiten" <[email protected]> >>> > > > > To: "[email protected]" <[email protected]> >>> > > > > Sent: Wednesday, December 17, 2014 10:53:39 AM >>> > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' >>> while >>> > > > creating VM from template >>> > > > > >>> > > > > Would this be limits for the vdsm process? Then what is the >>> proper way >>> > > > > to >>> > > > > change ulimits for VDSM? >>> > > > > >>> > > > > On 16 December 2014 at 20:45, Donny Davis < [email protected] >>> > wrote: >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > The only thing I can think of would be file hard and soft >>> limits, but I >>> > > > am no >>> > > > > oVirt pro. >>> > > > > >>> > > > > >>> > > > > >>> > > > > 'no free file handlers in pool' that would make sense to me… >>> > > > > >>> > > > > Donny >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > From: Tiemen Ruiten [mailto: [email protected] ] >>> > > > > Sent: Tuesday, December 16, 2014 12:40 PM >>> > > > > To: Donny Davis >>> > > > > Cc: [email protected] >>> > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' >>> while >>> > > > > creating VM from template >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with >>> 16 >>> > > > > hyperthreaded cores and 64 GB of RAM each. At the moment I >>> created the >>> > > > VM, >>> > > > > processor load on both nodes was less than 1. >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > On 16 December 2014 at 20:29, Donny Davis < [email protected] >>> > wrote: >>> > > > > >>> > > > > What is the VM load you are running on your servers? >>> > > > > >>> > > > > >>> > > > > >>> > > > > -----Original Message----- >>> > > > > From: [email protected] [mailto: [email protected] >>> ] On >>> > > > Behalf Of >>> > > > > Tiemen Ruiten >>> > > > > Sent: Tuesday, December 16, 2014 12:27 PM >>> > > > > To: [email protected] >>> > > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' >>> while >>> > > > creating >>> > > > > VM from template >>> > > > > >>> > > > > Hello, >>> > > > > >>> > > > > I ran into a nasty problem today when creating a new, cloned VM >>> from a >>> > > > > template (one virtual 20 GBdisk) on our two-node oVirt cluster: >>> on the >>> > > > node >>> > > > > where I started a VM creation job, load skyrocketed and some VMs >>> > > > > stopped >>> > > > > responding until and after the job failed. Everything recovered >>> without >>> > > > > intervention, but this obviously shouldn't happen. I have >>> attached the >>> > > > > relevant vdsm log file. The button to create the VM was pressed >>> around >>> > > > > 11:17, the first error in the vdsm log is at 11:23:58. >>> > > > > >>> > > > > The ISO domain is a gluster volume exposed via NFS, the storage >>> domain >>> > > > for >>> > > > > the VM's is also a gluster volume. The underlying filesystem is >>> ZFS. >>> > > > > The hypervisor nodes are full CentOS 6 installs. >>> > > > > >>> > > > > I'm guessing the 'no free file handlers in pool' in the vdsm log >>> file >>> > > > > is >>> > > > key >>> > > > > here. What can I do to prevent this from happening again? Apart >>> from >>> > > > > not >>> > > > > creating new VMs of course :) >>> > > > > >>> > > > > Tiemen >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > -- >>> > > > > >>> > > > > >>> > > > > Tiemen Ruiten >>> > > > > Systems Engineer >>> > > > > R&D Media >>> > > > > >>> > > > > >>> > > > > -- >>> > > > > Tiemen Ruiten >>> > > > > Systems Engineer >>> > > > > R&D Media >>> > > > > >>> > > > > _______________________________________________ >>> > > > > Users mailing list >>> > > > > [email protected] >>> > > > > http://lists.ovirt.org/mailman/listinfo/users >>> > > > > >>> > > > >>> > > >>> > > >>> > > -- >>> > > Tiemen Ruiten >>> > > Systems Engineer >>> > > R&D Media >>> > > >>> > _______________________________________________ >>> > Users mailing list >>> > [email protected] >>> > http://lists.ovirt.org/mailman/listinfo/users >>> > >>> >> >> >> >> -- >> Tiemen Ruiten >> Systems Engineer >> R&D Media >> > > > > -- > Tiemen Ruiten > Systems Engineer > R&D Media > -- Tiemen Ruiten Systems Engineer R&D Media
_______________________________________________ Users mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/users
