Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
After creating a default disk profile, I can add new disks and create VMs without issues. I have updated the bug I reported as a duplicate of 1145241. https://bugzilla.redhat.com/show_bug.cgi?id=1175255 On 5 January 2015 at 15:46, Tiemen Ruiten wrote: > Sorry, to be clear: no default disk policy exists for the 3.4 cluster > storage domain. > > On 5 January 2015 at 15:45, Tiemen Ruiten wrote: > >> Hello, >> >> I just tried creating a thin-provisioned disk and the same problem (load >> spike, non responsive VMs) occurred. I may have an idea what is going wrong >> now. The following is an excerpt from the engine log after starting the >> disk create job: >> >> 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery] >> (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is >> StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM >> vds_groups_view WHERE ( vds_group_id IN (SELECT distinct >> vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain >> LEFT OUTER JOIN storage_pool_with_storage_domain ON >> vds_groups_storage_domain.storage_pool_id= >> storage_pool_with_storage_domain.idWHERE >> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY >> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is >> org.postgresql.util.PSQLException: The column name disk_storage_type was >> not found in this ResultSet. : >> org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL >> grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id >> IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM >> vds_groups_storage_domain LEFT OUTER JOIN >> storage_pool_with_storage_domain ON >> vds_groups_storage_domain.storage_pool_id= >> storage_pool_with_storage_domain.idWHERE >> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY >> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is >> org.postgresql.util.PSQLException: The column name disk_storage_type was >> not found in this ResultSet.: >> org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL >> grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id >> IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM >> vds_groups_storage_domain LEFT OUTER JOIN >> storage_pool_with_storage_domain ON >> vds_groups_storage_domain.storage_pool_id= >> storage_pool_with_storage_domain.idWHERE >> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY >> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is >> org.postgresql.util.PSQLException: The column name disk_storage_type was >> not found in this ResultSet. >> >> This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5. >> There's also a 3.5 cluster managed by the same engine. Could this be >> related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>? >> >> >> >> On 17 December 2014 at 12:34, Yeela Kaplan wrote: >> >>> Just another thought, >>> from looking at the vdsm logs it looks like there are too many calls to >>> getVolumeSize that are eating up all handlers >>> and eventually a timeout occurs. >>> Adam, do you have any idea about this? >>> >>> - Original Message - >>> > From: "Yeela Kaplan" >>> > To: "Tiemen Ruiten" >>> > Cc: "Users@ovirt.org" >>> > Sent: Wednesday, December 17, 2014 1:30:52 PM >>> > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while >>> creating VM from template >>> > >>> > >>> > >>> > - Original Message - >>> > > From: "Tiemen Ruiten" >>> > > To: "Yeela Kaplan" >>> > > Cc: "Users@ovirt.org" >>> > > Sent: Wednesday, December 17, 2014 1:22:59 PM >>> > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' >>> while >>> > > creating VM from template >>> > > >>> > > Thank you, I will try to increase to 20 and see what happens. Bug is >>> filed: >>> > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255 >>> > >>> > Thank you >>> > and I forgot to mention that you have to restart vdsmd so the change >>> will >>> > apply... >>> > >>> > >>&g
Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Yeela, any chance you could shed any light on this? On 5 January 2015 at 15:46, Tiemen Ruiten wrote: > Sorry, to be clear: no default disk policy exists for the 3.4 cluster > storage domain. > > On 5 January 2015 at 15:45, Tiemen Ruiten wrote: > >> Hello, >> >> I just tried creating a thin-provisioned disk and the same problem (load >> spike, non responsive VMs) occurred. I may have an idea what is going wrong >> now. The following is an excerpt from the engine log after starting the >> disk create job: >> >> 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery] >> (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is >> StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM >> vds_groups_view WHERE ( vds_group_id IN (SELECT distinct >> vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain >> LEFT OUTER JOIN storage_pool_with_storage_domain ON >> vds_groups_storage_domain.storage_pool_id= >> storage_pool_with_storage_domain.idWHERE >> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY >> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is >> org.postgresql.util.PSQLException: The column name disk_storage_type was >> not found in this ResultSet. : >> org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL >> grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id >> IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM >> vds_groups_storage_domain LEFT OUTER JOIN >> storage_pool_with_storage_domain ON >> vds_groups_storage_domain.storage_pool_id= >> storage_pool_with_storage_domain.idWHERE >> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY >> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is >> org.postgresql.util.PSQLException: The column name disk_storage_type was >> not found in this ResultSet.: >> org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL >> grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id >> IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM >> vds_groups_storage_domain LEFT OUTER JOIN >> storage_pool_with_storage_domain ON >> vds_groups_storage_domain.storage_pool_id= >> storage_pool_with_storage_domain.idWHERE >> storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY >> name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is >> org.postgresql.util.PSQLException: The column name disk_storage_type was >> not found in this ResultSet. >> >> This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5. >> There's also a 3.5 cluster managed by the same engine. Could this be >> related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>? >> >> >> >> On 17 December 2014 at 12:34, Yeela Kaplan wrote: >> >>> Just another thought, >>> from looking at the vdsm logs it looks like there are too many calls to >>> getVolumeSize that are eating up all handlers >>> and eventually a timeout occurs. >>> Adam, do you have any idea about this? >>> >>> - Original Message - >>> > From: "Yeela Kaplan" >>> > To: "Tiemen Ruiten" >>> > Cc: "Users@ovirt.org" >>> > Sent: Wednesday, December 17, 2014 1:30:52 PM >>> > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while >>> creating VM from template >>> > >>> > >>> > >>> > - Original Message - >>> > > From: "Tiemen Ruiten" >>> > > To: "Yeela Kaplan" >>> > > Cc: "Users@ovirt.org" >>> > > Sent: Wednesday, December 17, 2014 1:22:59 PM >>> > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' >>> while >>> > > creating VM from template >>> > > >>> > > Thank you, I will try to increase to 20 and see what happens. Bug is >>> filed: >>> > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255 >>> > >>> > Thank you >>> > and I forgot to mention that you have to restart vdsmd so the change >>> will >>> > apply... >>> > >>> > >>> > > >>> > > On 17 December 2014 at 11:48, Yeela Kaplan >>> wrote: >>> > > > >>> > &
Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Sorry, to be clear: no default disk policy exists for the 3.4 cluster storage domain. On 5 January 2015 at 15:45, Tiemen Ruiten wrote: > Hello, > > I just tried creating a thin-provisioned disk and the same problem (load > spike, non responsive VMs) occurred. I may have an idea what is going wrong > now. The following is an excerpt from the engine log after starting the > disk create job: > > 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery] > (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is > StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM > vds_groups_view WHERE ( vds_group_id IN (SELECT distinct > vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain > LEFT OUTER JOIN storage_pool_with_storage_domain ON > vds_groups_storage_domain.storage_pool_id= > storage_pool_with_storage_domain.idWHERE > storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY > name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is > org.postgresql.util.PSQLException: The column name disk_storage_type was > not found in this ResultSet. : > org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL > grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id > IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM > vds_groups_storage_domain LEFT OUTER JOIN > storage_pool_with_storage_domain ON > vds_groups_storage_domain.storage_pool_id= > storage_pool_with_storage_domain.idWHERE > storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY > name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is > org.postgresql.util.PSQLException: The column name disk_storage_type was > not found in this ResultSet.: > org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL > grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id > IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM > vds_groups_storage_domain LEFT OUTER JOIN > storage_pool_with_storage_domain ON > vds_groups_storage_domain.storage_pool_id= > storage_pool_with_storage_domain.idWHERE > storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY > name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is > org.postgresql.util.PSQLException: The column name disk_storage_type was > not found in this ResultSet. > > This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5. > There's also a 3.5 cluster managed by the same engine. Could this be > related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>? > > > > On 17 December 2014 at 12:34, Yeela Kaplan wrote: > >> Just another thought, >> from looking at the vdsm logs it looks like there are too many calls to >> getVolumeSize that are eating up all handlers >> and eventually a timeout occurs. >> Adam, do you have any idea about this? >> >> ----- Original Message - >> > From: "Yeela Kaplan" >> > To: "Tiemen Ruiten" >> > Cc: "Users@ovirt.org" >> > Sent: Wednesday, December 17, 2014 1:30:52 PM >> > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while >> creating VM from template >> > >> > >> > >> > - Original Message - >> > > From: "Tiemen Ruiten" >> > > To: "Yeela Kaplan" >> > > Cc: "Users@ovirt.org" >> > > Sent: Wednesday, December 17, 2014 1:22:59 PM >> > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while >> > > creating VM from template >> > > >> > > Thank you, I will try to increase to 20 and see what happens. Bug is >> filed: >> > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255 >> > >> > Thank you >> > and I forgot to mention that you have to restart vdsmd so the change >> will >> > apply... >> > >> > >> > > >> > > On 17 December 2014 at 11:48, Yeela Kaplan >> wrote: >> > > > >> > > > Hi, >> > > > You are right, the problem is with the file handlers. >> > > > You can increase the number of handlers in pool using the vdsm >> config >> > > > file, which is supposed to be under the following directory in your >> > > > system: >> > > > >> > > > /usr/lib/python2.6/site-packages/vdsm/config.py >> > > > >> > > > The default value
Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Hello, I just tried creating a thin-provisioned disk and the same problem (load spike, non responsive VMs) occurred. I may have an idea what is going wrong now. The following is an excerpt from the engine log after starting the disk create job: 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery] (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.idWHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet. : org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.idWHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet.: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM vds_groups_storage_domain LEFT OUTER JOIN storage_pool_with_storage_domain ON vds_groups_storage_domain.storage_pool_id= storage_pool_with_storage_domain.idWHERE storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' )) ORDER BY name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is org.postgresql.util.PSQLException: The column name disk_storage_type was not found in this ResultSet. This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5. There's also a 3.5 cluster managed by the same engine. Could this be related to this BZ <https://bugzilla.redhat.com/show_bug.cgi?id=1145241>? On 17 December 2014 at 12:34, Yeela Kaplan wrote: > Just another thought, > from looking at the vdsm logs it looks like there are too many calls to > getVolumeSize that are eating up all handlers > and eventually a timeout occurs. > Adam, do you have any idea about this? > > - Original Message - > > From: "Yeela Kaplan" > > To: "Tiemen Ruiten" > > Cc: "Users@ovirt.org" > > Sent: Wednesday, December 17, 2014 1:30:52 PM > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > > > > > > > - Original Message ----- > > > From: "Tiemen Ruiten" > > > To: "Yeela Kaplan" > > > Cc: "Users@ovirt.org" > > > Sent: Wednesday, December 17, 2014 1:22:59 PM > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > > creating VM from template > > > > > > Thank you, I will try to increase to 20 and see what happens. Bug is > filed: > > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255 > > > > Thank you > > and I forgot to mention that you have to restart vdsmd so the change will > > apply... > > > > > > > > > > On 17 December 2014 at 11:48, Yeela Kaplan wrote: > > > > > > > > Hi, > > > > You are right, the problem is with the file handlers. > > > > You can increase the number of handlers in pool using the vdsm config > > > > file, which is supposed to be under the following directory in your > > > > system: > > > > > > > > /usr/lib/python2.6/site-packages/vdsm/config.py > > > > > > > > The default value for 'process_pool_max_slots_per_domain' is 10, so > you > > > > can increase it by a bit, but not too much. > > > > > > > > But I suspect the problem is in a larger scale, and this is only a > > > > temporary relief for your system and this needs much more attention > and a > > > > proper fix. > > > > could you please open a bug on RHEV/vdsm in bugzilla stating all of > the > > > > details of your setup and logs? > > > > > > > > thanks, > > > > Yeela > > > &g
Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Just another thought, from looking at the vdsm logs it looks like there are too many calls to getVolumeSize that are eating up all handlers and eventually a timeout occurs. Adam, do you have any idea about this? - Original Message - > From: "Yeela Kaplan" > To: "Tiemen Ruiten" > Cc: "Users@ovirt.org" > Sent: Wednesday, December 17, 2014 1:30:52 PM > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > > > - Original Message - > > From: "Tiemen Ruiten" > > To: "Yeela Kaplan" > > Cc: "Users@ovirt.org" > > Sent: Wednesday, December 17, 2014 1:22:59 PM > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > creating VM from template > > > > Thank you, I will try to increase to 20 and see what happens. Bug is filed: > > https://bugzilla.redhat.com/show_bug.cgi?id=1175255 > > Thank you > and I forgot to mention that you have to restart vdsmd so the change will > apply... > > > > > > On 17 December 2014 at 11:48, Yeela Kaplan wrote: > > > > > > Hi, > > > You are right, the problem is with the file handlers. > > > You can increase the number of handlers in pool using the vdsm config > > > file, which is supposed to be under the following directory in your > > > system: > > > > > > /usr/lib/python2.6/site-packages/vdsm/config.py > > > > > > The default value for 'process_pool_max_slots_per_domain' is 10, so you > > > can increase it by a bit, but not too much. > > > > > > But I suspect the problem is in a larger scale, and this is only a > > > temporary relief for your system and this needs much more attention and a > > > proper fix. > > > could you please open a bug on RHEV/vdsm in bugzilla stating all of the > > > details of your setup and logs? > > > > > > thanks, > > > Yeela > > > > > > - Original Message - > > > > From: "Tiemen Ruiten" > > > > To: "Users@ovirt.org" > > > > Sent: Wednesday, December 17, 2014 10:53:39 AM > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > > creating VM from template > > > > > > > > Would this be limits for the vdsm process? Then what is the proper way > > > > to > > > > change ulimits for VDSM? > > > > > > > > On 16 December 2014 at 20:45, Donny Davis < do...@cloudspin.me > wrote: > > > > > > > > > > > > > > > > > > > > The only thing I can think of would be file hard and soft limits, but I > > > am no > > > > oVirt pro. > > > > > > > > > > > > > > > > 'no free file handlers in pool' that would make sense to me… > > > > > > > > Donny > > > > > > > > > > > > > > > > > > > > > > > > From: Tiemen Ruiten [mailto: t.rui...@rdmedia.com ] > > > > Sent: Tuesday, December 16, 2014 12:40 PM > > > > To: Donny Davis > > > > Cc: Users@ovirt.org > > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > > > creating VM from template > > > > > > > > > > > > > > > > > > > > > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 > > > > hyperthreaded cores and 64 GB of RAM each. At the moment I created the > > > VM, > > > > processor load on both nodes was less than 1. > > > > > > > > > > > > > > > > > > > > > > > > On 16 December 2014 at 20:29, Donny Davis < do...@cloudspin.me > wrote: > > > > > > > > What is the VM load you are running on your servers? > > > > > > > > > > > > > > > > -Original Message- > > > > From: users-boun...@ovirt.org [mailto: users-boun...@ovirt.org ] On > > > Behalf Of > > > > Tiemen Ruiten > > > > Sent: Tuesday, December 16, 2014 12:27 PM > > > > To: users@ovirt.org > > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while > > > creating > > > > VM from template > > > > > > > > Hello, > > > > > > > > I
Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
- Original Message - > From: "Tiemen Ruiten" > To: "Yeela Kaplan" > Cc: "Users@ovirt.org" > Sent: Wednesday, December 17, 2014 1:22:59 PM > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > Thank you, I will try to increase to 20 and see what happens. Bug is filed: > https://bugzilla.redhat.com/show_bug.cgi?id=1175255 Thank you and I forgot to mention that you have to restart vdsmd so the change will apply... > > On 17 December 2014 at 11:48, Yeela Kaplan wrote: > > > > Hi, > > You are right, the problem is with the file handlers. > > You can increase the number of handlers in pool using the vdsm config > > file, which is supposed to be under the following directory in your system: > > > > /usr/lib/python2.6/site-packages/vdsm/config.py > > > > The default value for 'process_pool_max_slots_per_domain' is 10, so you > > can increase it by a bit, but not too much. > > > > But I suspect the problem is in a larger scale, and this is only a > > temporary relief for your system and this needs much more attention and a > > proper fix. > > could you please open a bug on RHEV/vdsm in bugzilla stating all of the > > details of your setup and logs? > > > > thanks, > > Yeela > > > > ----- Original Message - > > > From: "Tiemen Ruiten" > > > To: "Users@ovirt.org" > > > Sent: Wednesday, December 17, 2014 10:53:39 AM > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > creating VM from template > > > > > > Would this be limits for the vdsm process? Then what is the proper way to > > > change ulimits for VDSM? > > > > > > On 16 December 2014 at 20:45, Donny Davis < do...@cloudspin.me > wrote: > > > > > > > > > > > > > > > The only thing I can think of would be file hard and soft limits, but I > > am no > > > oVirt pro. > > > > > > > > > > > > 'no free file handlers in pool' that would make sense to me… > > > > > > Donny > > > > > > > > > > > > > > > > > > From: Tiemen Ruiten [mailto: t.rui...@rdmedia.com ] > > > Sent: Tuesday, December 16, 2014 12:40 PM > > > To: Donny Davis > > > Cc: Users@ovirt.org > > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > > creating VM from template > > > > > > > > > > > > > > > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 > > > hyperthreaded cores and 64 GB of RAM each. At the moment I created the > > VM, > > > processor load on both nodes was less than 1. > > > > > > > > > > > > > > > > > > On 16 December 2014 at 20:29, Donny Davis < do...@cloudspin.me > wrote: > > > > > > What is the VM load you are running on your servers? > > > > > > > > > > > > -Original Message- > > > From: users-boun...@ovirt.org [mailto: users-boun...@ovirt.org ] On > > Behalf Of > > > Tiemen Ruiten > > > Sent: Tuesday, December 16, 2014 12:27 PM > > > To: users@ovirt.org > > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while > > creating > > > VM from template > > > > > > Hello, > > > > > > I ran into a nasty problem today when creating a new, cloned VM from a > > > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the > > node > > > where I started a VM creation job, load skyrocketed and some VMs stopped > > > responding until and after the job failed. Everything recovered without > > > intervention, but this obviously shouldn't happen. I have attached the > > > relevant vdsm log file. The button to create the VM was pressed around > > > 11:17, the first error in the vdsm log is at 11:23:58. > > > > > > The ISO domain is a gluster volume exposed via NFS, the storage domain > > for > > > the VM's is also a gluster volume. The underlying filesystem is ZFS. > > > The hypervisor nodes are full CentOS 6 installs. > > > > > > I'm guessing the 'no free file handlers in pool' in the vdsm log file is > > key > > > here. What can I do to prevent this from happening again? Apart from not > > > creating new VMs of course :) > > > > > > Tiemen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Tiemen Ruiten > > > Systems Engineer > > > R&D Media > > > > > > > > > -- > > > Tiemen Ruiten > > > Systems Engineer > > > R&D Media > > > > > > ___ > > > Users mailing list > > > Users@ovirt.org > > > http://lists.ovirt.org/mailman/listinfo/users > > > > > > > > -- > Tiemen Ruiten > Systems Engineer > R&D Media > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Thank you, I will try to increase to 20 and see what happens. Bug is filed: https://bugzilla.redhat.com/show_bug.cgi?id=1175255 On 17 December 2014 at 11:48, Yeela Kaplan wrote: > > Hi, > You are right, the problem is with the file handlers. > You can increase the number of handlers in pool using the vdsm config > file, which is supposed to be under the following directory in your system: > > /usr/lib/python2.6/site-packages/vdsm/config.py > > The default value for 'process_pool_max_slots_per_domain' is 10, so you > can increase it by a bit, but not too much. > > But I suspect the problem is in a larger scale, and this is only a > temporary relief for your system and this needs much more attention and a > proper fix. > could you please open a bug on RHEV/vdsm in bugzilla stating all of the > details of your setup and logs? > > thanks, > Yeela > > - Original Message - > > From: "Tiemen Ruiten" > > To: "Users@ovirt.org" > > Sent: Wednesday, December 17, 2014 10:53:39 AM > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > > > Would this be limits for the vdsm process? Then what is the proper way to > > change ulimits for VDSM? > > > > On 16 December 2014 at 20:45, Donny Davis < do...@cloudspin.me > wrote: > > > > > > > > > > The only thing I can think of would be file hard and soft limits, but I > am no > > oVirt pro. > > > > > > > > 'no free file handlers in pool' that would make sense to me… > > > > Donny > > > > > > > > > > > > From: Tiemen Ruiten [mailto: t.rui...@rdmedia.com ] > > Sent: Tuesday, December 16, 2014 12:40 PM > > To: Donny Davis > > Cc: Users@ovirt.org > > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > > creating VM from template > > > > > > > > > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 > > hyperthreaded cores and 64 GB of RAM each. At the moment I created the > VM, > > processor load on both nodes was less than 1. > > > > > > > > > > > > On 16 December 2014 at 20:29, Donny Davis < do...@cloudspin.me > wrote: > > > > What is the VM load you are running on your servers? > > > > > > > > -Original Message- > > From: users-boun...@ovirt.org [mailto: users-boun...@ovirt.org ] On > Behalf Of > > Tiemen Ruiten > > Sent: Tuesday, December 16, 2014 12:27 PM > > To: users@ovirt.org > > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while > creating > > VM from template > > > > Hello, > > > > I ran into a nasty problem today when creating a new, cloned VM from a > > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the > node > > where I started a VM creation job, load skyrocketed and some VMs stopped > > responding until and after the job failed. Everything recovered without > > intervention, but this obviously shouldn't happen. I have attached the > > relevant vdsm log file. The button to create the VM was pressed around > > 11:17, the first error in the vdsm log is at 11:23:58. > > > > The ISO domain is a gluster volume exposed via NFS, the storage domain > for > > the VM's is also a gluster volume. The underlying filesystem is ZFS. > > The hypervisor nodes are full CentOS 6 installs. > > > > I'm guessing the 'no free file handlers in pool' in the vdsm log file is > key > > here. What can I do to prevent this from happening again? Apart from not > > creating new VMs of course :) > > > > Tiemen > > > > > > > > > > > > > > > > > > > > -- > > > > > > Tiemen Ruiten > > Systems Engineer > > R&D Media > > > > > > -- > > Tiemen Ruiten > > Systems Engineer > > R&D Media > > > > ___ > > Users mailing list > > Users@ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users > > > -- Tiemen Ruiten Systems Engineer R&D Media ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Hi, You are right, the problem is with the file handlers. You can increase the number of handlers in pool using the vdsm config file, which is supposed to be under the following directory in your system: /usr/lib/python2.6/site-packages/vdsm/config.py The default value for 'process_pool_max_slots_per_domain' is 10, so you can increase it by a bit, but not too much. But I suspect the problem is in a larger scale, and this is only a temporary relief for your system and this needs much more attention and a proper fix. could you please open a bug on RHEV/vdsm in bugzilla stating all of the details of your setup and logs? thanks, Yeela - Original Message - > From: "Tiemen Ruiten" > To: "Users@ovirt.org" > Sent: Wednesday, December 17, 2014 10:53:39 AM > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > Would this be limits for the vdsm process? Then what is the proper way to > change ulimits for VDSM? > > On 16 December 2014 at 20:45, Donny Davis < do...@cloudspin.me > wrote: > > > > > The only thing I can think of would be file hard and soft limits, but I am no > oVirt pro. > > > > 'no free file handlers in pool' that would make sense to me… > > Donny > > > > > > From: Tiemen Ruiten [mailto: t.rui...@rdmedia.com ] > Sent: Tuesday, December 16, 2014 12:40 PM > To: Donny Davis > Cc: Users@ovirt.org > Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 > hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, > processor load on both nodes was less than 1. > > > > > > On 16 December 2014 at 20:29, Donny Davis < do...@cloudspin.me > wrote: > > What is the VM load you are running on your servers? > > > > -----Original Message- > From: users-boun...@ovirt.org [mailto: users-boun...@ovirt.org ] On Behalf Of > Tiemen Ruiten > Sent: Tuesday, December 16, 2014 12:27 PM > To: users@ovirt.org > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating > VM from template > > Hello, > > I ran into a nasty problem today when creating a new, cloned VM from a > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node > where I started a VM creation job, load skyrocketed and some VMs stopped > responding until and after the job failed. Everything recovered without > intervention, but this obviously shouldn't happen. I have attached the > relevant vdsm log file. The button to create the VM was pressed around > 11:17, the first error in the vdsm log is at 11:23:58. > > The ISO domain is a gluster volume exposed via NFS, the storage domain for > the VM's is also a gluster volume. The underlying filesystem is ZFS. > The hypervisor nodes are full CentOS 6 installs. > > I'm guessing the 'no free file handlers in pool' in the vdsm log file is key > here. What can I do to prevent this from happening again? Apart from not > creating new VMs of course :) > > Tiemen > > > > > > > > > > -- > > > Tiemen Ruiten > Systems Engineer > R&D Media > > > -- > Tiemen Ruiten > Systems Engineer > R&D Media > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
Would this be limits for the vdsm process? Then what is the proper way to change ulimits for VDSM? On 16 December 2014 at 20:45, Donny Davis wrote: > > The only thing I can think of would be file hard and soft limits, but I am > no oVirt pro. > > > > 'no free file handlers in pool' that would make sense to me… > > Donny > > > > > > *From:* Tiemen Ruiten [mailto:t.rui...@rdmedia.com] > *Sent:* Tuesday, December 16, 2014 12:40 PM > *To:* Donny Davis > *Cc:* Users@ovirt.org > *Subject:* Re: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > > > About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 > hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, > processor load on both nodes was less than 1. > > > > On 16 December 2014 at 20:29, Donny Davis wrote: > > What is the VM load you are running on your servers? > > > -Original Message- > From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf > Of Tiemen Ruiten > Sent: Tuesday, December 16, 2014 12:27 PM > To: users@ovirt.org > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > Hello, > > I ran into a nasty problem today when creating a new, cloned VM from a > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node > where I started a VM creation job, load skyrocketed and some VMs stopped > responding until and after the job failed. Everything recovered without > intervention, but this obviously shouldn't happen. I have attached the > relevant vdsm log file. The button to create the VM was pressed around > 11:17, the first error in the vdsm log is at 11:23:58. > > The ISO domain is a gluster volume exposed via NFS, the storage domain for > the VM's is also a gluster volume. The underlying filesystem is ZFS. > The hypervisor nodes are full CentOS 6 installs. > > I'm guessing the 'no free file handlers in pool' in the vdsm log file is > key here. What can I do to prevent this from happening again? Apart from > not creating new VMs of course :) > > Tiemen > > > > > -- > > Tiemen Ruiten > Systems Engineer > R&D Media > -- Tiemen Ruiten Systems Engineer R&D Media ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
The only thing I can think of would be file hard and soft limits, but I am no oVirt pro. 'no free file handlers in pool' that would make sense to me… Donny From: Tiemen Ruiten [mailto:t.rui...@rdmedia.com] Sent: Tuesday, December 16, 2014 12:40 PM To: Donny Davis Cc: Users@ovirt.org Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, processor load on both nodes was less than 1. On 16 December 2014 at 20:29, Donny Davis wrote: What is the VM load you are running on your servers? -Original Message- From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users@ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template Hello, I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58. The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs. I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :) Tiemen -- Tiemen Ruiten Systems Engineer R&D Media ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, processor load on both nodes was less than 1. On 16 December 2014 at 20:29, Donny Davis wrote: > > What is the VM load you are running on your servers? > > -Original Message- > From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf > Of Tiemen Ruiten > Sent: Tuesday, December 16, 2014 12:27 PM > To: users@ovirt.org > Subject: [ovirt-users] ERROR 'no free file handlers in pool' while > creating VM from template > > Hello, > > I ran into a nasty problem today when creating a new, cloned VM from a > template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node > where I started a VM creation job, load skyrocketed and some VMs stopped > responding until and after the job failed. Everything recovered without > intervention, but this obviously shouldn't happen. I have attached the > relevant vdsm log file. The button to create the VM was pressed around > 11:17, the first error in the vdsm log is at 11:23:58. > > The ISO domain is a gluster volume exposed via NFS, the storage domain for > the VM's is also a gluster volume. The underlying filesystem is ZFS. > The hypervisor nodes are full CentOS 6 installs. > > I'm guessing the 'no free file handlers in pool' in the vdsm log file is > key here. What can I do to prevent this from happening again? Apart from > not creating new VMs of course :) > > Tiemen > > > -- Tiemen Ruiten Systems Engineer R&D Media ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template
What is the VM load you are running on your servers? -Original Message- From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf Of Tiemen Ruiten Sent: Tuesday, December 16, 2014 12:27 PM To: users@ovirt.org Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template Hello, I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58. The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs. I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :) Tiemen ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
