Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

2015-01-06 Thread Tiemen Ruiten
Yeela, any chance you could shed any light on this?

On 5 January 2015 at 15:46, Tiemen Ruiten t.rui...@rdmedia.com wrote:

 Sorry, to be clear: no default disk policy exists for the 3.4 cluster
 storage domain.

 On 5 January 2015 at 15:45, Tiemen Ruiten t.rui...@rdmedia.com wrote:

 Hello,

 I just tried creating a thin-provisioned disk and the same problem (load
 spike, non responsive VMs) occurred. I may have an idea what is going wrong
 now. The following is an excerpt from the engine log after starting the
 disk create job:

 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery]
 (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is
 StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM
 vds_groups_view WHERE ( vds_group_id IN (SELECT distinct
 vds_groups_storage_domain.vds_group_id FROM  vds_groups_storage_domain
 LEFT OUTER JOIN storage_pool_with_storage_domain ON
 vds_groups_storage_domain.storage_pool_id=
 storage_pool_with_storage_domain.idWHERE
 storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
 name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
 org.postgresql.util.PSQLException: The column name disk_storage_type was
 not found in this ResultSet. :
 org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
 grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
 IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
  vds_groups_storage_domain   LEFT OUTER JOIN
 storage_pool_with_storage_domain ON
 vds_groups_storage_domain.storage_pool_id=
 storage_pool_with_storage_domain.idWHERE
 storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
 name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
 org.postgresql.util.PSQLException: The column name disk_storage_type was
 not found in this ResultSet.:
 org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
 grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
 IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
  vds_groups_storage_domain   LEFT OUTER JOIN
 storage_pool_with_storage_domain ON
 vds_groups_storage_domain.storage_pool_id=
 storage_pool_with_storage_domain.idWHERE
 storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
 name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
 org.postgresql.util.PSQLException: The column name disk_storage_type was
 not found in this ResultSet.

 This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5.
 There's also a 3.5 cluster managed by the same engine. Could this be
 related to this BZ https://bugzilla.redhat.com/show_bug.cgi?id=1145241?



 On 17 December 2014 at 12:34, Yeela Kaplan ykap...@redhat.com wrote:

 Just another thought,
 from looking at the vdsm logs it looks like there are too many calls to
 getVolumeSize that are eating up all handlers
 and eventually a timeout occurs.
 Adam, do you have any idea about this?

 - Original Message -
  From: Yeela Kaplan ykap...@redhat.com
  To: Tiemen Ruiten t.rui...@rdmedia.com
  Cc: Users@ovirt.org users@ovirt.org
  Sent: Wednesday, December 17, 2014 1:30:52 PM
  Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
 creating VM from template
 
 
 
  - Original Message -
   From: Tiemen Ruiten t.rui...@rdmedia.com
   To: Yeela Kaplan ykap...@redhat.com
   Cc: Users@ovirt.org users@ovirt.org
   Sent: Wednesday, December 17, 2014 1:22:59 PM
   Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool'
 while
   creating VM from template
  
   Thank you, I will try to increase to 20 and see what happens. Bug is
 filed:
   https://bugzilla.redhat.com/show_bug.cgi?id=1175255
 
  Thank you
  and I forgot to mention that you have to restart vdsmd so the change
 will
  apply...
 
 
  
   On 17 December 2014 at 11:48, Yeela Kaplan ykap...@redhat.com
 wrote:
   
Hi,
You are right, the problem is with the file handlers.
You can increase the number of handlers in pool using the vdsm
 config
file, which is supposed to be under the following directory in your
system:
   
/usr/lib/python2.6/site-packages/vdsm/config.py
   
The default value for 'process_pool_max_slots_per_domain' is 10,
 so you
can increase it by a bit, but not too much.
   
But I suspect the problem is in a larger scale, and this is only a
temporary relief for your system and this needs much more
 attention and a
proper fix.
could you please open a bug on RHEV/vdsm in bugzilla stating all
 of the
details of your setup and logs?
   
thanks,
Yeela
   
- Original Message -
 From: Tiemen Ruiten t.rui...@rdmedia.com
 To: Users@ovirt.org users@ovirt.org
 Sent: Wednesday, December 17, 2014 10:53:39 AM
 Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool'
 while
creating VM from template

 Would

Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

2015-01-06 Thread Tiemen Ruiten
After creating a default disk profile, I can add new disks and create VMs
without issues. I have updated the bug I reported as a duplicate of 1145241.

https://bugzilla.redhat.com/show_bug.cgi?id=1175255


On 5 January 2015 at 15:46, Tiemen Ruiten t.rui...@rdmedia.com wrote:

 Sorry, to be clear: no default disk policy exists for the 3.4 cluster
 storage domain.

 On 5 January 2015 at 15:45, Tiemen Ruiten t.rui...@rdmedia.com wrote:

 Hello,

 I just tried creating a thin-provisioned disk and the same problem (load
 spike, non responsive VMs) occurred. I may have an idea what is going wrong
 now. The following is an excerpt from the engine log after starting the
 disk create job:

 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery]
 (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is
 StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM
 vds_groups_view WHERE ( vds_group_id IN (SELECT distinct
 vds_groups_storage_domain.vds_group_id FROM  vds_groups_storage_domain
 LEFT OUTER JOIN storage_pool_with_storage_domain ON
 vds_groups_storage_domain.storage_pool_id=
 storage_pool_with_storage_domain.idWHERE
 storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
 name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
 org.postgresql.util.PSQLException: The column name disk_storage_type was
 not found in this ResultSet. :
 org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
 grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
 IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
  vds_groups_storage_domain   LEFT OUTER JOIN
 storage_pool_with_storage_domain ON
 vds_groups_storage_domain.storage_pool_id=
 storage_pool_with_storage_domain.idWHERE
 storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
 name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
 org.postgresql.util.PSQLException: The column name disk_storage_type was
 not found in this ResultSet.:
 org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
 grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
 IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
  vds_groups_storage_domain   LEFT OUTER JOIN
 storage_pool_with_storage_domain ON
 vds_groups_storage_domain.storage_pool_id=
 storage_pool_with_storage_domain.idWHERE
 storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
 name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
 org.postgresql.util.PSQLException: The column name disk_storage_type was
 not found in this ResultSet.

 This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5.
 There's also a 3.5 cluster managed by the same engine. Could this be
 related to this BZ https://bugzilla.redhat.com/show_bug.cgi?id=1145241?



 On 17 December 2014 at 12:34, Yeela Kaplan ykap...@redhat.com wrote:

 Just another thought,
 from looking at the vdsm logs it looks like there are too many calls to
 getVolumeSize that are eating up all handlers
 and eventually a timeout occurs.
 Adam, do you have any idea about this?

 - Original Message -
  From: Yeela Kaplan ykap...@redhat.com
  To: Tiemen Ruiten t.rui...@rdmedia.com
  Cc: Users@ovirt.org users@ovirt.org
  Sent: Wednesday, December 17, 2014 1:30:52 PM
  Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
 creating VM from template
 
 
 
  - Original Message -
   From: Tiemen Ruiten t.rui...@rdmedia.com
   To: Yeela Kaplan ykap...@redhat.com
   Cc: Users@ovirt.org users@ovirt.org
   Sent: Wednesday, December 17, 2014 1:22:59 PM
   Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool'
 while
   creating VM from template
  
   Thank you, I will try to increase to 20 and see what happens. Bug is
 filed:
   https://bugzilla.redhat.com/show_bug.cgi?id=1175255
 
  Thank you
  and I forgot to mention that you have to restart vdsmd so the change
 will
  apply...
 
 
  
   On 17 December 2014 at 11:48, Yeela Kaplan ykap...@redhat.com
 wrote:
   
Hi,
You are right, the problem is with the file handlers.
You can increase the number of handlers in pool using the vdsm
 config
file, which is supposed to be under the following directory in your
system:
   
/usr/lib/python2.6/site-packages/vdsm/config.py
   
The default value for 'process_pool_max_slots_per_domain' is 10,
 so you
can increase it by a bit, but not too much.
   
But I suspect the problem is in a larger scale, and this is only a
temporary relief for your system and this needs much more
 attention and a
proper fix.
could you please open a bug on RHEV/vdsm in bugzilla stating all
 of the
details of your setup and logs?
   
thanks,
Yeela
   
- Original Message -
 From: Tiemen Ruiten t.rui...@rdmedia.com
 To: Users@ovirt.org users@ovirt.org
 Sent: Wednesday, December

Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

2015-01-05 Thread Tiemen Ruiten
Hello,

I just tried creating a thin-provisioned disk and the same problem (load
spike, non responsive VMs) occurred. I may have an idea what is going wrong
now. The following is an excerpt from the engine log after starting the
disk create job:

2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery]
(ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is
StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM
vds_groups_view WHERE ( vds_group_id IN (SELECT distinct
vds_groups_storage_domain.vds_group_id FROM  vds_groups_storage_domain
LEFT OUTER JOIN storage_pool_with_storage_domain ON
vds_groups_storage_domain.storage_pool_id=
storage_pool_with_storage_domain.idWHERE
storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
org.postgresql.util.PSQLException: The column name disk_storage_type was
not found in this ResultSet. :
org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
 vds_groups_storage_domain   LEFT OUTER JOIN
storage_pool_with_storage_domain ON
vds_groups_storage_domain.storage_pool_id=
storage_pool_with_storage_domain.idWHERE
storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
org.postgresql.util.PSQLException: The column name disk_storage_type was
not found in this ResultSet.:
org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
 vds_groups_storage_domain   LEFT OUTER JOIN
storage_pool_with_storage_domain ON
vds_groups_storage_domain.storage_pool_id=
storage_pool_with_storage_domain.idWHERE
storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
org.postgresql.util.PSQLException: The column name disk_storage_type was
not found in this ResultSet.

This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5.
There's also a 3.5 cluster managed by the same engine. Could this be
related to this BZ https://bugzilla.redhat.com/show_bug.cgi?id=1145241?



On 17 December 2014 at 12:34, Yeela Kaplan ykap...@redhat.com wrote:

 Just another thought,
 from looking at the vdsm logs it looks like there are too many calls to
 getVolumeSize that are eating up all handlers
 and eventually a timeout occurs.
 Adam, do you have any idea about this?

 - Original Message -
  From: Yeela Kaplan ykap...@redhat.com
  To: Tiemen Ruiten t.rui...@rdmedia.com
  Cc: Users@ovirt.org users@ovirt.org
  Sent: Wednesday, December 17, 2014 1:30:52 PM
  Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
 creating VM from template
 
 
 
  - Original Message -
   From: Tiemen Ruiten t.rui...@rdmedia.com
   To: Yeela Kaplan ykap...@redhat.com
   Cc: Users@ovirt.org users@ovirt.org
   Sent: Wednesday, December 17, 2014 1:22:59 PM
   Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
   creating VM from template
  
   Thank you, I will try to increase to 20 and see what happens. Bug is
 filed:
   https://bugzilla.redhat.com/show_bug.cgi?id=1175255
 
  Thank you
  and I forgot to mention that you have to restart vdsmd so the change will
  apply...
 
 
  
   On 17 December 2014 at 11:48, Yeela Kaplan ykap...@redhat.com wrote:
   
Hi,
You are right, the problem is with the file handlers.
You can increase the number of handlers in pool using the vdsm config
file, which is supposed to be under the following directory in your
system:
   
/usr/lib/python2.6/site-packages/vdsm/config.py
   
The default value for 'process_pool_max_slots_per_domain' is 10, so
 you
can increase it by a bit, but not too much.
   
But I suspect the problem is in a larger scale, and this is only a
temporary relief for your system and this needs much more attention
 and a
proper fix.
could you please open a bug on RHEV/vdsm in bugzilla stating all of
 the
details of your setup and logs?
   
thanks,
Yeela
   
- Original Message -
 From: Tiemen Ruiten t.rui...@rdmedia.com
 To: Users@ovirt.org users@ovirt.org
 Sent: Wednesday, December 17, 2014 10:53:39 AM
 Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool'
 while
creating VM from template

 Would this be limits for the vdsm process? Then what is the proper
 way
 to
 change ulimits for VDSM?

 On 16 December 2014 at 20:45, Donny Davis  do...@cloudspin.me 
 wrote:




 The only thing I can think of would be file hard and soft limits,
 but I
am no
 oVirt pro.



 'no free file

Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

2015-01-05 Thread Tiemen Ruiten
Sorry, to be clear: no default disk policy exists for the 3.4 cluster
storage domain.

On 5 January 2015 at 15:45, Tiemen Ruiten t.rui...@rdmedia.com wrote:

 Hello,

 I just tried creating a thin-provisioned disk and the same problem (load
 spike, non responsive VMs) occurred. I may have an idea what is going wrong
 now. The following is an excerpt from the engine log after starting the
 disk create job:

 2015-01-05 15:12:14,277 ERROR [org.ovirt.engine.core.bll.SearchQuery]
 (ajp--127.0.0.1-8702-1) Query SearchQuery failed. Exception message is
 StatementCallback; bad SQL grammar [SELECT * FROM (SELECT * FROM
 vds_groups_view WHERE ( vds_group_id IN (SELECT distinct
 vds_groups_storage_domain.vds_group_id FROM  vds_groups_storage_domain
 LEFT OUTER JOIN storage_pool_with_storage_domain ON
 vds_groups_storage_domain.storage_pool_id=
 storage_pool_with_storage_domain.idWHERE
 storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
 name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
 org.postgresql.util.PSQLException: The column name disk_storage_type was
 not found in this ResultSet. :
 org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
 grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
 IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
  vds_groups_storage_domain   LEFT OUTER JOIN
 storage_pool_with_storage_domain ON
 vds_groups_storage_domain.storage_pool_id=
 storage_pool_with_storage_domain.idWHERE
 storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
 name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
 org.postgresql.util.PSQLException: The column name disk_storage_type was
 not found in this ResultSet.:
 org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
 grammar [SELECT * FROM (SELECT * FROM vds_groups_view WHERE ( vds_group_id
 IN (SELECT distinct vds_groups_storage_domain.vds_group_id FROM
  vds_groups_storage_domain   LEFT OUTER JOIN
 storage_pool_with_storage_domain ON
 vds_groups_storage_domain.storage_pool_id=
 storage_pool_with_storage_domain.idWHERE
 storage_pool_with_storage_domain.name LIKE 'XS4ALL-Strozzi' ))  ORDER BY
 name ASC) as T1 OFFSET (1 -1) LIMIT 100]; nested exception is
 org.postgresql.util.PSQLException: The column name disk_storage_type was
 not found in this ResultSet.

 This cluster happens to be a oVirt 3.4 cluster, while the engine is 3.5.
 There's also a 3.5 cluster managed by the same engine. Could this be
 related to this BZ https://bugzilla.redhat.com/show_bug.cgi?id=1145241?



 On 17 December 2014 at 12:34, Yeela Kaplan ykap...@redhat.com wrote:

 Just another thought,
 from looking at the vdsm logs it looks like there are too many calls to
 getVolumeSize that are eating up all handlers
 and eventually a timeout occurs.
 Adam, do you have any idea about this?

 - Original Message -
  From: Yeela Kaplan ykap...@redhat.com
  To: Tiemen Ruiten t.rui...@rdmedia.com
  Cc: Users@ovirt.org users@ovirt.org
  Sent: Wednesday, December 17, 2014 1:30:52 PM
  Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
 creating VM from template
 
 
 
  - Original Message -
   From: Tiemen Ruiten t.rui...@rdmedia.com
   To: Yeela Kaplan ykap...@redhat.com
   Cc: Users@ovirt.org users@ovirt.org
   Sent: Wednesday, December 17, 2014 1:22:59 PM
   Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
   creating VM from template
  
   Thank you, I will try to increase to 20 and see what happens. Bug is
 filed:
   https://bugzilla.redhat.com/show_bug.cgi?id=1175255
 
  Thank you
  and I forgot to mention that you have to restart vdsmd so the change
 will
  apply...
 
 
  
   On 17 December 2014 at 11:48, Yeela Kaplan ykap...@redhat.com
 wrote:
   
Hi,
You are right, the problem is with the file handlers.
You can increase the number of handlers in pool using the vdsm
 config
file, which is supposed to be under the following directory in your
system:
   
/usr/lib/python2.6/site-packages/vdsm/config.py
   
The default value for 'process_pool_max_slots_per_domain' is 10, so
 you
can increase it by a bit, but not too much.
   
But I suspect the problem is in a larger scale, and this is only a
temporary relief for your system and this needs much more attention
 and a
proper fix.
could you please open a bug on RHEV/vdsm in bugzilla stating all of
 the
details of your setup and logs?
   
thanks,
Yeela
   
- Original Message -
 From: Tiemen Ruiten t.rui...@rdmedia.com
 To: Users@ovirt.org users@ovirt.org
 Sent: Wednesday, December 17, 2014 10:53:39 AM
 Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool'
 while
creating VM from template

 Would this be limits for the vdsm process? Then what is the
 proper way
 to
 change ulimits for VDSM?

 On 16 December 2014

Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

2014-12-17 Thread Tiemen Ruiten
Would this be limits for the vdsm process? Then what is the proper way to
change ulimits for VDSM?

On 16 December 2014 at 20:45, Donny Davis do...@cloudspin.me wrote:

 The only thing I can think of would be file hard and soft limits, but I am
 no oVirt pro.



 'no free file handlers in pool' that would make sense to me…

 Donny





 *From:* Tiemen Ruiten [mailto:t.rui...@rdmedia.com]
 *Sent:* Tuesday, December 16, 2014 12:40 PM
 *To:* Donny Davis
 *Cc:* Users@ovirt.org
 *Subject:* Re: [ovirt-users] ERROR 'no free file handlers in pool' while
 creating VM from template



 About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16
 hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM,
 processor load on both nodes was less than 1.



 On 16 December 2014 at 20:29, Donny Davis do...@cloudspin.me wrote:

 What is the VM load you are running on your servers?


 -Original Message-
 From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf
 Of Tiemen Ruiten
 Sent: Tuesday, December 16, 2014 12:27 PM
 To: users@ovirt.org
 Subject: [ovirt-users] ERROR 'no free file handlers in pool' while
 creating VM from template

 Hello,

 I ran into a nasty problem today when creating a new, cloned VM from a
 template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node
 where I started a VM creation job, load skyrocketed and some VMs stopped
 responding until and after the job failed. Everything recovered without
 intervention, but this obviously shouldn't happen. I have attached the
 relevant vdsm log file. The button to create the VM was pressed around
 11:17, the first error in the vdsm log is at 11:23:58.

 The ISO domain is a gluster volume exposed via NFS, the storage domain for
 the VM's is also a gluster volume. The underlying filesystem is ZFS.
 The hypervisor nodes are full CentOS 6 installs.

 I'm guessing the 'no free file handlers in pool' in the vdsm log file is
 key here. What can I do to prevent this from happening again? Apart from
 not creating new VMs of course :)

 Tiemen




 --

 Tiemen Ruiten
 Systems Engineer
 RD Media



-- 
Tiemen Ruiten
Systems Engineer
RD Media
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

2014-12-17 Thread Yeela Kaplan
Hi,
You are right, the problem is with the file handlers.
You can increase the number of handlers in pool using the vdsm config file, 
which is supposed to be under the following directory in your system:

/usr/lib/python2.6/site-packages/vdsm/config.py

The default value for 'process_pool_max_slots_per_domain' is 10, so you can 
increase it by a bit, but not too much.

But I suspect the problem is in a larger scale, and this is only a temporary 
relief for your system and this needs much more attention and a proper fix.
could you please open a bug on RHEV/vdsm in bugzilla stating all of the details 
of your setup and logs?

thanks,
Yeela

- Original Message -
 From: Tiemen Ruiten t.rui...@rdmedia.com
 To: Users@ovirt.org users@ovirt.org
 Sent: Wednesday, December 17, 2014 10:53:39 AM
 Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while 
 creating VM from template
 
 Would this be limits for the vdsm process? Then what is the proper way to
 change ulimits for VDSM?
 
 On 16 December 2014 at 20:45, Donny Davis  do...@cloudspin.me  wrote:
 
 
 
 
 The only thing I can think of would be file hard and soft limits, but I am no
 oVirt pro.
 
 
 
 'no free file handlers in pool' that would make sense to me…
 
 Donny
 
 
 
 
 
 From: Tiemen Ruiten [mailto: t.rui...@rdmedia.com ]
 Sent: Tuesday, December 16, 2014 12:40 PM
 To: Donny Davis
 Cc: Users@ovirt.org
 Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
 creating VM from template
 
 
 
 
 
 About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16
 hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM,
 processor load on both nodes was less than 1.
 
 
 
 
 
 On 16 December 2014 at 20:29, Donny Davis  do...@cloudspin.me  wrote:
 
 What is the VM load you are running on your servers?
 
 
 
 -Original Message-
 From: users-boun...@ovirt.org [mailto: users-boun...@ovirt.org ] On Behalf Of
 Tiemen Ruiten
 Sent: Tuesday, December 16, 2014 12:27 PM
 To: users@ovirt.org
 Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating
 VM from template
 
 Hello,
 
 I ran into a nasty problem today when creating a new, cloned VM from a
 template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node
 where I started a VM creation job, load skyrocketed and some VMs stopped
 responding until and after the job failed. Everything recovered without
 intervention, but this obviously shouldn't happen. I have attached the
 relevant vdsm log file. The button to create the VM was pressed around
 11:17, the first error in the vdsm log is at 11:23:58.
 
 The ISO domain is a gluster volume exposed via NFS, the storage domain for
 the VM's is also a gluster volume. The underlying filesystem is ZFS.
 The hypervisor nodes are full CentOS 6 installs.
 
 I'm guessing the 'no free file handlers in pool' in the vdsm log file is key
 here. What can I do to prevent this from happening again? Apart from not
 creating new VMs of course :)
 
 Tiemen
 
 
 
 
 
 
 
 
 
 --
 
 
 Tiemen Ruiten
 Systems Engineer
 RD Media
 
 
 --
 Tiemen Ruiten
 Systems Engineer
 RD Media
 
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
 
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

2014-12-17 Thread Tiemen Ruiten
Thank you, I will try to increase to 20 and see what happens. Bug is filed:
https://bugzilla.redhat.com/show_bug.cgi?id=1175255

On 17 December 2014 at 11:48, Yeela Kaplan ykap...@redhat.com wrote:

 Hi,
 You are right, the problem is with the file handlers.
 You can increase the number of handlers in pool using the vdsm config
 file, which is supposed to be under the following directory in your system:

 /usr/lib/python2.6/site-packages/vdsm/config.py

 The default value for 'process_pool_max_slots_per_domain' is 10, so you
 can increase it by a bit, but not too much.

 But I suspect the problem is in a larger scale, and this is only a
 temporary relief for your system and this needs much more attention and a
 proper fix.
 could you please open a bug on RHEV/vdsm in bugzilla stating all of the
 details of your setup and logs?

 thanks,
 Yeela

 - Original Message -
  From: Tiemen Ruiten t.rui...@rdmedia.com
  To: Users@ovirt.org users@ovirt.org
  Sent: Wednesday, December 17, 2014 10:53:39 AM
  Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
 creating VM from template
 
  Would this be limits for the vdsm process? Then what is the proper way to
  change ulimits for VDSM?
 
  On 16 December 2014 at 20:45, Donny Davis  do...@cloudspin.me  wrote:
 
 
 
 
  The only thing I can think of would be file hard and soft limits, but I
 am no
  oVirt pro.
 
 
 
  'no free file handlers in pool' that would make sense to me…
 
  Donny
 
 
 
 
 
  From: Tiemen Ruiten [mailto: t.rui...@rdmedia.com ]
  Sent: Tuesday, December 16, 2014 12:40 PM
  To: Donny Davis
  Cc: Users@ovirt.org
  Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
  creating VM from template
 
 
 
 
 
  About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16
  hyperthreaded cores and 64 GB of RAM each. At the moment I created the
 VM,
  processor load on both nodes was less than 1.
 
 
 
 
 
  On 16 December 2014 at 20:29, Donny Davis  do...@cloudspin.me  wrote:
 
  What is the VM load you are running on your servers?
 
 
 
  -Original Message-
  From: users-boun...@ovirt.org [mailto: users-boun...@ovirt.org ] On
 Behalf Of
  Tiemen Ruiten
  Sent: Tuesday, December 16, 2014 12:27 PM
  To: users@ovirt.org
  Subject: [ovirt-users] ERROR 'no free file handlers in pool' while
 creating
  VM from template
 
  Hello,
 
  I ran into a nasty problem today when creating a new, cloned VM from a
  template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the
 node
  where I started a VM creation job, load skyrocketed and some VMs stopped
  responding until and after the job failed. Everything recovered without
  intervention, but this obviously shouldn't happen. I have attached the
  relevant vdsm log file. The button to create the VM was pressed around
  11:17, the first error in the vdsm log is at 11:23:58.
 
  The ISO domain is a gluster volume exposed via NFS, the storage domain
 for
  the VM's is also a gluster volume. The underlying filesystem is ZFS.
  The hypervisor nodes are full CentOS 6 installs.
 
  I'm guessing the 'no free file handlers in pool' in the vdsm log file is
 key
  here. What can I do to prevent this from happening again? Apart from not
  creating new VMs of course :)
 
  Tiemen
 
 
 
 
 
 
 
 
 
  --
 
 
  Tiemen Ruiten
  Systems Engineer
  RD Media
 
 
  --
  Tiemen Ruiten
  Systems Engineer
  RD Media
 
  ___
  Users mailing list
  Users@ovirt.org
  http://lists.ovirt.org/mailman/listinfo/users
 



-- 
Tiemen Ruiten
Systems Engineer
RD Media
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

2014-12-17 Thread Yeela Kaplan


- Original Message -
 From: Tiemen Ruiten t.rui...@rdmedia.com
 To: Yeela Kaplan ykap...@redhat.com
 Cc: Users@ovirt.org users@ovirt.org
 Sent: Wednesday, December 17, 2014 1:22:59 PM
 Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while 
 creating VM from template
 
 Thank you, I will try to increase to 20 and see what happens. Bug is filed:
 https://bugzilla.redhat.com/show_bug.cgi?id=1175255

Thank you
and I forgot to mention that you have to restart vdsmd so the change will 
apply...


 
 On 17 December 2014 at 11:48, Yeela Kaplan ykap...@redhat.com wrote:
 
  Hi,
  You are right, the problem is with the file handlers.
  You can increase the number of handlers in pool using the vdsm config
  file, which is supposed to be under the following directory in your system:
 
  /usr/lib/python2.6/site-packages/vdsm/config.py
 
  The default value for 'process_pool_max_slots_per_domain' is 10, so you
  can increase it by a bit, but not too much.
 
  But I suspect the problem is in a larger scale, and this is only a
  temporary relief for your system and this needs much more attention and a
  proper fix.
  could you please open a bug on RHEV/vdsm in bugzilla stating all of the
  details of your setup and logs?
 
  thanks,
  Yeela
 
  - Original Message -
   From: Tiemen Ruiten t.rui...@rdmedia.com
   To: Users@ovirt.org users@ovirt.org
   Sent: Wednesday, December 17, 2014 10:53:39 AM
   Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
  creating VM from template
  
   Would this be limits for the vdsm process? Then what is the proper way to
   change ulimits for VDSM?
  
   On 16 December 2014 at 20:45, Donny Davis  do...@cloudspin.me  wrote:
  
  
  
  
   The only thing I can think of would be file hard and soft limits, but I
  am no
   oVirt pro.
  
  
  
   'no free file handlers in pool' that would make sense to me…
  
   Donny
  
  
  
  
  
   From: Tiemen Ruiten [mailto: t.rui...@rdmedia.com ]
   Sent: Tuesday, December 16, 2014 12:40 PM
   To: Donny Davis
   Cc: Users@ovirt.org
   Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
   creating VM from template
  
  
  
  
  
   About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16
   hyperthreaded cores and 64 GB of RAM each. At the moment I created the
  VM,
   processor load on both nodes was less than 1.
  
  
  
  
  
   On 16 December 2014 at 20:29, Donny Davis  do...@cloudspin.me  wrote:
  
   What is the VM load you are running on your servers?
  
  
  
   -Original Message-
   From: users-boun...@ovirt.org [mailto: users-boun...@ovirt.org ] On
  Behalf Of
   Tiemen Ruiten
   Sent: Tuesday, December 16, 2014 12:27 PM
   To: users@ovirt.org
   Subject: [ovirt-users] ERROR 'no free file handlers in pool' while
  creating
   VM from template
  
   Hello,
  
   I ran into a nasty problem today when creating a new, cloned VM from a
   template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the
  node
   where I started a VM creation job, load skyrocketed and some VMs stopped
   responding until and after the job failed. Everything recovered without
   intervention, but this obviously shouldn't happen. I have attached the
   relevant vdsm log file. The button to create the VM was pressed around
   11:17, the first error in the vdsm log is at 11:23:58.
  
   The ISO domain is a gluster volume exposed via NFS, the storage domain
  for
   the VM's is also a gluster volume. The underlying filesystem is ZFS.
   The hypervisor nodes are full CentOS 6 installs.
  
   I'm guessing the 'no free file handlers in pool' in the vdsm log file is
  key
   here. What can I do to prevent this from happening again? Apart from not
   creating new VMs of course :)
  
   Tiemen
  
  
  
  
  
  
  
  
  
   --
  
  
   Tiemen Ruiten
   Systems Engineer
   RD Media
  
  
   --
   Tiemen Ruiten
   Systems Engineer
   RD Media
  
   ___
   Users mailing list
   Users@ovirt.org
   http://lists.ovirt.org/mailman/listinfo/users
  
 
 
 
 --
 Tiemen Ruiten
 Systems Engineer
 RD Media
 
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

2014-12-17 Thread Yeela Kaplan
Just another thought,
from looking at the vdsm logs it looks like there are too many calls to 
getVolumeSize that are eating up all handlers
and eventually a timeout occurs.
Adam, do you have any idea about this? 

- Original Message -
 From: Yeela Kaplan ykap...@redhat.com
 To: Tiemen Ruiten t.rui...@rdmedia.com
 Cc: Users@ovirt.org users@ovirt.org
 Sent: Wednesday, December 17, 2014 1:30:52 PM
 Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while 
 creating VM from template
 
 
 
 - Original Message -
  From: Tiemen Ruiten t.rui...@rdmedia.com
  To: Yeela Kaplan ykap...@redhat.com
  Cc: Users@ovirt.org users@ovirt.org
  Sent: Wednesday, December 17, 2014 1:22:59 PM
  Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
  creating VM from template
  
  Thank you, I will try to increase to 20 and see what happens. Bug is filed:
  https://bugzilla.redhat.com/show_bug.cgi?id=1175255
 
 Thank you
 and I forgot to mention that you have to restart vdsmd so the change will
 apply...
 
 
  
  On 17 December 2014 at 11:48, Yeela Kaplan ykap...@redhat.com wrote:
  
   Hi,
   You are right, the problem is with the file handlers.
   You can increase the number of handlers in pool using the vdsm config
   file, which is supposed to be under the following directory in your
   system:
  
   /usr/lib/python2.6/site-packages/vdsm/config.py
  
   The default value for 'process_pool_max_slots_per_domain' is 10, so you
   can increase it by a bit, but not too much.
  
   But I suspect the problem is in a larger scale, and this is only a
   temporary relief for your system and this needs much more attention and a
   proper fix.
   could you please open a bug on RHEV/vdsm in bugzilla stating all of the
   details of your setup and logs?
  
   thanks,
   Yeela
  
   - Original Message -
From: Tiemen Ruiten t.rui...@rdmedia.com
To: Users@ovirt.org users@ovirt.org
Sent: Wednesday, December 17, 2014 10:53:39 AM
Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
   creating VM from template
   
Would this be limits for the vdsm process? Then what is the proper way
to
change ulimits for VDSM?
   
On 16 December 2014 at 20:45, Donny Davis  do...@cloudspin.me  wrote:
   
   
   
   
The only thing I can think of would be file hard and soft limits, but I
   am no
oVirt pro.
   
   
   
'no free file handlers in pool' that would make sense to me…
   
Donny
   
   
   
   
   
From: Tiemen Ruiten [mailto: t.rui...@rdmedia.com ]
Sent: Tuesday, December 16, 2014 12:40 PM
To: Donny Davis
Cc: Users@ovirt.org
Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while
creating VM from template
   
   
   
   
   
About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16
hyperthreaded cores and 64 GB of RAM each. At the moment I created the
   VM,
processor load on both nodes was less than 1.
   
   
   
   
   
On 16 December 2014 at 20:29, Donny Davis  do...@cloudspin.me  wrote:
   
What is the VM load you are running on your servers?
   
   
   
-Original Message-
From: users-boun...@ovirt.org [mailto: users-boun...@ovirt.org ] On
   Behalf Of
Tiemen Ruiten
Sent: Tuesday, December 16, 2014 12:27 PM
To: users@ovirt.org
Subject: [ovirt-users] ERROR 'no free file handlers in pool' while
   creating
VM from template
   
Hello,
   
I ran into a nasty problem today when creating a new, cloned VM from a
template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the
   node
where I started a VM creation job, load skyrocketed and some VMs
stopped
responding until and after the job failed. Everything recovered without
intervention, but this obviously shouldn't happen. I have attached the
relevant vdsm log file. The button to create the VM was pressed around
11:17, the first error in the vdsm log is at 11:23:58.
   
The ISO domain is a gluster volume exposed via NFS, the storage domain
   for
the VM's is also a gluster volume. The underlying filesystem is ZFS.
The hypervisor nodes are full CentOS 6 installs.
   
I'm guessing the 'no free file handlers in pool' in the vdsm log file
is
   key
here. What can I do to prevent this from happening again? Apart from
not
creating new VMs of course :)
   
Tiemen
   
   
   
   
   
   
   
   
   
--
   
   
Tiemen Ruiten
Systems Engineer
RD Media
   
   
--
Tiemen Ruiten
Systems Engineer
RD Media
   
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
   
  
  
  
  --
  Tiemen Ruiten
  Systems Engineer
  RD Media
  
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

2014-12-16 Thread Donny Davis
What is the VM load you are running on your servers?

-Original Message-
From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf Of 
Tiemen Ruiten
Sent: Tuesday, December 16, 2014 12:27 PM
To: users@ovirt.org
Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM 
from template

Hello,

I ran into a nasty problem today when creating a new, cloned VM from a template 
(one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I 
started a VM creation job, load skyrocketed and some VMs stopped responding 
until and after the job failed. Everything recovered without intervention, but 
this obviously shouldn't happen. I have attached the relevant vdsm log file. 
The button to create the VM was pressed around 11:17, the first error in the 
vdsm log is at 11:23:58.

The ISO domain is a gluster volume exposed via NFS, the storage domain for the 
VM's is also a gluster volume. The underlying filesystem is ZFS.
The hypervisor nodes are full CentOS 6 installs.

I'm guessing the 'no free file handlers in pool' in the vdsm log file is key 
here. What can I do to prevent this from happening again? Apart from not 
creating new VMs of course :)

Tiemen


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

2014-12-16 Thread Tiemen Ruiten
About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16
hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM,
processor load on both nodes was less than 1.

On 16 December 2014 at 20:29, Donny Davis do...@cloudspin.me wrote:

 What is the VM load you are running on your servers?

 -Original Message-
 From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf
 Of Tiemen Ruiten
 Sent: Tuesday, December 16, 2014 12:27 PM
 To: users@ovirt.org
 Subject: [ovirt-users] ERROR 'no free file handlers in pool' while
 creating VM from template

 Hello,

 I ran into a nasty problem today when creating a new, cloned VM from a
 template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node
 where I started a VM creation job, load skyrocketed and some VMs stopped
 responding until and after the job failed. Everything recovered without
 intervention, but this obviously shouldn't happen. I have attached the
 relevant vdsm log file. The button to create the VM was pressed around
 11:17, the first error in the vdsm log is at 11:23:58.

 The ISO domain is a gluster volume exposed via NFS, the storage domain for
 the VM's is also a gluster volume. The underlying filesystem is ZFS.
 The hypervisor nodes are full CentOS 6 installs.

 I'm guessing the 'no free file handlers in pool' in the vdsm log file is
 key here. What can I do to prevent this from happening again? Apart from
 not creating new VMs of course :)

 Tiemen




-- 
Tiemen Ruiten
Systems Engineer
RD Media
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM from template

2014-12-16 Thread Donny Davis
The only thing I can think of would be file hard and soft limits, but I am no 
oVirt pro. 

 

'no free file handlers in pool' that would make sense to me… 

Donny

 

 

From: Tiemen Ruiten [mailto:t.rui...@rdmedia.com] 
Sent: Tuesday, December 16, 2014 12:40 PM
To: Donny Davis
Cc: Users@ovirt.org
Subject: Re: [ovirt-users] ERROR 'no free file handlers in pool' while creating 
VM from template

 

About 25-30. The nodes are Xeon(R) CPU E5-2650 0 @ 2.00GHz with 16 
hyperthreaded cores and 64 GB of RAM each. At the moment I created the VM, 
processor load on both nodes was less than 1. 

 

On 16 December 2014 at 20:29, Donny Davis do...@cloudspin.me wrote:

What is the VM load you are running on your servers?


-Original Message-
From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf Of 
Tiemen Ruiten
Sent: Tuesday, December 16, 2014 12:27 PM
To: users@ovirt.org
Subject: [ovirt-users] ERROR 'no free file handlers in pool' while creating VM 
from template

Hello,

I ran into a nasty problem today when creating a new, cloned VM from a template 
(one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I 
started a VM creation job, load skyrocketed and some VMs stopped responding 
until and after the job failed. Everything recovered without intervention, but 
this obviously shouldn't happen. I have attached the relevant vdsm log file. 
The button to create the VM was pressed around 11:17, the first error in the 
vdsm log is at 11:23:58.

The ISO domain is a gluster volume exposed via NFS, the storage domain for the 
VM's is also a gluster volume. The underlying filesystem is ZFS.
The hypervisor nodes are full CentOS 6 installs.

I'm guessing the 'no free file handlers in pool' in the vdsm log file is key 
here. What can I do to prevent this from happening again? Apart from not 
creating new VMs of course :)

Tiemen






 

-- 

Tiemen Ruiten
Systems Engineer
RD Media

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users