Re: [Users] I am using the CentOS build from the 12th. I the engine is dieing with the following error.

Thu, 21 Jun 2012 10:20:41 -0700 


On 06/18/2012 04:47 AM, Juan Hernandez wrote:

On 06/18/2012 04:25 AM, Robert Middleswarth wrote:

2012-06-17 05:21:24,682 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(ajp--0.0.0.0-8009-9) Failed to decryptjava.io.FileNotFoundException:
/etc/pki/ovirt-engine/.keystore (Too many open files)
2012-06-17 05:21:26,121 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-88) Failed to
decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
(Too many open files)
2012-06-17 05:21:26,130 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
(QuartzScheduler_Worker-88) XML RPC error in command GetCapabilitiesVDS
( Vds: ann100ovirt01 ), the error was:
java.util.concurrent.ExecutionException:
java.lang.reflect.InvocationTargetException, SocketException: Too many
open files
2012-06-17 15:49:08,577 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-28) Failed to
decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
(Too many open files)
2012-06-17 20:21:13,943 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-64) Failed to
decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
(Too many open files)
2012-06-17 20:21:15,949 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-65) Failed to
decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
(Too many open files)
2012-06-17 21:00:15,133 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-98) Failed to
decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
(Too many open files)
2012-06-17 21:07:00,037 WARN  [org.ovirt.engine.core.ServletUtils]
(ajp--0.0.0.0-8009-5) File "/usr/share/vdsm-bootstrap/deployUtil.py is
47867 bytes long. Please reconsider using this servlet for files larger
than 8192 bytes.
2012-06-17 21:07:00,115 WARN  [org.ovirt.engine.core.ServletUtils]
(ajp--0.0.0.0-8009-6) File "/usr/share/vdsm-bootstrap/vds_bootstrap.py
is 33884 bytes long. Please reconsider using this servlet for files
larger than 8192 bytes.
2012-06-17 21:32:10,686 WARN  [org.ovirt.engine.core.ServletUtils]
(ajp--0.0.0.0-8009-6) File "/usr/share/vdsm-bootstrap/deployUtil.py is
47867 bytes long. Please reconsider using this servlet for files larger
than 8192 bytes.
2012-06-17 21:32:10,752 WARN  [org.ovirt.engine.core.ServletUtils]
(ajp--0.0.0.0-8009-1) File "/usr/share/vdsm-bootstrap/vds_bootstrap.py
is 33884 bytes long. Please reconsider using this servlet for files
larger than 8192 bytes.
2012-06-17 21:44:34,028 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
(QuartzScheduler_Worker-23) XML RPC error in command ListVDS ( Vds:
ann100ovirt01 ), the error was: java.util.concurrent.ExecutionException:
java.lang.reflect.InvocationTargetException, SocketException: Too many
open files
2012-06-17 21:44:36,034 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-26) Failed to
decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
(Too many open files)
2012-06-17 21:44:36,045 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
(QuartzScheduler_Worker-26) XML RPC error in command GetCapabilitiesVDS
( Vds: ann100ovirt01 ), the error was:
java.util.concurrent.ExecutionException:
java.lang.reflect.InvocationTargetException, SocketException: Too many
open files


I restarted ovirt-engine and ran lsof -p 3557

I see hundreds of the same open file.

java    3557 ovirt  839r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  840r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  841r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  842r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  843r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  844r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  845r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  846r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  847r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  848r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  849r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  850r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  851r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  852r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  853r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore
java    3557 ovirt  854r   REG  253,0     3230  7865608
/etc/pki/ovirt-engine/.keystore

Is anyone else seeing this?


There are two issues that might be affecting you here:


1. We may have a leak of file descriptors on the logic that opens the 
keystore. Can you take a look at the following change? Can you apply 
it and let us know if it fixes your problem?


http://gerrit.ovirt.org/#/c/5423


I am still tiring to get a working build environment setup.  However 
dre...@gmail.com added into his repo and my problem went away after the 
change was applied.  I recommend that this fix gets cherry-picked into 
the 3.1 tree

2. You may be having a problem with the file descriptors limit due to 
other issues. During setup we should increase the file descriptor 
limit from 1024 (the default) to 65535, but we do that for the "jboss" 
user instead of for "ovirt". The fix for this issue is in review:


http://gerrit.ovirt.org/#/c/5375


The workaround for this is to edit the /etc/security/limits.conf file 
and replace "jboss" with "ovirt", then restart the engine.


Thanks
Robert

_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users






















					Reply via email to