Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-19 Thread Alon Bar-Lev

As I recommended before, please open a new thread with 'how to rescue storage 
domain', I hope someone who is familiar with storage domain structure will be 
able to assist.
Your installation seems to be corrupted more than just permissions, 
certificates, stores.

- Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Alon Bar-Lev alo...@redhat.com, Users@ovirt.org
 Sent: Friday, April 19, 2013 3:40:55 AM
 Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
 
 Since I'm not able to reinstall the host from the ovirt-engine web
 interface, as another thought I wanted to see if I could bring up a
 third host and add it to the cluster.
 I have a host Fedora 17 box ready to go but I can't add it to the
 cluster.  It states that there are no available server in the cluster
 to probe the new host.
 
 What about approaching it from the other direction.  Would I be able
 to stand up an ovirt-h node on the same hardware and then add it to
 ovirt from the host itself, using the setup menu?
 
 Could it then obtain spm status and bring the storage domain online?
 
 On Thu, Apr 18, 2013 at 7:20 PM, Chris Smith whitehat...@gmail.com wrote:
  engine.log attached
 
  On Thu, Apr 18, 2013 at 7:11 PM, Alon Bar-Lev alo...@redhat.com wrote:
  Need to know precise error, please attach engine.log.
 
 
  - Original Message -
  From: Chris Smith whitehat...@gmail.com
  To: Alon Bar-Lev alo...@redhat.com
  Cc: Users@ovirt.org
  Sent: Friday, April 19, 2013 2:03:59 AM
  Subject: Re: [Users] Certificates and PKI seem to be broken after yum
  update
 
  So as of now, I can put the host into maintenance mode using the
  ovirt-engine web interface.  I can also try and activate it.  It
  states that the host was activated.   The host never actually comes up
  or contends for SPM status, and the data center never actually comes
  online.
 
  If I put the host into maintenance mode and try to reinstall it, it
  throws an error and size must be between 0 and 50.
 
  On Thu, Apr 18, 2013 at 6:51 PM, Alon Bar-Lev alo...@redhat.com wrote:
   I am not sure I understand the status.
  
   Everything is working or not.
   If not, what exactly fails?
   Why do you run it 'again'?
  
   What happens if you reinstall host? Go to maintenance and select
   reinstall?
  
   I cannot understand how all this results from upgrade, something had
   changed, the CA certificate installed on the host is probably not the
   CA
   certificate of the engine.
  
   - Original Message -
   From: Chris Smith whitehat...@gmail.com
   To: Alon Bar-Lev alo...@redhat.com, Users@ovirt.org
   Sent: Friday, April 19, 2013 1:45:23 AM
   Subject: Re: [Users] Certificates and PKI seem to be broken after yum
   update
  
   On Thu, Apr 18, 2013 at 6:44 PM, Chris Smith whitehat...@gmail.com
   wrote:
I made a backup of the .truststore, and then followed the steps and
then rebooted both the ovirt-engine and one of the hosts, and
everything worked properly.
   
If I run it again, or enter the wrong password it throws an error
about the key store already existing, or that the password was wrong
so I'm pretty sure it's good.
   
vdsm.log on the host still shows:
   
Traceback (most recent call last):
  File /usr/lib64/python2.7/SocketServer.py, line 582, in
process_request_thread
self.finish_request(request, client_address)
  File
  /usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py,
line 66, in finish_request
request.do_handshake()
  File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
   
engine.log on the host shows:
   
2013-04-18 18:42:43,632 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-68) Failed to decryptData must start with
zero
2013-04-18 18:42:43,642 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
(QuartzScheduler_Worker-68) XML RPC error in command
GetCapabilitiesVDS ( Vds: transporter ), the error was:
java.util.concurrent.ExecutionException:
java.lang.reflect.InvocationTargetException,
SunCertPathBuilderException: unable to find valid certification path
to requested target
   
   
On Thu, Apr 18, 2013 at 4:06 AM, Alon Bar-Lev alo...@redhat.com
wrote:
   
You should ask these question in separate thread so people may pick
them
up.
   
For the .truststore, try to remove it and then execute:
   
# rm -f /etc/pki/ovirt-engine/.truststore
# keytool -import -noprompt -trustcacerts -alias cacert -keypass
mypass
-file /etc/pki/ovirt-engine/certs/ca.der -keystore
/etc/pki/ovirt-engine/.truststore -storepass mypass
# chown ovirt:ovirt /etc/pki/ovirt-engine/.truststore
   
It should recreate the truststore

Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-18 Thread Alon Bar-Lev

You should ask these question in separate thread so people may pick them up.

For the .truststore, try to remove it and then execute:

# rm -f /etc/pki/ovirt-engine/.truststore
# keytool -import -noprompt -trustcacerts -alias cacert -keypass mypass -file 
/etc/pki/ovirt-engine/certs/ca.der -keystore /etc/pki/ovirt-engine/.truststore 
-storepass mypass
# chown ovirt:ovirt /etc/pki/ovirt-engine/.truststore

It should recreate the truststore with the ca certificate you have.

- Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Alon Bar-Lev alo...@redhat.com
 Cc: Users@ovirt.org
 Sent: Thursday, April 18, 2013 7:18:27 AM
 Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
 
 If it would be easier than re-setting up the certificates, I'm also
 willing to just start over and rebuild, but I would like to export the
 VM's I have first.
 One of them is a spacewalk server, another runs DNS, and DHCP for my
 test network, and I have an asterisk server.  I would like to avoid
 having to re-create all of them.
 
 The VM's are up and running now, so I could export all of the
 configurations / backup the file systems, etc.
 
 Preferably I could export the VM's to an NFS export domain, or a
 mounted NFS share so that I can import them to the new storage domain,
 after I run engine-cleanup and get everything set back up.  Is there
 an easy way to do this?  Is it possible to create and attach an NFS
 export domain directly from the CLI without access to the ovirt
 manager without communication between the manager and hosts due to the
 pki issue?  Can I export the VM's directly from the hosts to a
 standard NFS share?
 
 Is there an equivalent xml and image file for the VM?
 
 My storage domain is iscsi and is served out from another server over
 4 bonded 1 Gbps copper links.
 
 
 
 On Wed, Apr 17, 2013 at 11:46 PM, Chris Smith whitehat...@gmail.com wrote:
  I checked the .truststore on the ovirt engine, and it seems fine.
 
  [root@reliant ovirt-engine]# ls -l .truststore
  -rwxr-x---. 1 ovirt ovirt 918 Apr  6 21:56 .truststore
 
  It's not zero bytes anyway.
 
  It's also the same size as the .truststore in the ovirt engine backups.
 
  [root@reliant ovirt-engine-backups]# find ./ -name .truststore -exec ls -l
  {} \;
  -rwxr-x---. 1 ovirt ovirt 918 Aug 26  2012
  ./ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
  -rwxr-x---. 1 root root 918 Mar 24 12:42
  ./ovirt-engine-2013_03_24_11_15_19/ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
 
  I haven't looked at the installCA.sh script yet.
 
  On Mon, Apr 8, 2013 at 2:58 AM, Alon Bar-Lev alo...@redhat.com wrote:
  This error means that the /etc/pki/ovirt-engine/.truststore is unreadable
  or does not contain the /etc/pki/ovirt-engine/ca.pem certificate.
 
  Unfortunately, the pki administration is weak in current implementation,
  you can trace the installation script and checkout the calls to
  installCA.sh to how to reproduce, please note that password are encrypted
  in database using the private key locate in .keystore so if you are to
  re-generate anything remember to keep the engine private key.
 
  However, if you succeed in login, the remaining problem you have is the
  .truststore permissions and/or content.
 
  Regards,
  Alon Bar-Lev.
 
  - Original Message -
  From: Chris Smith whitehat...@gmail.com
  To: Alon Bar-Lev alo...@redhat.com
  Cc: Users@ovirt.org
  Sent: Monday, April 8, 2013 9:46:46 AM
  Subject: Re: [Users] Certificates and PKI seem to be broken after yum
  update
 
  After setting the .keystore owner and group owner to ovirt, and
  rebooting, I now have a new error in engine.log
 
  2013-04-08 02:39:16,787 ERROR
  [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
  (QuartzScheduler_Worker-95) Failed to decryptData must start with zero
  2013-04-08 02:39:16,845 ERROR
  [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
  (QuartzScheduler_Worker-95) XML RPC error in command
  GetCapabilitiesVDS ( Vds: transporter ), the error was:
  java.util.concurrent.ExecutionException:
  java.lang.reflect.InvocationTargetException,
  SunCertPathBuilderException: unable to find valid certification path
  to requested target
 
  Are there other files that may have been affected that I can also
  correct ownership or permissions on?
 
  On the host side, I get certificate unknown in vdsm.log
 
File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
  self._sslobj.do_handshake()
  SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
  routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
  Thread-757809::ERROR::2013-04-08
  02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client
  ('172.16.23.8', 54489)
  Traceback (most recent call last):
File /usr/lib64/python2.7/SocketServer.py, line 582, in
  process_request_thread
  self.finish_request(request, client_address)
File /usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py

Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-18 Thread Chris Smith
On Thu, Apr 18, 2013 at 6:44 PM, Chris Smith whitehat...@gmail.com wrote:
 I made a backup of the .truststore, and then followed the steps and
 then rebooted both the ovirt-engine and one of the hosts, and
 everything worked properly.

 If I run it again, or enter the wrong password it throws an error
 about the key store already existing, or that the password was wrong
 so I'm pretty sure it's good.

 vdsm.log on the host still shows:

 Traceback (most recent call last):
   File /usr/lib64/python2.7/SocketServer.py, line 582, in
 process_request_thread
 self.finish_request(request, client_address)
   File /usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py,
 line 66, in finish_request
 request.do_handshake()
   File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
 self._sslobj.do_handshake()
 SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
 routines:SSL3_READ_BYTES:sslv3 alert certificate unknown

 engine.log on the host shows:

 2013-04-18 18:42:43,632 ERROR
 [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
 (QuartzScheduler_Worker-68) Failed to decryptData must start with zero
 2013-04-18 18:42:43,642 ERROR
 [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
 (QuartzScheduler_Worker-68) XML RPC error in command
 GetCapabilitiesVDS ( Vds: transporter ), the error was:
 java.util.concurrent.ExecutionException:
 java.lang.reflect.InvocationTargetException,
 SunCertPathBuilderException: unable to find valid certification path
 to requested target


 On Thu, Apr 18, 2013 at 4:06 AM, Alon Bar-Lev alo...@redhat.com wrote:

 You should ask these question in separate thread so people may pick them up.

 For the .truststore, try to remove it and then execute:

 # rm -f /etc/pki/ovirt-engine/.truststore
 # keytool -import -noprompt -trustcacerts -alias cacert -keypass mypass 
 -file /etc/pki/ovirt-engine/certs/ca.der -keystore 
 /etc/pki/ovirt-engine/.truststore -storepass mypass
 # chown ovirt:ovirt /etc/pki/ovirt-engine/.truststore

 It should recreate the truststore with the ca certificate you have.

 - Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Alon Bar-Lev alo...@redhat.com
 Cc: Users@ovirt.org
 Sent: Thursday, April 18, 2013 7:18:27 AM
 Subject: Re: [Users] Certificates and PKI seem to be broken after yum update

 If it would be easier than re-setting up the certificates, I'm also
 willing to just start over and rebuild, but I would like to export the
 VM's I have first.
 One of them is a spacewalk server, another runs DNS, and DHCP for my
 test network, and I have an asterisk server.  I would like to avoid
 having to re-create all of them.

 The VM's are up and running now, so I could export all of the
 configurations / backup the file systems, etc.

 Preferably I could export the VM's to an NFS export domain, or a
 mounted NFS share so that I can import them to the new storage domain,
 after I run engine-cleanup and get everything set back up.  Is there
 an easy way to do this?  Is it possible to create and attach an NFS
 export domain directly from the CLI without access to the ovirt
 manager without communication between the manager and hosts due to the
 pki issue?  Can I export the VM's directly from the hosts to a
 standard NFS share?

 Is there an equivalent xml and image file for the VM?

 My storage domain is iscsi and is served out from another server over
 4 bonded 1 Gbps copper links.



 On Wed, Apr 17, 2013 at 11:46 PM, Chris Smith whitehat...@gmail.com wrote:
  I checked the .truststore on the ovirt engine, and it seems fine.
 
  [root@reliant ovirt-engine]# ls -l .truststore
  -rwxr-x---. 1 ovirt ovirt 918 Apr  6 21:56 .truststore
 
  It's not zero bytes anyway.
 
  It's also the same size as the .truststore in the ovirt engine backups.
 
  [root@reliant ovirt-engine-backups]# find ./ -name .truststore -exec ls -l
  {} \;
  -rwxr-x---. 1 ovirt ovirt 918 Aug 26  2012
  ./ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
  -rwxr-x---. 1 root root 918 Mar 24 12:42
  ./ovirt-engine-2013_03_24_11_15_19/ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
 
  I haven't looked at the installCA.sh script yet.
 
  On Mon, Apr 8, 2013 at 2:58 AM, Alon Bar-Lev alo...@redhat.com wrote:
  This error means that the /etc/pki/ovirt-engine/.truststore is unreadable
  or does not contain the /etc/pki/ovirt-engine/ca.pem certificate.
 
  Unfortunately, the pki administration is weak in current implementation,
  you can trace the installation script and checkout the calls to
  installCA.sh to how to reproduce, please note that password are encrypted
  in database using the private key locate in .keystore so if you are to
  re-generate anything remember to keep the engine private key.
 
  However, if you succeed in login, the remaining problem you have is the
  .truststore permissions and/or content.
 
  Regards,
  Alon Bar-Lev.
 
  - Original Message -
  From: Chris Smith whitehat...@gmail.com
  To: Alon Bar-Lev

Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-18 Thread Alon Bar-Lev
I am not sure I understand the status.

Everything is working or not.
If not, what exactly fails?
Why do you run it 'again'?

What happens if you reinstall host? Go to maintenance and select reinstall?

I cannot understand how all this results from upgrade, something had changed, 
the CA certificate installed on the host is probably not the CA certificate of 
the engine.

- Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Alon Bar-Lev alo...@redhat.com, Users@ovirt.org
 Sent: Friday, April 19, 2013 1:45:23 AM
 Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
 
 On Thu, Apr 18, 2013 at 6:44 PM, Chris Smith whitehat...@gmail.com wrote:
  I made a backup of the .truststore, and then followed the steps and
  then rebooted both the ovirt-engine and one of the hosts, and
  everything worked properly.
 
  If I run it again, or enter the wrong password it throws an error
  about the key store already existing, or that the password was wrong
  so I'm pretty sure it's good.
 
  vdsm.log on the host still shows:
 
  Traceback (most recent call last):
File /usr/lib64/python2.7/SocketServer.py, line 582, in
  process_request_thread
  self.finish_request(request, client_address)
File /usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py,
  line 66, in finish_request
  request.do_handshake()
File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
  self._sslobj.do_handshake()
  SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
  routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
 
  engine.log on the host shows:
 
  2013-04-18 18:42:43,632 ERROR
  [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
  (QuartzScheduler_Worker-68) Failed to decryptData must start with zero
  2013-04-18 18:42:43,642 ERROR
  [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
  (QuartzScheduler_Worker-68) XML RPC error in command
  GetCapabilitiesVDS ( Vds: transporter ), the error was:
  java.util.concurrent.ExecutionException:
  java.lang.reflect.InvocationTargetException,
  SunCertPathBuilderException: unable to find valid certification path
  to requested target
 
 
  On Thu, Apr 18, 2013 at 4:06 AM, Alon Bar-Lev alo...@redhat.com wrote:
 
  You should ask these question in separate thread so people may pick them
  up.
 
  For the .truststore, try to remove it and then execute:
 
  # rm -f /etc/pki/ovirt-engine/.truststore
  # keytool -import -noprompt -trustcacerts -alias cacert -keypass mypass
  -file /etc/pki/ovirt-engine/certs/ca.der -keystore
  /etc/pki/ovirt-engine/.truststore -storepass mypass
  # chown ovirt:ovirt /etc/pki/ovirt-engine/.truststore
 
  It should recreate the truststore with the ca certificate you have.
 
  - Original Message -
  From: Chris Smith whitehat...@gmail.com
  To: Alon Bar-Lev alo...@redhat.com
  Cc: Users@ovirt.org
  Sent: Thursday, April 18, 2013 7:18:27 AM
  Subject: Re: [Users] Certificates and PKI seem to be broken after yum
  update
 
  If it would be easier than re-setting up the certificates, I'm also
  willing to just start over and rebuild, but I would like to export the
  VM's I have first.
  One of them is a spacewalk server, another runs DNS, and DHCP for my
  test network, and I have an asterisk server.  I would like to avoid
  having to re-create all of them.
 
  The VM's are up and running now, so I could export all of the
  configurations / backup the file systems, etc.
 
  Preferably I could export the VM's to an NFS export domain, or a
  mounted NFS share so that I can import them to the new storage domain,
  after I run engine-cleanup and get everything set back up.  Is there
  an easy way to do this?  Is it possible to create and attach an NFS
  export domain directly from the CLI without access to the ovirt
  manager without communication between the manager and hosts due to the
  pki issue?  Can I export the VM's directly from the hosts to a
  standard NFS share?
 
  Is there an equivalent xml and image file for the VM?
 
  My storage domain is iscsi and is served out from another server over
  4 bonded 1 Gbps copper links.
 
 
 
  On Wed, Apr 17, 2013 at 11:46 PM, Chris Smith whitehat...@gmail.com
  wrote:
   I checked the .truststore on the ovirt engine, and it seems fine.
  
   [root@reliant ovirt-engine]# ls -l .truststore
   -rwxr-x---. 1 ovirt ovirt 918 Apr  6 21:56 .truststore
  
   It's not zero bytes anyway.
  
   It's also the same size as the .truststore in the ovirt engine backups.
  
   [root@reliant ovirt-engine-backups]# find ./ -name .truststore -exec ls
   -l
   {} \;
   -rwxr-x---. 1 ovirt ovirt 918 Aug 26  2012
   ./ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
   -rwxr-x---. 1 root root 918 Mar 24 12:42
   ./ovirt-engine-2013_03_24_11_15_19/ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
  
   I haven't looked at the installCA.sh script yet.
  
   On Mon, Apr 8, 2013 at 2:58 AM, Alon Bar-Lev alo...@redhat.com wrote

Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-18 Thread Chris Smith
So as of now, I can put the host into maintenance mode using the
ovirt-engine web interface.  I can also try and activate it.  It
states that the host was activated.   The host never actually comes up
or contends for SPM status, and the data center never actually comes
online.

If I put the host into maintenance mode and try to reinstall it, it
throws an error and size must be between 0 and 50.

On Thu, Apr 18, 2013 at 6:51 PM, Alon Bar-Lev alo...@redhat.com wrote:
 I am not sure I understand the status.

 Everything is working or not.
 If not, what exactly fails?
 Why do you run it 'again'?

 What happens if you reinstall host? Go to maintenance and select reinstall?

 I cannot understand how all this results from upgrade, something had changed, 
 the CA certificate installed on the host is probably not the CA certificate 
 of the engine.

 - Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Alon Bar-Lev alo...@redhat.com, Users@ovirt.org
 Sent: Friday, April 19, 2013 1:45:23 AM
 Subject: Re: [Users] Certificates and PKI seem to be broken after yum update

 On Thu, Apr 18, 2013 at 6:44 PM, Chris Smith whitehat...@gmail.com wrote:
  I made a backup of the .truststore, and then followed the steps and
  then rebooted both the ovirt-engine and one of the hosts, and
  everything worked properly.
 
  If I run it again, or enter the wrong password it throws an error
  about the key store already existing, or that the password was wrong
  so I'm pretty sure it's good.
 
  vdsm.log on the host still shows:
 
  Traceback (most recent call last):
File /usr/lib64/python2.7/SocketServer.py, line 582, in
  process_request_thread
  self.finish_request(request, client_address)
File /usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py,
  line 66, in finish_request
  request.do_handshake()
File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
  self._sslobj.do_handshake()
  SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
  routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
 
  engine.log on the host shows:
 
  2013-04-18 18:42:43,632 ERROR
  [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
  (QuartzScheduler_Worker-68) Failed to decryptData must start with zero
  2013-04-18 18:42:43,642 ERROR
  [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
  (QuartzScheduler_Worker-68) XML RPC error in command
  GetCapabilitiesVDS ( Vds: transporter ), the error was:
  java.util.concurrent.ExecutionException:
  java.lang.reflect.InvocationTargetException,
  SunCertPathBuilderException: unable to find valid certification path
  to requested target
 
 
  On Thu, Apr 18, 2013 at 4:06 AM, Alon Bar-Lev alo...@redhat.com wrote:
 
  You should ask these question in separate thread so people may pick them
  up.
 
  For the .truststore, try to remove it and then execute:
 
  # rm -f /etc/pki/ovirt-engine/.truststore
  # keytool -import -noprompt -trustcacerts -alias cacert -keypass mypass
  -file /etc/pki/ovirt-engine/certs/ca.der -keystore
  /etc/pki/ovirt-engine/.truststore -storepass mypass
  # chown ovirt:ovirt /etc/pki/ovirt-engine/.truststore
 
  It should recreate the truststore with the ca certificate you have.
 
  - Original Message -
  From: Chris Smith whitehat...@gmail.com
  To: Alon Bar-Lev alo...@redhat.com
  Cc: Users@ovirt.org
  Sent: Thursday, April 18, 2013 7:18:27 AM
  Subject: Re: [Users] Certificates and PKI seem to be broken after yum
  update
 
  If it would be easier than re-setting up the certificates, I'm also
  willing to just start over and rebuild, but I would like to export the
  VM's I have first.
  One of them is a spacewalk server, another runs DNS, and DHCP for my
  test network, and I have an asterisk server.  I would like to avoid
  having to re-create all of them.
 
  The VM's are up and running now, so I could export all of the
  configurations / backup the file systems, etc.
 
  Preferably I could export the VM's to an NFS export domain, or a
  mounted NFS share so that I can import them to the new storage domain,
  after I run engine-cleanup and get everything set back up.  Is there
  an easy way to do this?  Is it possible to create and attach an NFS
  export domain directly from the CLI without access to the ovirt
  manager without communication between the manager and hosts due to the
  pki issue?  Can I export the VM's directly from the hosts to a
  standard NFS share?
 
  Is there an equivalent xml and image file for the VM?
 
  My storage domain is iscsi and is served out from another server over
  4 bonded 1 Gbps copper links.
 
 
 
  On Wed, Apr 17, 2013 at 11:46 PM, Chris Smith whitehat...@gmail.com
  wrote:
   I checked the .truststore on the ovirt engine, and it seems fine.
  
   [root@reliant ovirt-engine]# ls -l .truststore
   -rwxr-x---. 1 ovirt ovirt 918 Apr  6 21:56 .truststore
  
   It's not zero bytes anyway.
  
   It's also the same size as the .truststore in the ovirt engine backups

Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-18 Thread Alon Bar-Lev
Need to know precise error, please attach engine.log.


- Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Alon Bar-Lev alo...@redhat.com
 Cc: Users@ovirt.org
 Sent: Friday, April 19, 2013 2:03:59 AM
 Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
 
 So as of now, I can put the host into maintenance mode using the
 ovirt-engine web interface.  I can also try and activate it.  It
 states that the host was activated.   The host never actually comes up
 or contends for SPM status, and the data center never actually comes
 online.
 
 If I put the host into maintenance mode and try to reinstall it, it
 throws an error and size must be between 0 and 50.
 
 On Thu, Apr 18, 2013 at 6:51 PM, Alon Bar-Lev alo...@redhat.com wrote:
  I am not sure I understand the status.
 
  Everything is working or not.
  If not, what exactly fails?
  Why do you run it 'again'?
 
  What happens if you reinstall host? Go to maintenance and select reinstall?
 
  I cannot understand how all this results from upgrade, something had
  changed, the CA certificate installed on the host is probably not the CA
  certificate of the engine.
 
  - Original Message -
  From: Chris Smith whitehat...@gmail.com
  To: Alon Bar-Lev alo...@redhat.com, Users@ovirt.org
  Sent: Friday, April 19, 2013 1:45:23 AM
  Subject: Re: [Users] Certificates and PKI seem to be broken after yum
  update
 
  On Thu, Apr 18, 2013 at 6:44 PM, Chris Smith whitehat...@gmail.com
  wrote:
   I made a backup of the .truststore, and then followed the steps and
   then rebooted both the ovirt-engine and one of the hosts, and
   everything worked properly.
  
   If I run it again, or enter the wrong password it throws an error
   about the key store already existing, or that the password was wrong
   so I'm pretty sure it's good.
  
   vdsm.log on the host still shows:
  
   Traceback (most recent call last):
 File /usr/lib64/python2.7/SocketServer.py, line 582, in
   process_request_thread
   self.finish_request(request, client_address)
 File /usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py,
   line 66, in finish_request
   request.do_handshake()
 File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
   self._sslobj.do_handshake()
   SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
   routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
  
   engine.log on the host shows:
  
   2013-04-18 18:42:43,632 ERROR
   [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
   (QuartzScheduler_Worker-68) Failed to decryptData must start with zero
   2013-04-18 18:42:43,642 ERROR
   [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
   (QuartzScheduler_Worker-68) XML RPC error in command
   GetCapabilitiesVDS ( Vds: transporter ), the error was:
   java.util.concurrent.ExecutionException:
   java.lang.reflect.InvocationTargetException,
   SunCertPathBuilderException: unable to find valid certification path
   to requested target
  
  
   On Thu, Apr 18, 2013 at 4:06 AM, Alon Bar-Lev alo...@redhat.com wrote:
  
   You should ask these question in separate thread so people may pick
   them
   up.
  
   For the .truststore, try to remove it and then execute:
  
   # rm -f /etc/pki/ovirt-engine/.truststore
   # keytool -import -noprompt -trustcacerts -alias cacert -keypass mypass
   -file /etc/pki/ovirt-engine/certs/ca.der -keystore
   /etc/pki/ovirt-engine/.truststore -storepass mypass
   # chown ovirt:ovirt /etc/pki/ovirt-engine/.truststore
  
   It should recreate the truststore with the ca certificate you have.
  
   - Original Message -
   From: Chris Smith whitehat...@gmail.com
   To: Alon Bar-Lev alo...@redhat.com
   Cc: Users@ovirt.org
   Sent: Thursday, April 18, 2013 7:18:27 AM
   Subject: Re: [Users] Certificates and PKI seem to be broken after yum
   update
  
   If it would be easier than re-setting up the certificates, I'm also
   willing to just start over and rebuild, but I would like to export the
   VM's I have first.
   One of them is a spacewalk server, another runs DNS, and DHCP for my
   test network, and I have an asterisk server.  I would like to avoid
   having to re-create all of them.
  
   The VM's are up and running now, so I could export all of the
   configurations / backup the file systems, etc.
  
   Preferably I could export the VM's to an NFS export domain, or a
   mounted NFS share so that I can import them to the new storage domain,
   after I run engine-cleanup and get everything set back up.  Is there
   an easy way to do this?  Is it possible to create and attach an NFS
   export domain directly from the CLI without access to the ovirt
   manager without communication between the manager and hosts due to the
   pki issue?  Can I export the VM's directly from the hosts to a
   standard NFS share?
  
   Is there an equivalent xml and image file for the VM?
  
   My storage domain is iscsi and is served out from another

Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-18 Thread Chris Smith
Since I'm not able to reinstall the host from the ovirt-engine web
interface, as another thought I wanted to see if I could bring up a
third host and add it to the cluster.
I have a host Fedora 17 box ready to go but I can't add it to the
cluster.  It states that there are no available server in the cluster
to probe the new host.

What about approaching it from the other direction.  Would I be able
to stand up an ovirt-h node on the same hardware and then add it to
ovirt from the host itself, using the setup menu?

Could it then obtain spm status and bring the storage domain online?

On Thu, Apr 18, 2013 at 7:20 PM, Chris Smith whitehat...@gmail.com wrote:
 engine.log attached

 On Thu, Apr 18, 2013 at 7:11 PM, Alon Bar-Lev alo...@redhat.com wrote:
 Need to know precise error, please attach engine.log.


 - Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Alon Bar-Lev alo...@redhat.com
 Cc: Users@ovirt.org
 Sent: Friday, April 19, 2013 2:03:59 AM
 Subject: Re: [Users] Certificates and PKI seem to be broken after yum update

 So as of now, I can put the host into maintenance mode using the
 ovirt-engine web interface.  I can also try and activate it.  It
 states that the host was activated.   The host never actually comes up
 or contends for SPM status, and the data center never actually comes
 online.

 If I put the host into maintenance mode and try to reinstall it, it
 throws an error and size must be between 0 and 50.

 On Thu, Apr 18, 2013 at 6:51 PM, Alon Bar-Lev alo...@redhat.com wrote:
  I am not sure I understand the status.
 
  Everything is working or not.
  If not, what exactly fails?
  Why do you run it 'again'?
 
  What happens if you reinstall host? Go to maintenance and select 
  reinstall?
 
  I cannot understand how all this results from upgrade, something had
  changed, the CA certificate installed on the host is probably not the CA
  certificate of the engine.
 
  - Original Message -
  From: Chris Smith whitehat...@gmail.com
  To: Alon Bar-Lev alo...@redhat.com, Users@ovirt.org
  Sent: Friday, April 19, 2013 1:45:23 AM
  Subject: Re: [Users] Certificates and PKI seem to be broken after yum
  update
 
  On Thu, Apr 18, 2013 at 6:44 PM, Chris Smith whitehat...@gmail.com
  wrote:
   I made a backup of the .truststore, and then followed the steps and
   then rebooted both the ovirt-engine and one of the hosts, and
   everything worked properly.
  
   If I run it again, or enter the wrong password it throws an error
   about the key store already existing, or that the password was wrong
   so I'm pretty sure it's good.
  
   vdsm.log on the host still shows:
  
   Traceback (most recent call last):
 File /usr/lib64/python2.7/SocketServer.py, line 582, in
   process_request_thread
   self.finish_request(request, client_address)
 File /usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py,
   line 66, in finish_request
   request.do_handshake()
 File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
   self._sslobj.do_handshake()
   SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
   routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
  
   engine.log on the host shows:
  
   2013-04-18 18:42:43,632 ERROR
   [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
   (QuartzScheduler_Worker-68) Failed to decryptData must start with zero
   2013-04-18 18:42:43,642 ERROR
   [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
   (QuartzScheduler_Worker-68) XML RPC error in command
   GetCapabilitiesVDS ( Vds: transporter ), the error was:
   java.util.concurrent.ExecutionException:
   java.lang.reflect.InvocationTargetException,
   SunCertPathBuilderException: unable to find valid certification path
   to requested target
  
  
   On Thu, Apr 18, 2013 at 4:06 AM, Alon Bar-Lev alo...@redhat.com 
   wrote:
  
   You should ask these question in separate thread so people may pick
   them
   up.
  
   For the .truststore, try to remove it and then execute:
  
   # rm -f /etc/pki/ovirt-engine/.truststore
   # keytool -import -noprompt -trustcacerts -alias cacert -keypass 
   mypass
   -file /etc/pki/ovirt-engine/certs/ca.der -keystore
   /etc/pki/ovirt-engine/.truststore -storepass mypass
   # chown ovirt:ovirt /etc/pki/ovirt-engine/.truststore
  
   It should recreate the truststore with the ca certificate you have.
  
   - Original Message -
   From: Chris Smith whitehat...@gmail.com
   To: Alon Bar-Lev alo...@redhat.com
   Cc: Users@ovirt.org
   Sent: Thursday, April 18, 2013 7:18:27 AM
   Subject: Re: [Users] Certificates and PKI seem to be broken after yum
   update
  
   If it would be easier than re-setting up the certificates, I'm also
   willing to just start over and rebuild, but I would like to export 
   the
   VM's I have first.
   One of them is a spacewalk server, another runs DNS, and DHCP for my
   test network, and I have an asterisk server.  I would like to avoid
   having to re-create all

Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-17 Thread Chris Smith
I checked the .truststore on the ovirt engine, and it seems fine.

[root@reliant ovirt-engine]# ls -l .truststore
-rwxr-x---. 1 ovirt ovirt 918 Apr  6 21:56 .truststore

It's not zero bytes anyway.

It's also the same size as the .truststore in the ovirt engine backups.

[root@reliant ovirt-engine-backups]# find ./ -name .truststore -exec ls -l {} \;
-rwxr-x---. 1 ovirt ovirt 918 Aug 26  2012
./ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
-rwxr-x---. 1 root root 918 Mar 24 12:42
./ovirt-engine-2013_03_24_11_15_19/ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore

I haven't looked at the installCA.sh script yet.

On Mon, Apr 8, 2013 at 2:58 AM, Alon Bar-Lev alo...@redhat.com wrote:
 This error means that the /etc/pki/ovirt-engine/.truststore is unreadable or 
 does not contain the /etc/pki/ovirt-engine/ca.pem certificate.

 Unfortunately, the pki administration is weak in current implementation, you 
 can trace the installation script and checkout the calls to installCA.sh to 
 how to reproduce, please note that password are encrypted in database using 
 the private key locate in .keystore so if you are to re-generate anything 
 remember to keep the engine private key.

 However, if you succeed in login, the remaining problem you have is the 
 .truststore permissions and/or content.

 Regards,
 Alon Bar-Lev.

 - Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Alon Bar-Lev alo...@redhat.com
 Cc: Users@ovirt.org
 Sent: Monday, April 8, 2013 9:46:46 AM
 Subject: Re: [Users] Certificates and PKI seem to be broken after yum update

 After setting the .keystore owner and group owner to ovirt, and
 rebooting, I now have a new error in engine.log

 2013-04-08 02:39:16,787 ERROR
 [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
 (QuartzScheduler_Worker-95) Failed to decryptData must start with zero
 2013-04-08 02:39:16,845 ERROR
 [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
 (QuartzScheduler_Worker-95) XML RPC error in command
 GetCapabilitiesVDS ( Vds: transporter ), the error was:
 java.util.concurrent.ExecutionException:
 java.lang.reflect.InvocationTargetException,
 SunCertPathBuilderException: unable to find valid certification path
 to requested target

 Are there other files that may have been affected that I can also
 correct ownership or permissions on?

 On the host side, I get certificate unknown in vdsm.log

   File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
 self._sslobj.do_handshake()
 SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
 routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
 Thread-757809::ERROR::2013-04-08
 02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client
 ('172.16.23.8', 54489)
 Traceback (most recent call last):
   File /usr/lib64/python2.7/SocketServer.py, line 582, in
 process_request_thread
 self.finish_request(request, client_address)
   File /usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py,
 line 66, in finish_request
 request.do_handshake()
   File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
 self._sslobj.do_handshake()
 SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
 routines:SSL3_READ_BYTES:sslv3 alert certificate unknown

 Is there a procedure for just re-establishing PKI and certs for the
 engine and hosts?

 On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev alo...@redhat.com wrote:
 
  OK... you are running a very old version of engine (3.1).
 
  The upgrade did not upgraded into 3.2, so nothing as far as I know should
  have been changed.
 
  But the .keystore permissions is owned by root now, so some other package
  (maybe selinux-policy) changed permissions...
 
  The simplest way to test is to:
  # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1
  # chown -R ovirt:ovirt /etc/pki/ovirt-engine
 
  But if that file permissions was changed, I can only assume other files
  were also changes...
 
  Regards,
  Alon
 
  - Original Message -
  From: Chris Smith whitehat...@gmail.com
  To: Alon Bar-Lev alo...@redhat.com
  Cc: Users@ovirt.org
  Sent: Sunday, April 7, 2013 11:51:17 AM
  Subject: Re: [Users] Certificates and PKI seem to be broken after yum
  update
 
  I did a yum update and rebooted.
 
  engine-upgrade was run on 24-March
 
  When run now, it states that there are no updates available.
 
  [root@reliant ~]# engine-upgrade
  Loaded plugins: versionlock
  Checking for updates... (This may take several minutes)
  No updates available
 
 
  [root@reliant ovirt-engine]# cat
  ovirt-engine-upgrade_2013_03_24_12_04_06.log
  2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
  pgpass file, fetching DB host value
  2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
  pgpass file, fetching DB port value
  2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
  pgpass file, fetching DB admin value
  2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list

Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-17 Thread Chris Smith
If it would be easier than re-setting up the certificates, I'm also
willing to just start over and rebuild, but I would like to export the
VM's I have first.
One of them is a spacewalk server, another runs DNS, and DHCP for my
test network, and I have an asterisk server.  I would like to avoid
having to re-create all of them.

The VM's are up and running now, so I could export all of the
configurations / backup the file systems, etc.

Preferably I could export the VM's to an NFS export domain, or a
mounted NFS share so that I can import them to the new storage domain,
after I run engine-cleanup and get everything set back up.  Is there
an easy way to do this?  Is it possible to create and attach an NFS
export domain directly from the CLI without access to the ovirt
manager without communication between the manager and hosts due to the
pki issue?  Can I export the VM's directly from the hosts to a
standard NFS share?

Is there an equivalent xml and image file for the VM?

My storage domain is iscsi and is served out from another server over
4 bonded 1 Gbps copper links.



On Wed, Apr 17, 2013 at 11:46 PM, Chris Smith whitehat...@gmail.com wrote:
 I checked the .truststore on the ovirt engine, and it seems fine.

 [root@reliant ovirt-engine]# ls -l .truststore
 -rwxr-x---. 1 ovirt ovirt 918 Apr  6 21:56 .truststore

 It's not zero bytes anyway.

 It's also the same size as the .truststore in the ovirt engine backups.

 [root@reliant ovirt-engine-backups]# find ./ -name .truststore -exec ls -l {} 
 \;
 -rwxr-x---. 1 ovirt ovirt 918 Aug 26  2012
 ./ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
 -rwxr-x---. 1 root root 918 Mar 24 12:42
 ./ovirt-engine-2013_03_24_11_15_19/ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore

 I haven't looked at the installCA.sh script yet.

 On Mon, Apr 8, 2013 at 2:58 AM, Alon Bar-Lev alo...@redhat.com wrote:
 This error means that the /etc/pki/ovirt-engine/.truststore is unreadable or 
 does not contain the /etc/pki/ovirt-engine/ca.pem certificate.

 Unfortunately, the pki administration is weak in current implementation, you 
 can trace the installation script and checkout the calls to installCA.sh to 
 how to reproduce, please note that password are encrypted in database using 
 the private key locate in .keystore so if you are to re-generate anything 
 remember to keep the engine private key.

 However, if you succeed in login, the remaining problem you have is the 
 .truststore permissions and/or content.

 Regards,
 Alon Bar-Lev.

 - Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Alon Bar-Lev alo...@redhat.com
 Cc: Users@ovirt.org
 Sent: Monday, April 8, 2013 9:46:46 AM
 Subject: Re: [Users] Certificates and PKI seem to be broken after yum update

 After setting the .keystore owner and group owner to ovirt, and
 rebooting, I now have a new error in engine.log

 2013-04-08 02:39:16,787 ERROR
 [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
 (QuartzScheduler_Worker-95) Failed to decryptData must start with zero
 2013-04-08 02:39:16,845 ERROR
 [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
 (QuartzScheduler_Worker-95) XML RPC error in command
 GetCapabilitiesVDS ( Vds: transporter ), the error was:
 java.util.concurrent.ExecutionException:
 java.lang.reflect.InvocationTargetException,
 SunCertPathBuilderException: unable to find valid certification path
 to requested target

 Are there other files that may have been affected that I can also
 correct ownership or permissions on?

 On the host side, I get certificate unknown in vdsm.log

   File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
 self._sslobj.do_handshake()
 SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
 routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
 Thread-757809::ERROR::2013-04-08
 02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client
 ('172.16.23.8', 54489)
 Traceback (most recent call last):
   File /usr/lib64/python2.7/SocketServer.py, line 582, in
 process_request_thread
 self.finish_request(request, client_address)
   File /usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py,
 line 66, in finish_request
 request.do_handshake()
   File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
 self._sslobj.do_handshake()
 SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
 routines:SSL3_READ_BYTES:sslv3 alert certificate unknown

 Is there a procedure for just re-establishing PKI and certs for the
 engine and hosts?

 On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev alo...@redhat.com wrote:
 
  OK... you are running a very old version of engine (3.1).
 
  The upgrade did not upgraded into 3.2, so nothing as far as I know should
  have been changed.
 
  But the .keystore permissions is owned by root now, so some other package
  (maybe selinux-policy) changed permissions...
 
  The simplest way to test is to:
  # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1
  # chown -R

Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-08 Thread Chris Smith
After setting the .keystore owner and group owner to ovirt, and
rebooting, I now have a new error in engine.log

2013-04-08 02:39:16,787 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-95) Failed to decryptData must start with zero
2013-04-08 02:39:16,845 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
(QuartzScheduler_Worker-95) XML RPC error in command
GetCapabilitiesVDS ( Vds: transporter ), the error was:
java.util.concurrent.ExecutionException:
java.lang.reflect.InvocationTargetException,
SunCertPathBuilderException: unable to find valid certification path
to requested target

Are there other files that may have been affected that I can also
correct ownership or permissions on?

On the host side, I get certificate unknown in vdsm.log

  File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
Thread-757809::ERROR::2013-04-08
02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client
('172.16.23.8', 54489)
Traceback (most recent call last):
  File /usr/lib64/python2.7/SocketServer.py, line 582, in
process_request_thread
self.finish_request(request, client_address)
  File /usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py,
line 66, in finish_request
request.do_handshake()
  File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
routines:SSL3_READ_BYTES:sslv3 alert certificate unknown

Is there a procedure for just re-establishing PKI and certs for the
engine and hosts?

On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev alo...@redhat.com wrote:

 OK... you are running a very old version of engine (3.1).

 The upgrade did not upgraded into 3.2, so nothing as far as I know should 
 have been changed.

 But the .keystore permissions is owned by root now, so some other package 
 (maybe selinux-policy) changed permissions...

 The simplest way to test is to:
 # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1
 # chown -R ovirt:ovirt /etc/pki/ovirt-engine

 But if that file permissions was changed, I can only assume other files were 
 also changes...

 Regards,
 Alon

 - Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Alon Bar-Lev alo...@redhat.com
 Cc: Users@ovirt.org
 Sent: Sunday, April 7, 2013 11:51:17 AM
 Subject: Re: [Users] Certificates and PKI seem to be broken after yum update

 I did a yum update and rebooted.

 engine-upgrade was run on 24-March

 When run now, it states that there are no updates available.

 [root@reliant ~]# engine-upgrade
 Loaded plugins: versionlock
 Checking for updates... (This may take several minutes)
 No updates available


 [root@reliant ovirt-engine]# cat ovirt-engine-upgrade_2013_03_24_12_04_06.log
 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
 pgpass file, fetching DB host value
 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
 pgpass file, fetching DB port value
 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
 pgpass file, fetching DB admin value
 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list updates
 started
 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock started
 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock
 completed successfully
 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list
 of packages to upgrade
 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock started
 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
 command -- '/bin/rpm -q ovirt-engine'
 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
 ovirt-engine-3.1.0-4.fc17.noarch

 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
 command -- '/bin/rpm -q ovirt-engine-backend'
 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
 ovirt-engine-backend-3.1.0-4.fc17.noarch

 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
 command -- '/bin/rpm -q ovirt-engine-config'
 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
 ovirt-engine-config-3.1.0-4.fc17.noarch

 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
 command -- '/bin/rpm -q ovirt-engine-genericapi'
 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
 ovirt-engine-genericapi-3.1.0-4.fc17.noarch

 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
 2013-03-24

Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-08 Thread Alon Bar-Lev
This error means that the /etc/pki/ovirt-engine/.truststore is unreadable or 
does not contain the /etc/pki/ovirt-engine/ca.pem certificate.

Unfortunately, the pki administration is weak in current implementation, you 
can trace the installation script and checkout the calls to installCA.sh to how 
to reproduce, please note that password are encrypted in database using the 
private key locate in .keystore so if you are to re-generate anything remember 
to keep the engine private key.

However, if you succeed in login, the remaining problem you have is the 
.truststore permissions and/or content.

Regards,
Alon Bar-Lev.

- Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Alon Bar-Lev alo...@redhat.com
 Cc: Users@ovirt.org
 Sent: Monday, April 8, 2013 9:46:46 AM
 Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
 
 After setting the .keystore owner and group owner to ovirt, and
 rebooting, I now have a new error in engine.log
 
 2013-04-08 02:39:16,787 ERROR
 [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
 (QuartzScheduler_Worker-95) Failed to decryptData must start with zero
 2013-04-08 02:39:16,845 ERROR
 [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
 (QuartzScheduler_Worker-95) XML RPC error in command
 GetCapabilitiesVDS ( Vds: transporter ), the error was:
 java.util.concurrent.ExecutionException:
 java.lang.reflect.InvocationTargetException,
 SunCertPathBuilderException: unable to find valid certification path
 to requested target
 
 Are there other files that may have been affected that I can also
 correct ownership or permissions on?
 
 On the host side, I get certificate unknown in vdsm.log
 
   File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
 self._sslobj.do_handshake()
 SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
 routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
 Thread-757809::ERROR::2013-04-08
 02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client
 ('172.16.23.8', 54489)
 Traceback (most recent call last):
   File /usr/lib64/python2.7/SocketServer.py, line 582, in
 process_request_thread
 self.finish_request(request, client_address)
   File /usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py,
 line 66, in finish_request
 request.do_handshake()
   File /usr/lib64/python2.7/ssl.py, line 305, in do_handshake
 self._sslobj.do_handshake()
 SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
 routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
 
 Is there a procedure for just re-establishing PKI and certs for the
 engine and hosts?
 
 On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev alo...@redhat.com wrote:
 
  OK... you are running a very old version of engine (3.1).
 
  The upgrade did not upgraded into 3.2, so nothing as far as I know should
  have been changed.
 
  But the .keystore permissions is owned by root now, so some other package
  (maybe selinux-policy) changed permissions...
 
  The simplest way to test is to:
  # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1
  # chown -R ovirt:ovirt /etc/pki/ovirt-engine
 
  But if that file permissions was changed, I can only assume other files
  were also changes...
 
  Regards,
  Alon
 
  - Original Message -
  From: Chris Smith whitehat...@gmail.com
  To: Alon Bar-Lev alo...@redhat.com
  Cc: Users@ovirt.org
  Sent: Sunday, April 7, 2013 11:51:17 AM
  Subject: Re: [Users] Certificates and PKI seem to be broken after yum
  update
 
  I did a yum update and rebooted.
 
  engine-upgrade was run on 24-March
 
  When run now, it states that there are no updates available.
 
  [root@reliant ~]# engine-upgrade
  Loaded plugins: versionlock
  Checking for updates... (This may take several minutes)
  No updates available
 
 
  [root@reliant ovirt-engine]# cat
  ovirt-engine-upgrade_2013_03_24_12_04_06.log
  2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
  pgpass file, fetching DB host value
  2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
  pgpass file, fetching DB port value
  2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
  pgpass file, fetching DB admin value
  2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list updates
  started
  2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock started
  2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock
  completed successfully
  2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list
  of packages to upgrade
  2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock started
  2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
  command -- '/bin/rpm -q ovirt-engine'
  2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
  ovirt-engine-3.1.0-4.fc17.noarch
 
  2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
  2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
  2013-03-24 12:04

Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-07 Thread Alon Bar-Lev
How exactly did you upgrade?

Usually yum upgrade will not touch ovirt-engine packages as it is in yum 
version lock.
From which version to which version have you upgraded?
Have you run engine-upgrade utility?
If you did not, please run it.
If you did, please attach logs from /var/log/ovirt-engine/ovirt-engine-upgrade*

Thanks!

- Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Users@ovirt.org
 Sent: Sunday, April 7, 2013 5:09:46 AM
 Subject: [Users] Certificates and PKI seem to be broken after yum update
 
 I have lost the ability to manage the hosts or VM's using ovirt
 engine web interface after performing yum update on the ovirt-engine
 host, and on one Fedora 17 host.  The data center is offline, and I
 can't place the hosts into maintenance mode.  I don't think that there
 are any actions I can perform in the web interface at all.
 
 From the logs it seems that PKI is broken between the engine and the hosts.
 
 I am wondering how I can restore or re-generate all of the
 certificates and get the hosts communicating with the ovirt-engine
 again so that I can bring the data center back online.
 
 I found this page which deals with changing the engine hostname, and
 thus re-creating the certificates and keystore on the ovirt-engine
 node, and was wondering if this could help.  Could I follow this
 process but keep the same hostname for the ovirt-engine node?
 
 http://wiki.ovirt.org/How_to_change_engine_host_name
 
 Currently I have 3 VM's running on two hosts.  The VM's are up, but I
 can't do anything with them in ovirt-engine.
 
 
 Here's the latest activity from engine.log from the ovirt-engine node:
 
 2013-04-06 21:58:47,472 ERROR
 [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
 (QuartzScheduler_Worker-61) Failed to
 decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
 (Permission denied)
 2013-04-06 21:58:47,478 ERROR
 [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
 (QuartzScheduler_Worker-62) Can't load keystore from file
 /etc/pki/ovirt-engine/.keystore.: java.io.FileNotFoundException:
 /etc/pki/ovirt-engine/.keystore (Permission denied)
 at java.io.FileInputStream.open(Native Method)
 [rt.jar:1.7.0_09-icedtea]
 at java.io.FileInputStream.init(FileInputStream.java:138)
 [rt.jar:1.7.0_09-icedtea]
 at
 
 org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214)
 [engine-encryptutils.jar:]
 at
 
 org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139)
 [engine-encryptutils.jar:]
 at
 
 org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139)
 [engine-dal.jar:]
 at
 
 org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253)
 [engine-dal.jar:]
 at
 
 org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169)
 [engine-dal.jar:]
 at
 
 org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92)
 [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
 at
 
 org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653)
 [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
 at
 
 org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591)
 [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
 at
 
 org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641)
 [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
 at
 
 org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670)
 [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
 at
 
 org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702)
 [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
 at
 
 org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155)
 [engine-dal.jar:]
 at
 
 org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121)
 [engine-dal.jar:]
 at
 
 org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164)
 [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
 at
 
 org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124)
 [engine-dal.jar:]
 at
 
 org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75)
 [engine-dal.jar:]
 at
 
 org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66)
 [engine-dal.jar:]
 at
 
 org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58

Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-07 Thread Chris Smith
:04:28::DEBUG::engine-upgrade::325::root::
['ovirt-engine-3.1.0-4.fc17.noarch',
'ovirt-engine-backend-3.1.0-4.fc17.noarch',
'ovirt-engine-config-3.1.0-4.fc17.noarch',
'ovirt-engine-dbscripts-3.1.0-4.fc17.noarch',
'ovirt-engine-genericapi-3.1.0-4.fc17.noarch',
'ovirt-engine-notification-service-3.1.0-4.fc17.noarch',
'ovirt-engine-restapi-3.1.0-4.fc17.noarch',
'ovirt-engine-setup-3.1.0-4.fc17.noarch',
'ovirt-engine-tools-common-3.1.0-4.fc17.noarch',
'ovirt-engine-userportal-3.1.0-4.fc17.noarch',
'ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch',
'ovirt-image-uploader-3.1.0-0.git9c42c8.fc17.noarch',
'ovirt-iso-uploader-3.1.0-0.git1841d9.fc17.noarch',
'ovirt-log-collector-3.1.0-0.git10d719.fc17.noarch',
'vdsm-bootstrap-4.10.0-13.fc17.noarch']
2013-03-24 12:04:28::DEBUG::engine-upgrade::327::root:: Yum list
updated completed successfully
2013-03-24 12:04:28::DEBUG::engine-upgrade::609::root:: No updates available


Here's what's installed.

[root@reliant yum.repos.d]# yum list installed | grep ovirt
ovirt-engine.noarch3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-backend.noarch3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-cli.noarch3.2.0.5-1.fc17   @updates
ovirt-engine-config.noarch 3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-dbscripts.noarch  3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-genericapi.noarch 3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-notification-service.noarch
   3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-restapi.noarch3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-sdk.noarch3.2.0.2-1.fc17   @updates
ovirt-engine-setup.noarch  3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-tools-common.noarch   3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-userportal.noarch 3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-webadmin-portal.noarch3.1.0-4.fc17
 @ovirt-stable
ovirt-image-uploader.noarch3.1.0-0.git9c42c8.fc17
 @ovirt-stable
ovirt-iso-uploader.noarch  3.1.0-0.git1841d9.fc17
 @ovirt-stable
ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17
 @ovirt-stable
ovirt-release-fedora.noarch4-2
 @/ovirt-release-fedora.noarch

On Sun, Apr 7, 2013 at 2:16 AM, Alon Bar-Lev alo...@redhat.com wrote:
 How exactly did you upgrade?

 Usually yum upgrade will not touch ovirt-engine packages as it is in yum 
 version lock.
 From which version to which version have you upgraded?
 Have you run engine-upgrade utility?
 If you did not, please run it.
 If you did, please attach logs from 
 /var/log/ovirt-engine/ovirt-engine-upgrade*

 Thanks!

 - Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Users@ovirt.org
 Sent: Sunday, April 7, 2013 5:09:46 AM
 Subject: [Users] Certificates and PKI seem to be broken after yum update

 I have lost the ability to manage the hosts or VM's using ovirt
 engine web interface after performing yum update on the ovirt-engine
 host, and on one Fedora 17 host.  The data center is offline, and I
 can't place the hosts into maintenance mode.  I don't think that there
 are any actions I can perform in the web interface at all.

 From the logs it seems that PKI is broken between the engine and the hosts.

 I am wondering how I can restore or re-generate all of the
 certificates and get the hosts communicating with the ovirt-engine
 again so that I can bring the data center back online.

 I found this page which deals with changing the engine hostname, and
 thus re-creating the certificates and keystore on the ovirt-engine
 node, and was wondering if this could help.  Could I follow this
 process but keep the same hostname for the ovirt-engine node?

 http://wiki.ovirt.org/How_to_change_engine_host_name

 Currently I have 3 VM's running on two hosts.  The VM's are up, but I
 can't do anything with them in ovirt-engine.


 Here's the latest activity from engine.log from the ovirt-engine node:

 2013-04-06 21:58:47,472 ERROR
 [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
 (QuartzScheduler_Worker-61) Failed to
 decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
 (Permission denied)
 2013-04-06 21:58:47,478 ERROR
 [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
 (QuartzScheduler_Worker-62) Can't load keystore from file
 /etc/pki/ovirt-engine/.keystore.: java.io.FileNotFoundException:
 /etc/pki/ovirt-engine/.keystore (Permission denied)
 at java.io.FileInputStream.open(Native Method)
 [rt.jar:1.7.0_09-icedtea]
 at java.io.FileInputStream.init(FileInputStream.java:138)
 [rt.jar:1.7.0_09-icedtea]
 at
 
 org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214)
 [engine-encryptutils.jar:]
 at
 
 org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139)
 [engine-encryptutils.jar

Re: [Users] Certificates and PKI seem to be broken after yum update

2013-04-07 Thread Alon Bar-Lev

OK... you are running a very old version of engine (3.1).

The upgrade did not upgraded into 3.2, so nothing as far as I know should have 
been changed.

But the .keystore permissions is owned by root now, so some other package 
(maybe selinux-policy) changed permissions...

The simplest way to test is to:
# cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1
# chown -R ovirt:ovirt /etc/pki/ovirt-engine

But if that file permissions was changed, I can only assume other files were 
also changes...

Regards,
Alon

- Original Message -
 From: Chris Smith whitehat...@gmail.com
 To: Alon Bar-Lev alo...@redhat.com
 Cc: Users@ovirt.org
 Sent: Sunday, April 7, 2013 11:51:17 AM
 Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
 
 I did a yum update and rebooted.
 
 engine-upgrade was run on 24-March
 
 When run now, it states that there are no updates available.
 
 [root@reliant ~]# engine-upgrade
 Loaded plugins: versionlock
 Checking for updates... (This may take several minutes)
 No updates available
 
 
 [root@reliant ovirt-engine]# cat ovirt-engine-upgrade_2013_03_24_12_04_06.log
 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
 pgpass file, fetching DB host value
 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
 pgpass file, fetching DB port value
 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
 pgpass file, fetching DB admin value
 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list updates
 started
 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock started
 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock
 completed successfully
 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list
 of packages to upgrade
 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock started
 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
 command -- '/bin/rpm -q ovirt-engine'
 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
 ovirt-engine-3.1.0-4.fc17.noarch
 
 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
 command -- '/bin/rpm -q ovirt-engine-backend'
 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
 ovirt-engine-backend-3.1.0-4.fc17.noarch
 
 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
 command -- '/bin/rpm -q ovirt-engine-config'
 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
 ovirt-engine-config-3.1.0-4.fc17.noarch
 
 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
 command -- '/bin/rpm -q ovirt-engine-genericapi'
 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
 ovirt-engine-genericapi-3.1.0-4.fc17.noarch
 
 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
 command -- '/bin/rpm -q ovirt-engine-notification-service'
 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
 ovirt-engine-notification-service-3.1.0-4.fc17.noarch
 
 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
 command -- '/bin/rpm -q ovirt-engine-restapi'
 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
 ovirt-engine-restapi-3.1.0-4.fc17.noarch
 
 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
 command -- '/bin/rpm -q ovirt-engine-tools-common'
 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
 ovirt-engine-tools-common-3.1.0-4.fc17.noarch
 
 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
 command -- '/bin/rpm -q ovirt-engine-userportal'
 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
 ovirt-engine-userportal-3.1.0-4.fc17.noarch
 
 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
 command -- '/bin/rpm -q ovirt-engine-webadmin-portal'
 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
 ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch
 
 2013-03-24 12:04:27::DEBUG::common_utils::336

[Users] Certificates and PKI seem to be broken after yum update

2013-04-06 Thread Chris Smith
I have lost the ability to manage the hosts or VM's using ovirt
engine web interface after performing yum update on the ovirt-engine
host, and on one Fedora 17 host.  The data center is offline, and I
can't place the hosts into maintenance mode.  I don't think that there
are any actions I can perform in the web interface at all.

From the logs it seems that PKI is broken between the engine and the hosts.

I am wondering how I can restore or re-generate all of the
certificates and get the hosts communicating with the ovirt-engine
again so that I can bring the data center back online.

I found this page which deals with changing the engine hostname, and
thus re-creating the certificates and keystore on the ovirt-engine
node, and was wondering if this could help.  Could I follow this
process but keep the same hostname for the ovirt-engine node?

http://wiki.ovirt.org/How_to_change_engine_host_name

Currently I have 3 VM's running on two hosts.  The VM's are up, but I
can't do anything with them in ovirt-engine.


Here's the latest activity from engine.log from the ovirt-engine node:

2013-04-06 21:58:47,472 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-61) Failed to
decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
(Permission denied)
2013-04-06 21:58:47,478 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-62) Can't load keystore from file
/etc/pki/ovirt-engine/.keystore.: java.io.FileNotFoundException:
/etc/pki/ovirt-engine/.keystore (Permission denied)
at java.io.FileInputStream.open(Native Method) [rt.jar:1.7.0_09-icedtea]
at java.io.FileInputStream.init(FileInputStream.java:138)
[rt.jar:1.7.0_09-icedtea]
at 
org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214)
[engine-encryptutils.jar:]
at 
org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139)
[engine-encryptutils.jar:]
at 
org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169)
[engine-dal.jar:]
at 
org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121)
[engine-dal.jar:]
at 
org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31)
[engine-dal.jar:]
at 
org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219)
[engine-vdsbroker.jar:]
at 
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168)
[engine-utils.jar:]
at 
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107)
[engine-utils.jar:]
at 
org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215)
[engine-vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown