OK... you are running a very old version of engine (3.1). The upgrade did not upgraded into 3.2, so nothing as far as I know should have been changed.
But the .keystore permissions is owned by root now, so some other package (maybe selinux-policy) changed permissions... The simplest way to test is to: # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1 # chown -R ovirt:ovirt /etc/pki/ovirt-engine But if that file permissions was changed, I can only assume other files were also changes... Regards, Alon ----- Original Message ----- > From: "Chris Smith" <whitehat...@gmail.com> > To: "Alon Bar-Lev" <alo...@redhat.com> > Cc: Users@ovirt.org > Sent: Sunday, April 7, 2013 11:51:17 AM > Subject: Re: [Users] Certificates and PKI seem to be broken after yum update > > I did a yum update and rebooted. > > engine-upgrade was run on 24-March > > When run now, it states that there are no updates available. > > [root@reliant ~]# engine-upgrade > Loaded plugins: versionlock > Checking for updates... (This may take several minutes) > No updates available > > > [root@reliant ovirt-engine]# cat ovirt-engine-upgrade_2013_03_24_12_04_06.log > 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing > pgpass file, fetching DB host value > 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing > pgpass file, fetching DB port value > 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing > pgpass file, fetching DB admin value > 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list updates > started > 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock started > 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock > completed successfully > 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list > of packages to upgrade > 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock started > 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing > command --> '/bin/rpm -q ovirt-engine' > 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = > ovirt-engine-3.1.0-4.fc17.noarch > > 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = > 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 > 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing > command --> '/bin/rpm -q ovirt-engine-backend' > 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = > ovirt-engine-backend-3.1.0-4.fc17.noarch > > 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = > 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 > 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing > command --> '/bin/rpm -q ovirt-engine-config' > 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = > ovirt-engine-config-3.1.0-4.fc17.noarch > > 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = > 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 > 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing > command --> '/bin/rpm -q ovirt-engine-genericapi' > 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = > ovirt-engine-genericapi-3.1.0-4.fc17.noarch > > 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = > 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 > 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing > command --> '/bin/rpm -q ovirt-engine-notification-service' > 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = > ovirt-engine-notification-service-3.1.0-4.fc17.noarch > > 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = > 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 > 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing > command --> '/bin/rpm -q ovirt-engine-restapi' > 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = > ovirt-engine-restapi-3.1.0-4.fc17.noarch > > 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = > 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 > 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing > command --> '/bin/rpm -q ovirt-engine-tools-common' > 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = > ovirt-engine-tools-common-3.1.0-4.fc17.noarch > > 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = > 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 > 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing > command --> '/bin/rpm -q ovirt-engine-userportal' > 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = > ovirt-engine-userportal-3.1.0-4.fc17.noarch > > 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = > 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 > 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing > command --> '/bin/rpm -q ovirt-engine-webadmin-portal' > 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output = > ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch > > 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr = > 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0 > 2013-03-24 12:04:27::DEBUG::common_utils::286::root:: cmd = /bin/rpm > -q ovirt-engine ovirt-engine-backend ovirt-engine-config > ovirt-engine-genericapi ovirt-engine-notification-service > ovirt-engine-restapi ovirt-engine-tools-common ovirt-engine-userportal > ovirt-engine-webadmin-portal >> /etc/yum/pluginconf.d/versionlock.list > 2013-03-24 12:04:28::DEBUG::common_utils::291::root:: output = > 2013-03-24 12:04:28::DEBUG::common_utils::292::root:: stderr = > 2013-03-24 12:04:28::DEBUG::common_utils::293::root:: retcode = 0 > 2013-03-24 12:04:28::DEBUG::engine-upgrade::270::root:: Yum lock > completed successfully > 2013-03-24 12:04:28::DEBUG::engine-upgrade::320::root:: No packages > marked for update > 2013-03-24 12:04:28::DEBUG::engine-upgrade::324::root:: Installed packages: > 2013-03-24 12:04:28::DEBUG::engine-upgrade::325::root:: > ['ovirt-engine-3.1.0-4.fc17.noarch', > 'ovirt-engine-backend-3.1.0-4.fc17.noarch', > 'ovirt-engine-config-3.1.0-4.fc17.noarch', > 'ovirt-engine-dbscripts-3.1.0-4.fc17.noarch', > 'ovirt-engine-genericapi-3.1.0-4.fc17.noarch', > 'ovirt-engine-notification-service-3.1.0-4.fc17.noarch', > 'ovirt-engine-restapi-3.1.0-4.fc17.noarch', > 'ovirt-engine-setup-3.1.0-4.fc17.noarch', > 'ovirt-engine-tools-common-3.1.0-4.fc17.noarch', > 'ovirt-engine-userportal-3.1.0-4.fc17.noarch', > 'ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch', > 'ovirt-image-uploader-3.1.0-0.git9c42c8.fc17.noarch', > 'ovirt-iso-uploader-3.1.0-0.git1841d9.fc17.noarch', > 'ovirt-log-collector-3.1.0-0.git10d719.fc17.noarch', > 'vdsm-bootstrap-4.10.0-13.fc17.noarch'] > 2013-03-24 12:04:28::DEBUG::engine-upgrade::327::root:: Yum list > updated completed successfully > 2013-03-24 12:04:28::DEBUG::engine-upgrade::609::root:: No updates available > > > Here's what's installed. > > [root@reliant yum.repos.d]# yum list installed | grep ovirt > ovirt-engine.noarch 3.1.0-4.fc17 > @ovirt-stable > ovirt-engine-backend.noarch 3.1.0-4.fc17 > @ovirt-stable > ovirt-engine-cli.noarch 3.2.0.5-1.fc17 > @updates > ovirt-engine-config.noarch 3.1.0-4.fc17 > @ovirt-stable > ovirt-engine-dbscripts.noarch 3.1.0-4.fc17 > @ovirt-stable > ovirt-engine-genericapi.noarch 3.1.0-4.fc17 > @ovirt-stable > ovirt-engine-notification-service.noarch > 3.1.0-4.fc17 > @ovirt-stable > ovirt-engine-restapi.noarch 3.1.0-4.fc17 > @ovirt-stable > ovirt-engine-sdk.noarch 3.2.0.2-1.fc17 > @updates > ovirt-engine-setup.noarch 3.1.0-4.fc17 > @ovirt-stable > ovirt-engine-tools-common.noarch 3.1.0-4.fc17 > @ovirt-stable > ovirt-engine-userportal.noarch 3.1.0-4.fc17 > @ovirt-stable > ovirt-engine-webadmin-portal.noarch 3.1.0-4.fc17 > @ovirt-stable > ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17 > @ovirt-stable > ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17 > @ovirt-stable > ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17 > @ovirt-stable > ovirt-release-fedora.noarch 4-2 > @/ovirt-release-fedora.noarch > > On Sun, Apr 7, 2013 at 2:16 AM, Alon Bar-Lev <alo...@redhat.com> wrote: > > How exactly did you upgrade? > > > > Usually yum upgrade will not touch ovirt-engine packages as it is in yum > > version lock. > > From which version to which version have you upgraded? > > Have you run engine-upgrade utility? > > If you did not, please run it. > > If you did, please attach logs from > > /var/log/ovirt-engine/ovirt-engine-upgrade* > > > > Thanks! > > > > ----- Original Message ----- > >> From: "Chris Smith" <whitehat...@gmail.com> > >> To: Users@ovirt.org > >> Sent: Sunday, April 7, 2013 5:09:46 AM > >> Subject: [Users] Certificates and PKI seem to be broken after yum update > >> > >> I have lost the ability to manage the hosts or VM's using ovirt > >> engine web interface after performing yum update on the ovirt-engine > >> host, and on one Fedora 17 host. The data center is offline, and I > >> can't place the hosts into maintenance mode. I don't think that there > >> are any actions I can perform in the web interface at all. > >> > >> From the logs it seems that PKI is broken between the engine and the > >> hosts. > >> > >> I am wondering how I can restore or re-generate all of the > >> certificates and get the hosts communicating with the ovirt-engine > >> again so that I can bring the data center back online. > >> > >> I found this page which deals with changing the engine hostname, and > >> thus re-creating the certificates and keystore on the ovirt-engine > >> node, and was wondering if this could help. Could I follow this > >> process but keep the same hostname for the ovirt-engine node? > >> > >> http://wiki.ovirt.org/How_to_change_engine_host_name > >> > >> Currently I have 3 VM's running on two hosts. The VM's are up, but I > >> can't do anything with them in ovirt-engine. > >> > >> > >> Here's the latest activity from engine.log from the ovirt-engine node: > >> > >> 2013-04-06 21:58:47,472 ERROR > >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] > >> (QuartzScheduler_Worker-61) Failed to > >> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore > >> (Permission denied) > >> 2013-04-06 21:58:47,478 ERROR > >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] > >> (QuartzScheduler_Worker-62) Can't load keystore from file > >> "/etc/pki/ovirt-engine/.keystore".: java.io.FileNotFoundException: > >> /etc/pki/ovirt-engine/.keystore (Permission denied) > >> at java.io.FileInputStream.open(Native Method) > >> [rt.jar:1.7.0_09-icedtea] > >> at java.io.FileInputStream.<init>(FileInputStream.java:138) > >> [rt.jar:1.7.0_09-icedtea] > >> at > >> > >> org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214) > >> [engine-encryptutils.jar:] > >> at > >> > >> org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139) > >> [engine-encryptutils.jar:] > >> at > >> > >> org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139) > >> [engine-dal.jar:] > >> at > >> > >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253) > >> [engine-dal.jar:] > >> at > >> > >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169) > >> [engine-dal.jar:] > >> at > >> > >> org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) > >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > >> at > >> > >> org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653) > >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > >> at > >> > >> org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591) > >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > >> at > >> > >> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641) > >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > >> at > >> > >> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670) > >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > >> at > >> > >> org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702) > >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > >> at > >> > >> org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) > >> [engine-dal.jar:] > >> at > >> > >> org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) > >> [engine-dal.jar:] > >> at > >> > >> org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164) > >> [spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02] > >> at > >> > >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) > >> [engine-dal.jar:] > >> at > >> > >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) > >> [engine-dal.jar:] > >> at > >> > >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) > >> [engine-dal.jar:] > >> at > >> > >> org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) > >> [engine-dal.jar:] > >> at > >> > >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36) > >> [engine-dal.jar:] > >> at > >> > >> org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31) > >> [engine-dal.jar:] > >> at > >> > >> org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219) > >> [engine-vdsbroker.jar:] > >> at > >> > >> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) > >> [engine-utils.jar:] > >> at > >> > >> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) > >> [engine-utils.jar:] > >> at > >> > >> org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215) > >> [engine-vdsbroker.jar:] > >> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown > >> Source) [:1.7.0_09-icedtea] > >> at > >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> [rt.jar:1.7.0_09-icedtea] > >> at java.lang.reflect.Method.invoke(Method.java:601) > >> [rt.jar:1.7.0_09-icedtea] > >> at > >> > >> org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:64) > >> [engine-scheduler.jar:] > >> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > >> [quartz.jar:] > >> at > >> > >> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) > >> [quartz.jar:] > >> > >> 2013-04-06 21:58:47,576 ERROR > >> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] > >> (QuartzScheduler_Worker-61) XML RPC error in command > >> GetCapabilitiesVDS ( Vds: defiant ), the error was: > >> java.util.concurrent.ExecutionException: > >> java.lang.reflect.InvocationTargetException, > >> SSLPeerUnverifiedException: peer not authenticated > >> 2013-04-06 21:58:47,606 ERROR > >> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] > >> (QuartzScheduler_Worker-62) Failed to > >> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore > >> (Permission denied) > >> 2013-04-06 21:58:47,671 ERROR > >> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] > >> (QuartzScheduler_Worker-62) XML RPC error in command > >> GetCapabilitiesVDS ( Vds: transporter ), the error was: > >> java.util.concurrent.ExecutionException: > >> java.lang.reflect.InvocationTargetException, > >> SSLPeerUnverifiedException: peer not authenticated > >> > >> > >> Here's the message I seem to get over and over on the fedora 17 host in > >> vdsm.log > >> > >> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL > >> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown > >> Thread-562520::ERROR::2013-04-06 > >> 22:08:44,268::SecureXMLRPCServer::73::root::(handle_error) client > >> ('172.16.23.8', 36127) > >> Traceback (most recent call last): > >> File "/usr/lib64/python2.7/SocketServer.py", line 582, in > >> process_request_thread > >> self.finish_request(request, client_address) > >> File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", > >> line 66, in finish_request > >> request.do_handshake() > >> File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake > >> self._sslobj.do_handshake() > >> > >> I'm also wondering about the permission denied on the .keystore > >> directory. What should the permissions be? Here's what they are > >> currently. > >> > >> [root@reliant pki]# ls -ldZ /etc/pki/ovirt-engine/.keystore > >> -rwxr-x---. root root unconfined_u:object_r:cert_t:s0 > >> /etc/pki/ovirt-engine/.keystore > >> > >> I also seem to have a backup of the ovirt-engine directory at the time > >> the update was performed, but replacing ovirt-engine with the backup > >> does no good. > >> > >> I appreciate any assistance, and please let me know what other > >> information I can post to help with this. > >> > >> Thanks > >> _______________________________________________ > >> Users mailing list > >> Users@ovirt.org > >> http://lists.ovirt.org/mailman/listinfo/users > >> > _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users