Re: [ovirt-users] node unresponsive after reboot
To update the thread here after some offline discussion with Piotr, the problem was that vdsmd had stopped running and wasn't able to start. It wasn't reported in the logs that I could see, but I noticed it after trying to restart it at Piotr's suggestion, after running a 'systemctl'. A 'journalctl -xe' reported: [root@kvm-ldn-02 vdsm]# journalctl -xe -- Unit vdsmd.service has failed. -- -- The result is failed. May 05 15:15:15 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: vdsm: Running check_is_configured May 05 15:15:16 kvm-ldn-02.ldn.org sasldblistusers2[19047]: DIGEST-MD5 common mech free May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: Error: May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: One of the modules is not configured to work with VDSM. May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: To configure the module use the following: May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: 'vdsm-tool configure [--module module-name]'. May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: If all modules are not configured try to use: May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: 'vdsm-tool configure --force' May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: (The force flag will stop the module's service and start it May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: afterwards automatically to load the new configuration.) May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: Current revision of multipath.conf detected, preserving May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: libvirt is already configured for vdsm May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: Modules sebool are not configured May 05 15:15:16 kvm-ldn-02.ldn.org vdsmd_init_common.sh[18999]: vdsm: stopped during execute check_is_configured task (task returned with error code 1). May 05 15:15:16 kvm-ldn-02.ldn.org systemd[1]: vdsmd.service: control process exited, code=exited status=1 May 05 15:15:16 kvm-ldn-02.ldn.org systemd[1]: Failed to start Virtual Desktop Server Manager. -- Subject: Unit vdsmd.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- So I ran a 'vdsm-tool configure --module sebool', and then I was able to start vdsmd and mom-vdsm. This fixed the connectivity issue and the node is now back online. Many thanks to Piotr for all his help. Cheers, Cam On Mon, May 2, 2016 at 3:35 PM, Cam Mac wrote: > Hi Piotr, > > Attached are the vdsm log, the engine log and the supervdsm log. I've > attached them as a .tgz. > > I noticed it is complaining about configuring an interface in one of the > node logs. It shows as up in the engine web GUI though (and on the command > line). > > Thanks for the help. > > -Cam > > On Mon, May 2, 2016 at 1:38 PM, Piotr Kliczewski < > piotr.kliczew...@gmail.com> wrote: > >> Cam, >> >> Please provide engine and failing vdsm logs. >> >> Thanks, >> Piotr >> >> On Sun, May 1, 2016 at 4:05 PM, Cam Mac wrote: >> > Hi, >> > >> > I have a two node + engine ovirt setup, and I was having problems >> > doing a live migration between nodes. I looked in the vdsm logs and >> > noticed selinux errors, so I checked the selinux config, and both the >> > ovirt-engine host and one of the nodes had selinux disabled. So I >> > thought I would enable it on these two hosts, as it is officially >> > supported anyway. I started with the node, and put it into maintenance >> > mode, which interestingly, migrated the VMs off to the other node >> > without issue. After modifying the selinux config, I then rebooted >> > that node, which came back up. I then tried to activate the node but >> > it fails and marks it as unresponsive. >> > >> > --8<-- >> > >> > 2016-04-28 16:34:31,326 INFO >> > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp >> > Reactor) [29acb18b] Connecting to >> > kvm-ldn-02/172.16.23.12 >> > 2016-04-28 16:34:31,327 ERROR >> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Command >> > 'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02, >> > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', >> > hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa', >> > vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})' >> > execution failed: >> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection >> > failed >> > 2016-04-28 16:34:31,327 ERROR >> > [org.ovirt.engine.core.vdsbroker.HostMonitoring] >> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds >> > runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: >> > Connection failed >> > 2016-04-28 16:34:31,327 ERROR >> > [org.ovirt.engine.core.vdsbroker.HostMonitoring] >> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Exception: >> > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException
Re: [ovirt-users] node unresponsive after reboot
No problem Piotr, I will send you the full logs shortly. Thanks, Campbell On 5 May 2016 at 11:37, Piotr Kliczewski wrote: > Cam, > > It is really hard to understand partial logs. Please send me directly > full logs and I will analyze. > > Thanks, > Piotr > > On Mon, May 2, 2016 at 4:35 PM, Cam Mac wrote: >> Hi Piotr, >> >> Attached are the vdsm log, the engine log and the supervdsm log. I've >> attached them as a .tgz. >> >> I noticed it is complaining about configuring an interface in one of the >> node logs. It shows as up in the engine web GUI though (and on the command >> line). >> >> Thanks for the help. >> >> -Cam >> >> On Mon, May 2, 2016 at 1:38 PM, Piotr Kliczewski >> wrote: >>> >>> Cam, >>> >>> Please provide engine and failing vdsm logs. >>> >>> Thanks, >>> Piotr >>> >>> On Sun, May 1, 2016 at 4:05 PM, Cam Mac wrote: >>> > Hi, >>> > >>> > I have a two node + engine ovirt setup, and I was having problems >>> > doing a live migration between nodes. I looked in the vdsm logs and >>> > noticed selinux errors, so I checked the selinux config, and both the >>> > ovirt-engine host and one of the nodes had selinux disabled. So I >>> > thought I would enable it on these two hosts, as it is officially >>> > supported anyway. I started with the node, and put it into maintenance >>> > mode, which interestingly, migrated the VMs off to the other node >>> > without issue. After modifying the selinux config, I then rebooted >>> > that node, which came back up. I then tried to activate the node but >>> > it fails and marks it as unresponsive. >>> > >>> > --8<-- >>> > >>> > 2016-04-28 16:34:31,326 INFO >>> > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp >>> > Reactor) [29acb18b] Connecting to >>> > kvm-ldn-02/172.16.23.12 >>> > 2016-04-28 16:34:31,327 ERROR >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Command >>> > 'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02, >>> > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', >>> > hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa', >>> > vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})' >>> > execution failed: >>> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection >>> > failed >>> > 2016-04-28 16:34:31,327 ERROR >>> > [org.ovirt.engine.core.vdsbroker.HostMonitoring] >>> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds >>> > runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: >>> > Connection failed >>> > 2016-04-28 16:34:31,327 ERROR >>> > [org.ovirt.engine.core.vdsbroker.HostMonitoring] >>> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Exception: >>> > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: >>> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection >>> > failed >>> > at >>> > >>> > org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157) >>> > [vdsbroker.jar:] >>> > at >>> > >>> > org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120) >>> > [vdsbroker.jar:] >>> > at >>> > >>> > org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) >>> > [vdsbroker.jar:] >>> > at >>> > org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) >>> > [dal.jar:] >>> > at >>> > >>> > org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467) >>> > [vdsbroker.jar:] >>> > at >>> > >>> > org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652) >>> > [vdsbroker.jar:] >>> > at >>> > >>> > org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119) >>> > [vdsbroker.jar:] >>> > at >>> > >>> > org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84) >>> > [vdsbroker.jar:] >>> > at >>> > org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227) >>> > [vdsbroker.jar:] >>> > at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown >>> > Source) [:1.8.0_71] >>> > at >>> > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> > [rt.jar:1.8.0_71] >>> > at java.lang.reflect.Method.invoke(Method.java:497) >>> > [rt.jar:1.8.0_71] >>> > at >>> > >>> > org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81) >>> > [scheduler.jar:] >>> > at >>> > org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52) >>> > [scheduler.jar:] >>> > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>> > [quartz.jar:] >>> > at >>> > >>> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >>> > [quartz.jar:] >>> > Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: >>> > Con
Re: [ovirt-users] node unresponsive after reboot
Cam, It is really hard to understand partial logs. Please send me directly full logs and I will analyze. Thanks, Piotr On Mon, May 2, 2016 at 4:35 PM, Cam Mac wrote: > Hi Piotr, > > Attached are the vdsm log, the engine log and the supervdsm log. I've > attached them as a .tgz. > > I noticed it is complaining about configuring an interface in one of the > node logs. It shows as up in the engine web GUI though (and on the command > line). > > Thanks for the help. > > -Cam > > On Mon, May 2, 2016 at 1:38 PM, Piotr Kliczewski > wrote: >> >> Cam, >> >> Please provide engine and failing vdsm logs. >> >> Thanks, >> Piotr >> >> On Sun, May 1, 2016 at 4:05 PM, Cam Mac wrote: >> > Hi, >> > >> > I have a two node + engine ovirt setup, and I was having problems >> > doing a live migration between nodes. I looked in the vdsm logs and >> > noticed selinux errors, so I checked the selinux config, and both the >> > ovirt-engine host and one of the nodes had selinux disabled. So I >> > thought I would enable it on these two hosts, as it is officially >> > supported anyway. I started with the node, and put it into maintenance >> > mode, which interestingly, migrated the VMs off to the other node >> > without issue. After modifying the selinux config, I then rebooted >> > that node, which came back up. I then tried to activate the node but >> > it fails and marks it as unresponsive. >> > >> > --8<-- >> > >> > 2016-04-28 16:34:31,326 INFO >> > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp >> > Reactor) [29acb18b] Connecting to >> > kvm-ldn-02/172.16.23.12 >> > 2016-04-28 16:34:31,327 ERROR >> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Command >> > 'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02, >> > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', >> > hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa', >> > vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})' >> > execution failed: >> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection >> > failed >> > 2016-04-28 16:34:31,327 ERROR >> > [org.ovirt.engine.core.vdsbroker.HostMonitoring] >> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds >> > runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: >> > Connection failed >> > 2016-04-28 16:34:31,327 ERROR >> > [org.ovirt.engine.core.vdsbroker.HostMonitoring] >> > (DefaultQuartzScheduler_Worker-32) [ac322cb] Exception: >> > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: >> > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection >> > failed >> > at >> > >> > org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157) >> > [vdsbroker.jar:] >> > at >> > >> > org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120) >> > [vdsbroker.jar:] >> > at >> > >> > org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) >> > [vdsbroker.jar:] >> > at >> > org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) >> > [dal.jar:] >> > at >> > >> > org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467) >> > [vdsbroker.jar:] >> > at >> > >> > org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652) >> > [vdsbroker.jar:] >> > at >> > >> > org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119) >> > [vdsbroker.jar:] >> > at >> > >> > org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84) >> > [vdsbroker.jar:] >> > at >> > org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227) >> > [vdsbroker.jar:] >> > at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown >> > Source) [:1.8.0_71] >> > at >> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > [rt.jar:1.8.0_71] >> > at java.lang.reflect.Method.invoke(Method.java:497) >> > [rt.jar:1.8.0_71] >> > at >> > >> > org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81) >> > [scheduler.jar:] >> > at >> > org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52) >> > [scheduler.jar:] >> > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >> > [quartz.jar:] >> > at >> > >> > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >> > [quartz.jar:] >> > Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: >> > Connection failed >> > at >> > >> > org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157) >> > [vdsm-jsonrpc-java-client.jar:] >> > at >> > >> > org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:
Re: [ovirt-users] node unresponsive after reboot
Hi Piotr, Attached are the vdsm log, the engine log and the supervdsm log. I've attached them as a .tgz. I noticed it is complaining about configuring an interface in one of the node logs. It shows as up in the engine web GUI though (and on the command line). Thanks for the help. -Cam On Mon, May 2, 2016 at 1:38 PM, Piotr Kliczewski wrote: > Cam, > > Please provide engine and failing vdsm logs. > > Thanks, > Piotr > > On Sun, May 1, 2016 at 4:05 PM, Cam Mac wrote: > > Hi, > > > > I have a two node + engine ovirt setup, and I was having problems > > doing a live migration between nodes. I looked in the vdsm logs and > > noticed selinux errors, so I checked the selinux config, and both the > > ovirt-engine host and one of the nodes had selinux disabled. So I > > thought I would enable it on these two hosts, as it is officially > > supported anyway. I started with the node, and put it into maintenance > > mode, which interestingly, migrated the VMs off to the other node > > without issue. After modifying the selinux config, I then rebooted > > that node, which came back up. I then tried to activate the node but > > it fails and marks it as unresponsive. > > > > --8<-- > > > > 2016-04-28 16:34:31,326 INFO > > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp > > Reactor) [29acb18b] Connecting to > > kvm-ldn-02/172.16.23.12 > > 2016-04-28 16:34:31,327 ERROR > > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] > > (DefaultQuartzScheduler_Worker-32) [ac322cb] Command > > 'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02, > > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', > > hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa', > > vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})' > > execution failed: > > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection > > failed > > 2016-04-28 16:34:31,327 ERROR > > [org.ovirt.engine.core.vdsbroker.HostMonitoring] > > (DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds > > runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: > > Connection failed > > 2016-04-28 16:34:31,327 ERROR > > [org.ovirt.engine.core.vdsbroker.HostMonitoring] > > (DefaultQuartzScheduler_Worker-32) [ac322cb] Exception: > > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: > > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection > > failed > > at > > > org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157) > > [vdsbroker.jar:] > > at > > > org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120) > > [vdsbroker.jar:] > > at > > > org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) > > [vdsbroker.jar:] > > at > > org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) > > [dal.jar:] > > at > > > org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467) > > [vdsbroker.jar:] > > at > > > org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652) > > [vdsbroker.jar:] > > at > > > org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119) > > [vdsbroker.jar:] > > at > > > org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84) > > [vdsbroker.jar:] > > at > > org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227) > > [vdsbroker.jar:] > > at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown > > Source) [:1.8.0_71] > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > [rt.jar:1.8.0_71] > > at java.lang.reflect.Method.invoke(Method.java:497) > > [rt.jar:1.8.0_71] > > at > > > org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81) > > [scheduler.jar:] > > at > > org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52) > > [scheduler.jar:] > > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > > [quartz.jar:] > > at > > > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) > > [quartz.jar:] > > Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: > > Connection failed > > at > > > org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157) > > [vdsm-jsonrpc-java-client.jar:] > > at > > > org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114) > > [vdsm-jsonrpc-java-client.jar:] > > at > > org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73) > > [vdsm-jsonrpc-java-client.jar:] > > at > > > org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.(FutureMap.java:68) > > [vdsbroker.jar:] > > at > > > org.ovirt.engine.core.vdsbroker.
Re: [ovirt-users] node unresponsive after reboot
Cam, Please provide engine and failing vdsm logs. Thanks, Piotr On Sun, May 1, 2016 at 4:05 PM, Cam Mac wrote: > Hi, > > I have a two node + engine ovirt setup, and I was having problems > doing a live migration between nodes. I looked in the vdsm logs and > noticed selinux errors, so I checked the selinux config, and both the > ovirt-engine host and one of the nodes had selinux disabled. So I > thought I would enable it on these two hosts, as it is officially > supported anyway. I started with the node, and put it into maintenance > mode, which interestingly, migrated the VMs off to the other node > without issue. After modifying the selinux config, I then rebooted > that node, which came back up. I then tried to activate the node but > it fails and marks it as unresponsive. > > --8<-- > > 2016-04-28 16:34:31,326 INFO > [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp > Reactor) [29acb18b] Connecting to > kvm-ldn-02/172.16.23.12 > 2016-04-28 16:34:31,327 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] > (DefaultQuartzScheduler_Worker-32) [ac322cb] Command > 'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02, > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', > hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa', > vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})' > execution failed: > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection > failed > 2016-04-28 16:34:31,327 ERROR > [org.ovirt.engine.core.vdsbroker.HostMonitoring] > (DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds > runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: > Connection failed > 2016-04-28 16:34:31,327 ERROR > [org.ovirt.engine.core.vdsbroker.HostMonitoring] > (DefaultQuartzScheduler_Worker-32) [ac322cb] Exception: > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: > org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection > failed > at > org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) > [vdsbroker.jar:] > at > org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) > [dal.jar:] > at > org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227) > [vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown > Source) [:1.8.0_71] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [rt.jar:1.8.0_71] > at java.lang.reflect.Method.invoke(Method.java:497) > [rt.jar:1.8.0_71] > at > org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81) > [scheduler.jar:] > at > org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52) > [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > [quartz.jar:] > at > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) > [quartz.jar:] > Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: > Connection failed > at > org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157) > [vdsm-jsonrpc-java-client.jar:] > at > org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114) > [vdsm-jsonrpc-java-client.jar:] > at > org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73) > [vdsm-jsonrpc-java-client.jar:] > at > org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.(FutureMap.java:68) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) > [vdsbroker.jar:] > at > org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110) > [vdsbroker.jar:] > ... 14 more > > --8<-- > > Any ideas? > > Thanks for any help, > > Cam > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovi
[ovirt-users] node unresponsive after reboot
Hi, I have a two node + engine ovirt setup, and I was having problems doing a live migration between nodes. I looked in the vdsm logs and noticed selinux errors, so I checked the selinux config, and both the ovirt-engine host and one of the nodes had selinux disabled. So I thought I would enable it on these two hosts, as it is officially supported anyway. I started with the node, and put it into maintenance mode, which interestingly, migrated the VMs off to the other node without issue. After modifying the selinux config, I then rebooted that node, which came back up. I then tried to activate the node but it fails and marks it as unresponsive. --8<-- 2016-04-28 16:34:31,326 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [29acb18b] Connecting to kvm-ldn-02/172.16.23.12 2016-04-28 16:34:31,327 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) [ac322cb] Command 'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa', vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed 2016-04-28 16:34:31,327 ERROR [org.ovirt.engine.core.vdsbroker.HostMonitoring] (DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed 2016-04-28 16:34:31,327 ERROR [org.ovirt.engine.core.vdsbroker.HostMonitoring] (DefaultQuartzScheduler_Worker-32) [ac322cb] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) [:1.8.0_71] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_71] at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_71] at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81) [scheduler.jar:] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.(FutureMap.java:68) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110) [vdsbroker.jar:] ... 14 more --8<-- Any ideas? Thanks for any help, Cam ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] node unresponsive after reboot
Hi, I have a two node + engine ovirt setup, and I was having problems doing a live migration between nodes. I looked in the vdsm logs and noticed selinux errors, so I checked the selinux config, and both the ovirt-engine host and one of the nodes had selinux disabled. So I thought I would enable it on these two hosts, as it is officially supported anyway. I started with the node, and put it into maintenance mode, which interestingly, migrated the VMs off to the other node without issue. After modifying the selinux config, I then rebooted that node, which came back up. I then tried to activate the node but it fails and marks it as unresponsive. From the log: --8<-- 2016-04-28 16:34:31,326 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [29acb18b] Connecting to kvm-ldn-02.ldn.framestore.com/172.16.75.189 2016-04-28 16:34:31,327 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-32) [ac322cb] Command 'GetCapabilitiesVDSCommand(HostName = kvm-ldn-02, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='b12c0b80-d64d-42fd-8a55-94f92b9ca3aa', vds='Host[kvm-ldn-02,b12c0b80-d64d-42fd-8a55-94f92b9ca3aa]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed 2016-04-28 16:34:31,327 ERROR [org.ovirt.engine.core.vdsbroker.HostMonitoring] (DefaultQuartzScheduler_Worker-32) [ac322cb] Failure to refresh Vds runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed 2016-04-28 16:34:31,327 ERROR [org.ovirt.engine.core.vdsbroker.HostMonitoring] (DefaultQuartzScheduler_Worker-32) [ac322cb] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:652) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) [:1.8.0_71] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_71] at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_71] at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81) [scheduler.jar:] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:114) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:73) [vdsm-jsonrpc-java-client.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.(FutureMap.java:68) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:268) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:15) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110) [vdsbroker.jar:] ... 14 more --8<-- Any ideas? Thanks, Cam ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users