Re: [ovirt-users] oVirt Engine 4.0.5 and CentOS 7.3 Instability
Rogério, This bug is fixed in vdsm-jsonrpc-java-1.2.9. Do you have a really non-responsive host? From: Rogério Ceni Coelho <rogeriocenicoe...@gmail.com> Date: Friday 6 January 2017 at 21:35 To: Pavel Gashev <p...@acronis.com>, users <users@ovirt.org> Subject: Re: [ovirt-users] oVirt Engine 4.0.5 and CentOS 7.3 Instability I found https://bugzilla.redhat.com/show_bug.cgi?id=1393714 which seem to be the same problem that occour with me ... Em sex, 6 de jan de 2017 às 16:28, Rogério Ceni Coelho <rogeriocenicoe...@gmail.com<mailto:rogeriocenicoe...@gmail.com>> escreveu: Pavel, Take a look ... [root@prd-rbs-ovirt01-poa ~]# grep -i error /var/log/ovirt-engine/engine.log | grep -v org.ovirt.engine.core.vdsbroker.HostDevListByCapsVDSCommand | tail -50 2017-01-06 16:20:06,897 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [a6fc972] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm10-poa.rbs.com.br<http://prd-rbs-ovirt-kvm10-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='8b4bc7dc-af8c-4415-a2ef-bccc11ddf23a', vds='Host[prd-rbs-ovirt-kvm10-poa.rbs.com.br<http://prd-rbs-ovirt-kvm10-poa.rbs.com.br>,8b4bc7dc-af8c-4415-a2ef-bccc11ddf23a]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-01-06 16:20:06,899 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [19b85a77] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm17-poa.rbs.com.br<http://prd-rbs-ovirt-kvm17-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='795b917b-ea5b-499d-80c5-aa3aad4f2537', vds='Host[prd-rbs-ovirt-kvm17-poa.rbs.com.br<http://prd-rbs-ovirt-kvm17-poa.rbs.com.br>,795b917b-ea5b-499d-80c5-aa3aad4f2537]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-01-06 16:20:06,901 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [1bccfc8a] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm19-poa.rbs.com.br<http://prd-rbs-ovirt-kvm19-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='887b6e35-1fd1-4cd6-9e78-05bcab12a417', vds='Host[prd-rbs-ovirt-kvm19-poa.rbs.com.br<http://prd-rbs-ovirt-kvm19-poa.rbs.com.br>,887b6e35-1fd1-4cd6-9e78-05bcab12a417]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-01-06 16:23:26,020 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler1) [7771902b] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm06-poa.rbs.com.br<http://prd-rbs-ovirt-kvm06-poa.rbs.com.br>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='23b750ed-4ba6-4782-a9bd-c018c0f36e44', vds='Host[prd-rbs-ovirt-kvm06-poa.rbs.com.br<http://prd-rbs-ovirt-kvm06-poa.rbs.com.br>,23b750ed-4ba6-4782-a9bd-c018c0f36e44]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-01-06 16:23:26,043 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "e5f7a478-841d-413c-baa1-d63632be7748" 2017-01-06 16:24:32,480 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "75bf7e9a-fb5f-4253-bdf6-ff0fcbf8c876" 2017-01-06 16:24:32,485 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [35cc958c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM prd-rbs-ovirt-kvm06-poa.rbs.com.br<http://prd-rbs-ovirt-kvm06-poa.rbs.com.br> command failed: Heartbeat exceeded 2017-01-06 16:24:32,485 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [35cc958c] Command 'SpmStatusVDSCommand(HostName = prd-rbs-ovirt-kvm06-poa.rbs.com.br<http://prd-rbs-ovirt-kvm06-poa.rbs.com.br>, SpmStatusVDSCommandParameters:{runAsync='true', hostId='23b750ed-4ba6-4782-a9bd-c018c0f36e44', storagePoolId='0001-0001-0001-0001-0198'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-01-06 16:24:32,524 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [152cde35] Correlation ID: 152cde35, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center RBS. Setting Data Center status to Non Responsive (On host prd-rbs-ovirt-kvm06-poa.rbs.com.br<http://prd-rbs-ovirt-kvm06-poa.rbs.com.br>, Error: Network error during communication with the Host.). 2017-01-06 16:25:37,493 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: null 2017-01-06 16:25:37,494 ERROR [org.ovirt.vds
Re: [ovirt-users] oVirt Engine 4.0.5 and CentOS 7.3 Instability
.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] > Internal server error: null > 2017-01-06 16:27:49,661 ERROR > [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] > Internal server error: null > 2017-01-06 16:27:49,661 ERROR > [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] > Internal server error: null > 2017-01-06 16:27:49,663 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] > (DefaultQuartzScheduler6) [54a978b8] Command > 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm06-poa.rbs.com.br, > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', > hostId='23b750ed-4ba6-4782-a9bd-c018c0f36e44', vds='Host[ > prd-rbs-ovirt-kvm06-poa.rbs.com.br,23b750ed-4ba6-4782-a9bd-c018c0f36e44]'})' > execution failed: VDSGenericException: VDSNetworkException: Heartbeat > exceeded > 2017-01-06 16:27:49,663 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] > (DefaultQuartzScheduler9) [27c382ce] Command > 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm14-poa.rbs.com.br, > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', > hostId='60997c2f-3ba0-4911-928a-b70f640f30b8', vds='Host[ > prd-rbs-ovirt-kvm14-poa.rbs.com.br,60997c2f-3ba0-4911-928a-b70f640f30b8]'})' > execution failed: VDSGenericException: VDSNetworkException: Heartbeat > exceeded > 2017-01-06 16:27:49,666 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] > (DefaultQuartzScheduler6) [54a978b8] Command > 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm03-poa.rbs.com.br, > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', > hostId='f7842244-646c-400a-9736-f8d4aa9b1cef', vds='Host[ > prd-rbs-ovirt-kvm03-poa.rbs.com.br,f7842244-646c-400a-9736-f8d4aa9b1cef]'})' > execution failed: VDSGenericException: VDSNetworkException: Heartbeat > exceeded > 2017-01-06 16:27:49,666 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] > (DefaultQuartzScheduler7) [68fa51f0] Command > 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm11-poa.rbs.com.br, > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', > hostId='d3c5cf1f-b203-4818-9767-5249387e3553', vds='Host[ > prd-rbs-ovirt-kvm11-poa.rbs.com.br,d3c5cf1f-b203-4818-9767-5249387e3553]'})' > execution failed: VDSGenericException: VDSNetworkException: Heartbeat > exceeded > 2017-01-06 16:27:49,667 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] > (DefaultQuartzScheduler11) [2d5a3838] Command > 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm19-poa.rbs.com.br, > VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', > hostId='887b6e35-1fd1-4cd6-9e78-05bcab12a417', vds='Host[ > prd-rbs-ovirt-kvm19-poa.rbs.com.br,887b6e35-1fd1-4cd6-9e78-05bcab12a417]'})' > execution failed: VDSGenericException: VDSNetworkException: Heartbeat > exceeded > 2017-01-06 16:27:49,682 ERROR > [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able > to update response for "edd8b375-4598-4671-a2cf-0d62fc9bdd0a" > > > Em sex, 6 de jan de 2017 às 15:16, Rogério Ceni Coelho < > rogeriocenicoe...@gmail.com> escreveu: > > After apply the new vdsm-json package ? Yes, restart hole vm. :-( > > [root@prd-rbs-ovirt01-poa ~]# rpm -q > vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch > vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch > > [root@prd-rbs-ovirt01-poa ~]# rpm -qi > vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch > Name: vdsm-jsonrpc-java > Version : 1.2.10 > Release : 1.20161211091707.gitaf70e3f.el7.centos > Architecture: noarch > Install Date: Fri Jan 6 09:57:25 2017 > Group : Development/Libraries > Size: 144305 > License : LGPLv2+ > Signature : (none) > Source RPM : > vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.src.rpm > Build Date : Sun Dec 11 07:17:40 2016 > Build Host : vm0065.workers-phx.ovirt.org > Relocations : (not relocatable) > URL : http://www.ovirt.org > Summary : JsonRpc java client (vdsm-jsonrpc-java) for oVirt > Description : > vdsm jsonrpc java > [root@prd-rbs-ovirt01-poa ~]# uptime > 15:15:11 up 19 min, 1 user, load average: 0.50, 0.46, 0.37 > [root@prd-rbs-ovirt01-poa ~]# last | head > rogerio_ pts/010.40.134.150 Fri Jan 6 15:06 still logged in > > reboot system boot 3.10.0-514.2.2.e Fri Jan 6 14:56 - 15:15 (00:19) > > rogerio_ pts/010.40.134.150Fri Jan 6 14:46 - 14:55 (00:08) > > reboot system boot 3.10.0-514.2.2.e Fri Jan 6 10:11 - 14:55 (04:44) > > rogerio_ pts/010.40.134.150Fri Jan 6 09:56 - 10:10 (00:14) > > reboot syst
Re: [ovirt-users] oVirt Engine 4.0.5 and CentOS 7.3 Instability
AndVdsVDSCommandParametersBase:{runAsync='true', hostId='f7842244-646c-400a-9736-f8d4aa9b1cef', vds='Host[ prd-rbs-ovirt-kvm03-poa.rbs.com.br,f7842244-646c-400a-9736-f8d4aa9b1cef]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-01-06 16:27:49,666 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [68fa51f0] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm11-poa.rbs.com.br, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='d3c5cf1f-b203-4818-9767-5249387e3553', vds='Host[ prd-rbs-ovirt-kvm11-poa.rbs.com.br,d3c5cf1f-b203-4818-9767-5249387e3553]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-01-06 16:27:49,667 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler11) [2d5a3838] Command 'GetAllVmStatsVDSCommand(HostName = prd-rbs-ovirt-kvm19-poa.rbs.com.br, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='887b6e35-1fd1-4cd6-9e78-05bcab12a417', vds='Host[ prd-rbs-ovirt-kvm19-poa.rbs.com.br,887b6e35-1fd1-4cd6-9e78-05bcab12a417]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-01-06 16:27:49,682 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "edd8b375-4598-4671-a2cf-0d62fc9bdd0a" Em sex, 6 de jan de 2017 às 15:16, Rogério Ceni Coelho < rogeriocenicoe...@gmail.com> escreveu: > After apply the new vdsm-json package ? Yes, restart hole vm. :-( > > [root@prd-rbs-ovirt01-poa ~]# rpm -q > vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch > vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch > > [root@prd-rbs-ovirt01-poa ~]# rpm -qi > vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch > Name: vdsm-jsonrpc-java > Version : 1.2.10 > Release : 1.20161211091707.gitaf70e3f.el7.centos > Architecture: noarch > Install Date: Fri Jan 6 09:57:25 2017 > Group : Development/Libraries > Size: 144305 > License : LGPLv2+ > Signature : (none) > Source RPM : > vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.src.rpm > Build Date : Sun Dec 11 07:17:40 2016 > Build Host : vm0065.workers-phx.ovirt.org > Relocations : (not relocatable) > URL : http://www.ovirt.org > Summary : JsonRpc java client (vdsm-jsonrpc-java) for oVirt > Description : > vdsm jsonrpc java > [root@prd-rbs-ovirt01-poa ~]# uptime > 15:15:11 up 19 min, 1 user, load average: 0.50, 0.46, 0.37 > [root@prd-rbs-ovirt01-poa ~]# last | head > rogerio_ pts/010.40.134.150Fri Jan 6 15:06 still logged in > > reboot system boot 3.10.0-514.2.2.e Fri Jan 6 14:56 - 15:15 (00:19) > > rogerio_ pts/010.40.134.150Fri Jan 6 14:46 - 14:55 (00:08) > > reboot system boot 3.10.0-514.2.2.e Fri Jan 6 10:11 - 14:55 (04:44) > > rogerio_ pts/010.40.134.150Fri Jan 6 09:56 - 10:10 (00:14) > > reboot system boot 3.10.0-514.2.2.e Fri Jan 6 09:55 - 10:11 (00:15) > > rogerio_ pts/010.40.134.150Fri Jan 6 09:53 - 09:53 (00:00) > > reboot system boot 3.10.0-514.2.2.e Thu Jan 5 10:00 - 10:11 (1+00:10) > > rogerio_ pts/010.40.134.150Thu Jan 5 09:59 - 09:59 (00:00) > > reboot system boot 3.10.0-514.2.2.e Wed Jan 4 14:53 - 10:11 (1+19:18) > > [root@prd-rbs-ovirt01-poa ~]# > > > Em sex, 6 de jan de 2017 às 15:12, Pavel Gashev <p...@acronis.com> > escreveu: > > Rogério, > > > > Did you restart the engine? > > > > *From: *Rogério Ceni Coelho <rogeriocenicoe...@gmail.com> > *Date: *Friday 6 January 2017 at 19:56 > *To: *Pavel Gashev <p...@acronis.com>, users <users@ovirt.org> > *Subject: *Re: [ovirt-users] oVirt Engine 4.0.5 and CentOS 7.3 Instability > > > > Pavel and oVirt Admins, > > > > Problem still the same ... Take a look ... > > > > [image: asted1] > > [root@prd-rbs-ovirt01-poa ~]# tail -f /var/log/ovirt-engine/engine.log > > 2017-01-06 14:24:48,145 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] > (org.ovirt.thread.pool-6-thread-34) [] Host ' > prd-rbs-ovirt-kvm19-poa.rbs.com.br' is not responding. It will stay in > Connecting state for a grace period of 62 seconds and after that an attempt > to fence the host will be issued. > > 2017-01-06 14:24:48,168 WARN > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (org.ovirt.thread.pool-6-thread-34) [] Correlation ID: null, Call Stack: > null, Custom Event ID: -1, Message: Host > prd-rbs-ovirt-kvm19-poa.rbs.com.br is not responding. It will stay in > Connecting state for a grace period of 62
Re: [ovirt-users] oVirt Engine 4.0.5 and CentOS 7.3 Instability
Hi Pavel and oVirt Elves, Today, I install vdsm-jsonrpc-java. Take a look. Let´s see if problem are solve. Thanks again. [root@hlg-rbs-ovirt01-poa ~]# rpm -qa | grep -i vdsm-jsonrpc vdsm-jsonrpc-java-1.2.7-1.el7.centos.noarch [root@hlg-rbs-ovirt01-poa ~]# wget http://resources.ovirt.org/pub/ovirt-4.0-snapshot/rpm/el7/noarch/vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm --2017-01-05 17:52:51-- http://resources.ovirt.org/pub/ovirt-4.0-snapshot/rpm/el7/noarch/vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm Resolving resources.ovirt.org (resources.ovirt.org)... 66.187.230.28 Connecting to resources.ovirt.org (resources.ovirt.org)|66.187.230.28|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 123420 (121K) [application/x-rpm] Saving to: 'vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm' 100%[=>] 123,420 202KB/s in 0.6s 2017-01-05 17:52:52 (202 KB/s) - 'vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm' saved [123420/123420] [root@hlg-rbs-ovirt01-poa ~]# ll vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm -rw-r--r--. 1 root root 123420 Jan 4 22:39 vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm [root@hlg-rbs-ovirt01-poa ~]# rpm -Uvh vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm Preparing... # [100%] Updating / installing... 1:vdsm-jsonrpc-java-1.2.10-1.201612# [ 50%] Cleaning up / removing... 2:vdsm-jsonrpc-java-1.2.7-1.el7.cen# [100%] [root@hlg-rbs-ovirt01-poa ~]# Em qua, 4 de jan de 2017 às 17:17, Rogério Ceni Coelho < rogeriocenicoe...@gmail.com> escreveu: > Wonderfull ... Seems to be the original problem with me on 4.0.5 ... I > will try this and do some feedback about soon ... > > Thanks > > Em qua, 4 de jan de 2017 às 17:08, Pavel Gashev <p...@acronis.com> > escreveu: > > Rogério, > > > > It looks like https://bugzilla.redhat.com/show_bug.cgi?id=1401585 > > > > Please try to install the following package, and restart the engine. > > > http://resources.ovirt.org/pub/ovirt-4.0-snapshot/rpm/el7/noarch/vdsm-jsonrpc-java-1.2.10-1.20161211091707.gitaf70e3f.el7.centos.noarch.rpm > > > > > > *From: *<users-boun...@ovirt.org> on behalf of Rogério Ceni Coelho < > rogeriocenicoe...@gmail.com> > *Date: *Monday 2 January 2017 at 21:51 > *To: *users <users@ovirt.org> > *Subject: *[ovirt-users] oVirt Engine 4.0.5 and CentOS 7.3 Instability > > > > Hi oVirt Gurus, > > > > Happy new year to everyone !!! > > > > I update oVirt Engine to 4.0.5 from 4.0.4 and Centos to 7.3 from 7.2 last > week and after that I have instability four times. Every time ovirt engine > seems to loose communication with one or more node servers like this image > below. Every time I rebooted oVirt engine server and everything came back > to normal. > > > > Anyone with this kind of problem ??? > > > > [image: asted1] > > > > After reboot : > > > > [image: asted2] > > > > > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users