[ovirt-users] Re: Unassigned hosts
Hi Artur Hope you are well, please see below, this after I restarted the engine: host: [root@ovirt-aa-1-21:~]↥ # tcpdump -i ovirtmgmt -c 1000 -nnvvS dst ovirt-engine-aa-1-01 tcpdump: listening on ovirtmgmt, link-type EN10MB (Ethernet), capture size 262144 bytes 2020-08-07 12:09:32.553543 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.140.220.111 is-at 00:25:b5:04:00:25, length 28 2020-08-07 12:10:05.584594 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 172.140.220.111.54321 > 172.140.220.23.56202: Flags [S.], cksum 0x5cd5 (incorrect -> 0xc8ca), seq 4036072905, ack 3265413231, win 28960, options [mss 1460,sackOK,TS val 3039504636 ecr 341411251,nop,wscale 7], length 0 2020-08-07 12:10:10.589276 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.140.220.111 is-at 00:25:b5:04:00:25, length 28 2020-08-07 12:10:15.596230 IP (tos 0x0, ttl 64, id 48438, offset 0, flags [DF], proto TCP (6), length 52) 172.140.220.111.54321 > 172.140.220.23.56202: Flags [F.], cksum 0x5ccd (incorrect -> 0x40b8), seq 4036072906, ack 3265413231, win 227, options [nop,nop,TS val 3039514647 ecr 341411251], length 0 2020-08-07 12:10:20.596429 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.140.220.23 tell 172.140.220.111, length 28 2020-08-07 12:10:20.663699 IP (tos 0x0, ttl 64, id 64726, offset 0, flags [DF], proto TCP (6), length 40) 172.140.220.111.54321 > 172.140.220.23.56202: Flags [R], cksum 0x1d20 (correct), seq 4036072907, win 0, length 0 engine [root@ovirt-engine-aa-1-01 ~]# tcpdump -i eth0 -c 1000 -nnvvS src ovirt-aa-1-21 tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 2020-08-07 12:09:31.891242 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 172.140.220.111.54321 > 172.140.220.23.56202: Flags [S.], cksum 0xc8ca (correct), seq 4036072905, ack 3265413231, win 28960, options [mss 1460,sackOK,TS val 3039504636 ecr 341411251,nop,wscale 7], length 0 2020-08-07 12:09:36.895502 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.140.220.111 is-at 00:25:b5:04:00:25, length 42 2020-08-07 12:09:41.901981 IP (tos 0x0, ttl 64, id 48438, offset 0, flags [DF], proto TCP (6), length 52) 172.140.220.111.54321 > 172.140.220.23.56202: Flags [F.], cksum 0x40b8 (correct), seq 4036072906, ack 3265413231, win 227, options [nop,nop,TS val 3039514647 ecr 341411251], length 0 2020-08-07 12:09:46.901681 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.140.220.23 tell 172.140.220.111, length 42 2020-08-07 12:09:46.968911 IP (tos 0x0, ttl 64, id 64726, offset 0, flags [DF], proto TCP (6), length 40) 172.140.220.111.54321 > 172.140.220.23.56202: Flags [R], cksum 0x1d20 (correct), seq 4036072907, win 0, length 0 Regards Nar On Fri, 7 Aug 2020 at 11:54, Artur Socha wrote: > Hi Nardus, > There is one more thing to be checked. > > 1) could you check if there are any packets sent from the affected host to > the engine? > on host: > # outgoing traffic > sudo tcpdump -i -c 1000 -nnvvS dst > > > 2) same the other way round. Check if there are packets received on engine > side from affected host > on engine: > # incoming traffic > sudo tcpdump -i -c 1000 -nnvvS src > > > Artur > > > On Thu, Aug 6, 2020 at 4:51 PM Artur Socha wrote: > >> Thanks Nardus, >> After a quick look I found what I was suspecting - there are way too many >> threads in Blocked state. I don't know yet the reason but this is very >> helpful. I'll let you know about the findings/investigation. Meanwhile, you >> may try restarting the engine as (a very brute and ugly) workaround). >> You may try to setup slightly bigger thread pool - may save you some time >> until the next hiccup. However, please be aware that this may come with the >> cost in memory usage and higher cpu usage (due to increased context >> switching) >> Here are some docs: >> >> # Specify the thread pool size for jboss managed scheduled executor service >> used by commands to periodically execute >> # methods. It is generally not necessary to increase the number of threads >> in this thread pool. To change the value >> # permanently create a conf file 99-engine-scheduled-thread-pool.conf in >> /etc/ovirt-engine/engine.conf.d/ >> ENGINE_SCHEDULED_THREAD_POOL_SIZE=100 >> >> >> A. >> >> >> On Thu, Aug 6, 2020 at 4:19 PM Nardus Geldenhuys >> wrote: >> >>> Hi Artur >>> >>> Please find attached, also let me know if I need to rerun. They 5 min >>> apart >>> >>> [root@engine-aa-1-01 ovirt-engine]# ps -ef | grep jboss | grep -v grep >>> | awk '{ print $2 }' >>> 27390 >>> [root@engine-aa-1-01 ovirt-engine]# jstack -F 27390 > >>> your_engine_thread_dump_1.txt >>> [root@engine-aa-1-01 ovirt-engine]# jstack -F 27390 > >>> your_engine_thread_dump_2.txt >>> [root@engine-aa-1-01 ovirt-engine]# jstack -F 27390 > >>> your_engine_thread_dump_3.txt >>> >>> Regards >>> >>> Nar >>> >>> On Thu, 6 Aug 2020 at 15:55, Artur Socha wrote: >>> Sure thing. On engine host please find jboss pid.
[ovirt-users] Re: Unassigned hosts
Hi Nardus, There is one more thing to be checked. 1) could you check if there are any packets sent from the affected host to the engine? on host: # outgoing traffic sudo tcpdump -i -c 1000 -nnvvS dst 2) same the other way round. Check if there are packets received on engine side from affected host on engine: # incoming traffic sudo tcpdump -i -c 1000 -nnvvS src Artur On Thu, Aug 6, 2020 at 4:51 PM Artur Socha wrote: > Thanks Nardus, > After a quick look I found what I was suspecting - there are way too many > threads in Blocked state. I don't know yet the reason but this is very > helpful. I'll let you know about the findings/investigation. Meanwhile, you > may try restarting the engine as (a very brute and ugly) workaround). > You may try to setup slightly bigger thread pool - may save you some time > until the next hiccup. However, please be aware that this may come with the > cost in memory usage and higher cpu usage (due to increased context > switching) > Here are some docs: > > # Specify the thread pool size for jboss managed scheduled executor service > used by commands to periodically execute > # methods. It is generally not necessary to increase the number of threads in > this thread pool. To change the value > # permanently create a conf file 99-engine-scheduled-thread-pool.conf in > /etc/ovirt-engine/engine.conf.d/ > ENGINE_SCHEDULED_THREAD_POOL_SIZE=100 > > > A. > > > On Thu, Aug 6, 2020 at 4:19 PM Nardus Geldenhuys > wrote: > >> Hi Artur >> >> Please find attached, also let me know if I need to rerun. They 5 min >> apart >> >> [root@engine-aa-1-01 ovirt-engine]# ps -ef | grep jboss | grep -v grep >> | awk '{ print $2 }' >> 27390 >> [root@engine-aa-1-01 ovirt-engine]# jstack -F 27390 > >> your_engine_thread_dump_1.txt >> [root@engine-aa-1-01 ovirt-engine]# jstack -F 27390 > >> your_engine_thread_dump_2.txt >> [root@engine-aa-1-01 ovirt-engine]# jstack -F 27390 > >> your_engine_thread_dump_3.txt >> >> Regards >> >> Nar >> >> On Thu, 6 Aug 2020 at 15:55, Artur Socha wrote: >> >>> Sure thing. >>> On engine host please find jboss pid. You can use this command: >>> >>> ps -ef | grep jboss | grep -v grep | awk '{ print $2 }' >>> >>> or jps tool from jdk. Sample output on my dev environment is: >>> >>> ± % jps >>>!2860 >>> 64853 jboss-modules.jar >>> 196217 Jps >>> >>> Then use jstack from jdk: >>> jstack > your_engine_thread_dump.txt >>> 2 or 3 dumps taken in approximately 5 minutes intervals would be even >>> more useful. >>> >>> Here you can find even more options >>> https://www.baeldung.com/java-thread-dump >>> >>> Artur >>> >>> On Thu, Aug 6, 2020 at 3:15 PM Nardus Geldenhuys >>> wrote: >>> Hi Can create thread dump, please send details on howto. Regards Nardus On Thu, 6 Aug 2020 at 14:17, Artur Socha wrote: > Hi Nardus, > You might have hit an issue I have been hunting for some time ( [1] > and [2] ). > [1] could not be properly resolved because at a time was not able to > recreate an issue on dev setup. > I suspect [2] is related. > > Would you be able to prepare a thread dump from your engine instance? > Additionally, please check for potential libvirt errors/warnings. > Can you also paste the output of: > sudo yum list installed | grep vdsm > sudo yum list installed | grep ovirt-engine > sudo yum list installed | grep libvirt > > Usually, according to previous reports, restarting the engine helps to > restore connectivity with hosts ... at least for some time. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1845152 > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1846338 > > regards, > Artur > > > > On Thu, Aug 6, 2020 at 8:01 AM Nardus Geldenhuys > wrote: > >> Also see this in engine: >> >> Aug 6, 2020, 7:37:17 AM >> VDSM someserver command Get Host Capabilities failed: Message timeout >> which can be caused by communication issues >> >> On Thu, 6 Aug 2020 at 07:09, Strahil Nikolov >> wrote: >> >>> Can you fheck for errors on the affected host. Most probably you >>> need the vdsm logs. >>> >>> Best Regards, >>> Strahil Nikolov >>> >>> На 6 август 2020 г. 7:40:23 GMT+03:00, Nardus Geldenhuys < >>> nard...@gmail.com> написа: >>> >Hi Strahil >>> > >>> >Hope you are well. I get the following error when I tried to confirm >>> >reboot: >>> > >>> >Error while executing action: Cannot confirm 'Host has been >>> rebooted' >>> >Host. >>> >Valid Host statuses are "Non operational", "Maintenance" or >>> >"Connecting". >>> > >>> >And I can't put it in maintenance, only option is "restart" or >>> "stop". >>> > >>> >Regards >>> > >>> >Nar >>> > >>> >On Thu, 6 Aug 2020 at 06:16, Strahil Nikolov >> > >>
[ovirt-users] Re: Unassigned hosts
Hi Nardus, I'm assuming that your setup was stable and you were able to run your VMs without problems. If so, then below is not a solution to your problem, you should really check engine and VDSM logs for reasons why your hosts become NonResponsive. Most probably there is underlying storage or network issue which prevents correct engine <-> hosts communications and which made your hosts NonResponsive. The solution below will just hide the issues you currently have. If your problem started suddenly when you significantly increased the number of running VMs or decreased the number of available hosts, then you are suffering from those issues because of not having enough resources. Regards, Martin On Thu, Aug 6, 2020 at 4:51 PM Artur Socha wrote: > Thanks Nardus, > After a quick look I found what I was suspecting - there are way too many > threads in Blocked state. I don't know yet the reason but this is very > helpful. I'll let you know about the findings/investigation. Meanwhile, you > may try restarting the engine as (a very brute and ugly) workaround). > You may try to setup slightly bigger thread pool - may save you some time > until the next hiccup. However, please be aware that this may come with the > cost in memory usage and higher cpu usage (due to increased context > switching) > Here are some docs: > > # Specify the thread pool size for jboss managed scheduled executor service > used by commands to periodically execute > # methods. It is generally not necessary to increase the number of threads in > this thread pool. To change the value > # permanently create a conf file 99-engine-scheduled-thread-pool.conf in > /etc/ovirt-engine/engine.conf.d/ > ENGINE_SCHEDULED_THREAD_POOL_SIZE=100 > > > A. > > > On Thu, Aug 6, 2020 at 4:19 PM Nardus Geldenhuys > wrote: > >> Hi Artur >> >> Please find attached, also let me know if I need to rerun. They 5 min >> apart >> >> [root@engine-aa-1-01 ovirt-engine]# ps -ef | grep jboss | grep -v grep >> | awk '{ print $2 }' >> 27390 >> [root@engine-aa-1-01 ovirt-engine]# jstack -F 27390 > >> your_engine_thread_dump_1.txt >> [root@engine-aa-1-01 ovirt-engine]# jstack -F 27390 > >> your_engine_thread_dump_2.txt >> [root@engine-aa-1-01 ovirt-engine]# jstack -F 27390 > >> your_engine_thread_dump_3.txt >> >> Regards >> >> Nar >> >> On Thu, 6 Aug 2020 at 15:55, Artur Socha wrote: >> >>> Sure thing. >>> On engine host please find jboss pid. You can use this command: >>> >>> ps -ef | grep jboss | grep -v grep | awk '{ print $2 }' >>> >>> or jps tool from jdk. Sample output on my dev environment is: >>> >>> ± % jps >>>!2860 >>> 64853 jboss-modules.jar >>> 196217 Jps >>> >>> Then use jstack from jdk: >>> jstack > your_engine_thread_dump.txt >>> 2 or 3 dumps taken in approximately 5 minutes intervals would be even >>> more useful. >>> >>> Here you can find even more options >>> https://www.baeldung.com/java-thread-dump >>> >>> Artur >>> >>> On Thu, Aug 6, 2020 at 3:15 PM Nardus Geldenhuys >>> wrote: >>> Hi Can create thread dump, please send details on howto. Regards Nardus On Thu, 6 Aug 2020 at 14:17, Artur Socha wrote: > Hi Nardus, > You might have hit an issue I have been hunting for some time ( [1] > and [2] ). > [1] could not be properly resolved because at a time was not able to > recreate an issue on dev setup. > I suspect [2] is related. > > Would you be able to prepare a thread dump from your engine instance? > Additionally, please check for potential libvirt errors/warnings. > Can you also paste the output of: > sudo yum list installed | grep vdsm > sudo yum list installed | grep ovirt-engine > sudo yum list installed | grep libvirt > > Usually, according to previous reports, restarting the engine helps to > restore connectivity with hosts ... at least for some time. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1845152 > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1846338 > > regards, > Artur > > > > On Thu, Aug 6, 2020 at 8:01 AM Nardus Geldenhuys > wrote: > >> Also see this in engine: >> >> Aug 6, 2020, 7:37:17 AM >> VDSM someserver command Get Host Capabilities failed: Message timeout >> which can be caused by communication issues >> >> On Thu, 6 Aug 2020 at 07:09, Strahil Nikolov >> wrote: >> >>> Can you fheck for errors on the affected host. Most probably you >>> need the vdsm logs. >>> >>> Best Regards, >>> Strahil Nikolov >>> >>> На 6 август 2020 г. 7:40:23 GMT+03:00, Nardus Geldenhuys < >>> nard...@gmail.com> написа: >>> >Hi Strahil >>> > >>> >Hope you are well. I get the following error when I tried to confirm >>> >reboot: >>> > >>> >Error while executing action: Cannot confirm 'Host has been >>> rebooted' >>> >Host. >>
[ovirt-users] Re: Unassigned hosts
Thanks Nardus, After a quick look I found what I was suspecting - there are way too many threads in Blocked state. I don't know yet the reason but this is very helpful. I'll let you know about the findings/investigation. Meanwhile, you may try restarting the engine as (a very brute and ugly) workaround). You may try to setup slightly bigger thread pool - may save you some time until the next hiccup. However, please be aware that this may come with the cost in memory usage and higher cpu usage (due to increased context switching) Here are some docs: # Specify the thread pool size for jboss managed scheduled executor service used by commands to periodically execute # methods. It is generally not necessary to increase the number of threads in this thread pool. To change the value # permanently create a conf file 99-engine-scheduled-thread-pool.conf in /etc/ovirt-engine/engine.conf.d/ ENGINE_SCHEDULED_THREAD_POOL_SIZE=100 A. On Thu, Aug 6, 2020 at 4:19 PM Nardus Geldenhuys wrote: > Hi Artur > > Please find attached, also let me know if I need to rerun. They 5 min apart > > [root@engine-aa-1-01 ovirt-engine]# ps -ef | grep jboss | grep -v grep | > awk '{ print $2 }' > 27390 > [root@engine-aa-1-01 ovirt-engine]# jstack -F 27390 > > your_engine_thread_dump_1.txt > [root@engine-aa-1-01 ovirt-engine]# jstack -F 27390 > > your_engine_thread_dump_2.txt > [root@engine-aa-1-01 ovirt-engine]# jstack -F 27390 > > your_engine_thread_dump_3.txt > > Regards > > Nar > > On Thu, 6 Aug 2020 at 15:55, Artur Socha wrote: > >> Sure thing. >> On engine host please find jboss pid. You can use this command: >> >> ps -ef | grep jboss | grep -v grep | awk '{ print $2 }' >> >> or jps tool from jdk. Sample output on my dev environment is: >> >> ± % jps >> !2860 >> 64853 jboss-modules.jar >> 196217 Jps >> >> Then use jstack from jdk: >> jstack > your_engine_thread_dump.txt >> 2 or 3 dumps taken in approximately 5 minutes intervals would be even >> more useful. >> >> Here you can find even more options >> https://www.baeldung.com/java-thread-dump >> >> Artur >> >> On Thu, Aug 6, 2020 at 3:15 PM Nardus Geldenhuys >> wrote: >> >>> Hi >>> >>> Can create thread dump, please send details on howto. >>> >>> Regards >>> >>> Nardus >>> >>> On Thu, 6 Aug 2020 at 14:17, Artur Socha wrote: >>> Hi Nardus, You might have hit an issue I have been hunting for some time ( [1] and [2] ). [1] could not be properly resolved because at a time was not able to recreate an issue on dev setup. I suspect [2] is related. Would you be able to prepare a thread dump from your engine instance? Additionally, please check for potential libvirt errors/warnings. Can you also paste the output of: sudo yum list installed | grep vdsm sudo yum list installed | grep ovirt-engine sudo yum list installed | grep libvirt Usually, according to previous reports, restarting the engine helps to restore connectivity with hosts ... at least for some time. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1845152 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1846338 regards, Artur On Thu, Aug 6, 2020 at 8:01 AM Nardus Geldenhuys wrote: > Also see this in engine: > > Aug 6, 2020, 7:37:17 AM > VDSM someserver command Get Host Capabilities failed: Message timeout > which can be caused by communication issues > > On Thu, 6 Aug 2020 at 07:09, Strahil Nikolov > wrote: > >> Can you fheck for errors on the affected host. Most probably you need >> the vdsm logs. >> >> Best Regards, >> Strahil Nikolov >> >> На 6 август 2020 г. 7:40:23 GMT+03:00, Nardus Geldenhuys < >> nard...@gmail.com> написа: >> >Hi Strahil >> > >> >Hope you are well. I get the following error when I tried to confirm >> >reboot: >> > >> >Error while executing action: Cannot confirm 'Host has been rebooted' >> >Host. >> >Valid Host statuses are "Non operational", "Maintenance" or >> >"Connecting". >> > >> >And I can't put it in maintenance, only option is "restart" or >> "stop". >> > >> >Regards >> > >> >Nar >> > >> >On Thu, 6 Aug 2020 at 06:16, Strahil Nikolov >> >wrote: >> > >> >> After rebooting the node, have you "marked" it that it was >> rebooted ? >> >> >> >> Best Regards, >> >> Strahil Nikolov >> >> >> >> На 5 август 2020 г. 21:29:04 GMT+03:00, Nardus Geldenhuys < >> >> nard...@gmail.com> написа: >> >> >Hi oVirt land >> >> > >> >> >Hope you are well. Got a bit of an issue, actually a big issue. We >> >had >> >> >some >> >> >sort of dip of some sort. All the VM's is still running, but some >> of >> >> >the >> >> >hosts is show "Unassigned" or "NonResponsive". So all the hosts >> was >> >> >showin
[ovirt-users] Re: Unassigned hosts
Sure thing. On engine host please find jboss pid. You can use this command: ps -ef | grep jboss | grep -v grep | awk '{ print $2 }' or jps tool from jdk. Sample output on my dev environment is: ± % jps !2860 64853 jboss-modules.jar 196217 Jps Then use jstack from jdk: jstack > your_engine_thread_dump.txt 2 or 3 dumps taken in approximately 5 minutes intervals would be even more useful. Here you can find even more options https://www.baeldung.com/java-thread-dump Artur On Thu, Aug 6, 2020 at 3:15 PM Nardus Geldenhuys wrote: > Hi > > Can create thread dump, please send details on howto. > > Regards > > Nardus > > On Thu, 6 Aug 2020 at 14:17, Artur Socha wrote: > >> Hi Nardus, >> You might have hit an issue I have been hunting for some time ( [1] and >> [2] ). >> [1] could not be properly resolved because at a time was not able to >> recreate an issue on dev setup. >> I suspect [2] is related. >> >> Would you be able to prepare a thread dump from your engine instance? >> Additionally, please check for potential libvirt errors/warnings. >> Can you also paste the output of: >> sudo yum list installed | grep vdsm >> sudo yum list installed | grep ovirt-engine >> sudo yum list installed | grep libvirt >> >> Usually, according to previous reports, restarting the engine helps to >> restore connectivity with hosts ... at least for some time. >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1845152 >> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1846338 >> >> regards, >> Artur >> >> >> >> On Thu, Aug 6, 2020 at 8:01 AM Nardus Geldenhuys >> wrote: >> >>> Also see this in engine: >>> >>> Aug 6, 2020, 7:37:17 AM >>> VDSM someserver command Get Host Capabilities failed: Message timeout >>> which can be caused by communication issues >>> >>> On Thu, 6 Aug 2020 at 07:09, Strahil Nikolov >>> wrote: >>> Can you fheck for errors on the affected host. Most probably you need the vdsm logs. Best Regards, Strahil Nikolov На 6 август 2020 г. 7:40:23 GMT+03:00, Nardus Geldenhuys < nard...@gmail.com> написа: >Hi Strahil > >Hope you are well. I get the following error when I tried to confirm >reboot: > >Error while executing action: Cannot confirm 'Host has been rebooted' >Host. >Valid Host statuses are "Non operational", "Maintenance" or >"Connecting". > >And I can't put it in maintenance, only option is "restart" or "stop". > >Regards > >Nar > >On Thu, 6 Aug 2020 at 06:16, Strahil Nikolov >wrote: > >> After rebooting the node, have you "marked" it that it was rebooted ? >> >> Best Regards, >> Strahil Nikolov >> >> На 5 август 2020 г. 21:29:04 GMT+03:00, Nardus Geldenhuys < >> nard...@gmail.com> написа: >> >Hi oVirt land >> > >> >Hope you are well. Got a bit of an issue, actually a big issue. We >had >> >some >> >sort of dip of some sort. All the VM's is still running, but some of >> >the >> >hosts is show "Unassigned" or "NonResponsive". So all the hosts was >> >showing >> >UP and was fine before our dip. So I did increase >vdsHeartbeatInSecond >> >to >> >240, no luck. >> > >> >I still get a timeout on the engine lock even thou I can connect to >> >that >> >host from the engine using nc to test to port 54321. I also did >restart >> >vdsmd and also rebooted the host with no luck. >> > >> > nc -v someserver 54321 >> >Ncat: Version 7.50 ( https://nmap.org/ncat ) >> >Ncat: Connected to 172.40.2.172:54321. >> > >> >2020-08-05 20:20:34,256+02 ERROR >> >>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> >(EE-ManagedThreadFactory-engineScheduled-Thread-70) [] EVENT_ID: >> >VDS_BROKER_COMMAND_FAILURE(10,802), VDSM someserver command Get Host >> >Capabilities failed: Message timeout which can be caused by >> >communication >> >issues >> > >> >Any troubleshoot ideas will be gladly appreciated. >> > >> >Regards >> > >> >Nar >> >>> ___ >>> Users mailing list -- users@ovirt.org >>> To unsubscribe send an email to users-le...@ovirt.org >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/C4HB2J3MH76FI2325Z4AV4VCCEKH4M3S/ >>> >> >> >> -- >> Artur Socha >> Senior Software Engineer, RHV >> Red Hat >> > -- Artur Socha Senior Software Engineer, RHV Red Hat ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt C
[ovirt-users] Re: Unassigned hosts
Hi Can create thread dump, please send details on howto. Regards Nardus On Thu, 6 Aug 2020 at 14:17, Artur Socha wrote: > Hi Nardus, > You might have hit an issue I have been hunting for some time ( [1] and > [2] ). > [1] could not be properly resolved because at a time was not able to > recreate an issue on dev setup. > I suspect [2] is related. > > Would you be able to prepare a thread dump from your engine instance? > Additionally, please check for potential libvirt errors/warnings. > Can you also paste the output of: > sudo yum list installed | grep vdsm > sudo yum list installed | grep ovirt-engine > sudo yum list installed | grep libvirt > > Usually, according to previous reports, restarting the engine helps to > restore connectivity with hosts ... at least for some time. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1845152 > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1846338 > > regards, > Artur > > > > On Thu, Aug 6, 2020 at 8:01 AM Nardus Geldenhuys > wrote: > >> Also see this in engine: >> >> Aug 6, 2020, 7:37:17 AM >> VDSM someserver command Get Host Capabilities failed: Message timeout >> which can be caused by communication issues >> >> On Thu, 6 Aug 2020 at 07:09, Strahil Nikolov >> wrote: >> >>> Can you fheck for errors on the affected host. Most probably you need >>> the vdsm logs. >>> >>> Best Regards, >>> Strahil Nikolov >>> >>> На 6 август 2020 г. 7:40:23 GMT+03:00, Nardus Geldenhuys < >>> nard...@gmail.com> написа: >>> >Hi Strahil >>> > >>> >Hope you are well. I get the following error when I tried to confirm >>> >reboot: >>> > >>> >Error while executing action: Cannot confirm 'Host has been rebooted' >>> >Host. >>> >Valid Host statuses are "Non operational", "Maintenance" or >>> >"Connecting". >>> > >>> >And I can't put it in maintenance, only option is "restart" or "stop". >>> > >>> >Regards >>> > >>> >Nar >>> > >>> >On Thu, 6 Aug 2020 at 06:16, Strahil Nikolov >>> >wrote: >>> > >>> >> After rebooting the node, have you "marked" it that it was rebooted ? >>> >> >>> >> Best Regards, >>> >> Strahil Nikolov >>> >> >>> >> На 5 август 2020 г. 21:29:04 GMT+03:00, Nardus Geldenhuys < >>> >> nard...@gmail.com> написа: >>> >> >Hi oVirt land >>> >> > >>> >> >Hope you are well. Got a bit of an issue, actually a big issue. We >>> >had >>> >> >some >>> >> >sort of dip of some sort. All the VM's is still running, but some of >>> >> >the >>> >> >hosts is show "Unassigned" or "NonResponsive". So all the hosts was >>> >> >showing >>> >> >UP and was fine before our dip. So I did increase >>> >vdsHeartbeatInSecond >>> >> >to >>> >> >240, no luck. >>> >> > >>> >> >I still get a timeout on the engine lock even thou I can connect to >>> >> >that >>> >> >host from the engine using nc to test to port 54321. I also did >>> >restart >>> >> >vdsmd and also rebooted the host with no luck. >>> >> > >>> >> > nc -v someserver 54321 >>> >> >Ncat: Version 7.50 ( https://nmap.org/ncat ) >>> >> >Ncat: Connected to 172.40.2.172:54321. >>> >> > >>> >> >2020-08-05 20:20:34,256+02 ERROR >>> >> >>> >>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> >> >(EE-ManagedThreadFactory-engineScheduled-Thread-70) [] EVENT_ID: >>> >> >VDS_BROKER_COMMAND_FAILURE(10,802), VDSM someserver command Get Host >>> >> >Capabilities failed: Message timeout which can be caused by >>> >> >communication >>> >> >issues >>> >> > >>> >> >Any troubleshoot ideas will be gladly appreciated. >>> >> > >>> >> >Regards >>> >> > >>> >> >Nar >>> >> >>> >> ___ >> Users mailing list -- users@ovirt.org >> To unsubscribe send an email to users-le...@ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/C4HB2J3MH76FI2325Z4AV4VCCEKH4M3S/ >> > > > -- > Artur Socha > Senior Software Engineer, RHV > Red Hat > ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YYUSMPZHIKV57X3L44ODZV47IMFQEVZE/
[ovirt-users] Re: Unassigned hosts
Hi [root@engine-aa-1-01 ovirt-engine]# sudo yum list installed | grep vdsm vdsm-jsonrpc-java.noarch 1.4.18-1.el7 @ovirt-4.3 [root@engine-aa-1-01 ovirt-engine]# sudo yum list installed | grep vdsm vdsm-jsonrpc-java.noarch 1.4.18-1.el7 @ovirt-4.3 [root@engine-aa-1-01 ovirt-engine]# sudo yum list installed | grep ovirt-engine ovirt-engine.noarch4.3.6.7-1.el7 @ovirt-4.3 ovirt-engine-api-explorer.noarch 0.0.5-1.el7 @ovirt-4.3 ovirt-engine-backend.noarch4.3.6.7-1.el7 @ovirt-4.3 ovirt-engine-dbscripts.noarch 4.3.6.7-1.el7 @ovirt-4.3 ovirt-engine-dwh.noarch4.3.6-1.el7 @ovirt-4.3 ovirt-engine-dwh-setup.noarch 4.3.6-1.el7 @ovirt-4.3 ovirt-engine-extension-aaa-jdbc.noarch 1.1.10-1.el7 @ovirt-4.3 ovirt-engine-extension-aaa-ldap.noarch 1.3.10-1.el7 @ovirt-4.3 ovirt-engine-extension-aaa-ldap-setup.noarch ovirt-engine-extensions-api-impl.noarch4.3.6.7-1.el7 @ovirt-4.3 ovirt-engine-metrics.noarch1.3.4.1-1.el7 @ovirt-4.3 ovirt-engine-restapi.noarch4.3.6.7-1.el7 @ovirt-4.3 ovirt-engine-setup.noarch 4.3.6.7-1.el7 @ovirt-4.3 ovirt-engine-setup-base.noarch 4.3.6.7-1.el7 @ovirt-4.3 ovirt-engine-setup-plugin-cinderlib.noarch 4.3.6.7-1.el7 @ovirt-4.3 ovirt-engine-setup-plugin-ovirt-engine.noarch ovirt-engine-setup-plugin-ovirt-engine-common.noarch ovirt-engine-setup-plugin-vmconsole-proxy-helper.noarch ovirt-engine-setup-plugin-websocket-proxy.noarch ovirt-engine-tools.noarch 4.3.6.7-1.el7 @ovirt-4.3 ovirt-engine-tools-backup.noarch 4.3.6.7-1.el7 @ovirt-4.3 ovirt-engine-ui-extensions.noarch 1.0.10-1.el7 @ovirt-4.3 ovirt-engine-vmconsole-proxy-helper.noarch 4.3.6.7-1.el7 @ovirt-4.3 ovirt-engine-webadmin-portal.noarch4.3.6.7-1.el7 @ovirt-4.3 ovirt-engine-websocket-proxy.noarch4.3.6.7-1.el7 @ovirt-4.3 ovirt-engine-wildfly.x86_6417.0.1-1.el7 @ovirt-4.3 ovirt-engine-wildfly-overlay.noarch17.0.1-1.el7 @ovirt-4.3 python-ovirt-engine-sdk4.x86_644.3.2-2.el7 @ovirt-4.3 python2-ovirt-engine-lib.noarch4.3.6.7-1.el7 @ovirt-4.3 [root@engine-aa-1-01 ovirt-engine]# sudo yum list installed | grep libvirt [root@engine-aa-1-01 ovirt-engine]# I can send more info if needed. And yes, it looks like sometimes it helps if you restart the engine. Regards Nardus On Thu, 6 Aug 2020 at 14:17, Artur Socha wrote: > Hi Nardus, > You might have hit an issue I have been hunting for some time ( [1] and > [2] ). > [1] could not be properly resolved because at a time was not able to > recreate an issue on dev setup. > I suspect [2] is related. > > Would you be able to prepare a thread dump from your engine instance? > Additionally, please check for potential libvirt errors/warnings. > Can you also paste the output of: > sudo yum list installed | grep vdsm > sudo yum list installed | grep ovirt-engine > sudo yum list installed | grep libvirt > > Usually, according to previous reports, restarting the engine helps to > restore connectivity with hosts ... at least for some time. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1845152 > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1846338 > > regards, > Artur > > > > On Thu, Aug 6, 2020 at 8:01 AM Nardus Geldenhuys > wrote: > >> Also see this in engine: >> >> Aug 6, 2020, 7:37:17 AM >> VDSM someserver command Get Host Capabilities failed: Message timeout >> which can be caused by communication issues >> >> On Thu, 6 Aug 2020 at 07:09, Strahil Nikolov >> wrote: >> >>> Can you fheck for errors on the affected host. Most probably you need >>> the vdsm logs. >>> >>> Best Regards, >>> Strahil Nikolov >>> >>> На 6 август 2020 г. 7:40:23 GMT+03:00, Nardus Geldenhuys < >>> nard...@gmail.com> написа: >>> >Hi Strahil >>> > >>> >Hope you are well. I get the following error when I tried to confirm >>> >reboot: >>> > >>> >Error while executing action: Cannot confirm 'Host has been rebooted' >>> >Host. >>> >Valid Host statuses are "Non operational", "Maintenance" or >>> >"Connecting". >>> > >>> >And I can't put it in maintenance, only option is "restart" or "stop". >>> > >>> >Regards >>> > >>> >Nar >>> > >>> >On Thu, 6 Aug 2020 at 06:16, Strahil Nikolov >>> >wrote: >>> > >>> >> After rebooting the node, have you "marked" it that it was rebooted ? >>> >> >>> >> Best Regards, >>> >> Strahil Nikolov >>> >> >>> >> На 5 август 2020 г. 21:29:04 GMT+03:00, Nardus Geldenhuys < >>> >> nard...@gmail.com> написа: >>> >> >Hi oVirt land >>> >> > >>> >> >Hope you are well. Got a bit of an issue, actually a big issue. We >>> >had >>> >> >some >>> >> >sort of dip of some sort. All the VM's is still running, but some of >>> >> >the >>> >> >hosts is show "Unassigned" or "NonResponsive". So all the hosts was >>> >> >showing >>> >> >UP and was fine before our dip. So I did increase >>> >vdsHeartbeatInSecond >>> >> >to
[ovirt-users] Re: Unassigned hosts
Hi Nardus, You might have hit an issue I have been hunting for some time ( [1] and [2] ). [1] could not be properly resolved because at a time was not able to recreate an issue on dev setup. I suspect [2] is related. Would you be able to prepare a thread dump from your engine instance? Additionally, please check for potential libvirt errors/warnings. Can you also paste the output of: sudo yum list installed | grep vdsm sudo yum list installed | grep ovirt-engine sudo yum list installed | grep libvirt Usually, according to previous reports, restarting the engine helps to restore connectivity with hosts ... at least for some time. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1845152 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1846338 regards, Artur On Thu, Aug 6, 2020 at 8:01 AM Nardus Geldenhuys wrote: > Also see this in engine: > > Aug 6, 2020, 7:37:17 AM > VDSM someserver command Get Host Capabilities failed: Message timeout > which can be caused by communication issues > > On Thu, 6 Aug 2020 at 07:09, Strahil Nikolov > wrote: > >> Can you fheck for errors on the affected host. Most probably you need the >> vdsm logs. >> >> Best Regards, >> Strahil Nikolov >> >> На 6 август 2020 г. 7:40:23 GMT+03:00, Nardus Geldenhuys < >> nard...@gmail.com> написа: >> >Hi Strahil >> > >> >Hope you are well. I get the following error when I tried to confirm >> >reboot: >> > >> >Error while executing action: Cannot confirm 'Host has been rebooted' >> >Host. >> >Valid Host statuses are "Non operational", "Maintenance" or >> >"Connecting". >> > >> >And I can't put it in maintenance, only option is "restart" or "stop". >> > >> >Regards >> > >> >Nar >> > >> >On Thu, 6 Aug 2020 at 06:16, Strahil Nikolov >> >wrote: >> > >> >> After rebooting the node, have you "marked" it that it was rebooted ? >> >> >> >> Best Regards, >> >> Strahil Nikolov >> >> >> >> На 5 август 2020 г. 21:29:04 GMT+03:00, Nardus Geldenhuys < >> >> nard...@gmail.com> написа: >> >> >Hi oVirt land >> >> > >> >> >Hope you are well. Got a bit of an issue, actually a big issue. We >> >had >> >> >some >> >> >sort of dip of some sort. All the VM's is still running, but some of >> >> >the >> >> >hosts is show "Unassigned" or "NonResponsive". So all the hosts was >> >> >showing >> >> >UP and was fine before our dip. So I did increase >> >vdsHeartbeatInSecond >> >> >to >> >> >240, no luck. >> >> > >> >> >I still get a timeout on the engine lock even thou I can connect to >> >> >that >> >> >host from the engine using nc to test to port 54321. I also did >> >restart >> >> >vdsmd and also rebooted the host with no luck. >> >> > >> >> > nc -v someserver 54321 >> >> >Ncat: Version 7.50 ( https://nmap.org/ncat ) >> >> >Ncat: Connected to 172.40.2.172:54321. >> >> > >> >> >2020-08-05 20:20:34,256+02 ERROR >> >> >> >>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> >> >(EE-ManagedThreadFactory-engineScheduled-Thread-70) [] EVENT_ID: >> >> >VDS_BROKER_COMMAND_FAILURE(10,802), VDSM someserver command Get Host >> >> >Capabilities failed: Message timeout which can be caused by >> >> >communication >> >> >issues >> >> > >> >> >Any troubleshoot ideas will be gladly appreciated. >> >> > >> >> >Regards >> >> > >> >> >Nar >> >> >> > ___ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/C4HB2J3MH76FI2325Z4AV4VCCEKH4M3S/ > -- Artur Socha Senior Software Engineer, RHV Red Hat ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RZPEGTZ6WD35MMSHF357RQI34E66N7MB/
[ovirt-users] Re: Unassigned hosts
Also see this in engine: Aug 6, 2020, 7:37:17 AM VDSM someserver command Get Host Capabilities failed: Message timeout which can be caused by communication issues On Thu, 6 Aug 2020 at 07:09, Strahil Nikolov wrote: > Can you fheck for errors on the affected host. Most probably you need the > vdsm logs. > > Best Regards, > Strahil Nikolov > > На 6 август 2020 г. 7:40:23 GMT+03:00, Nardus Geldenhuys < > nard...@gmail.com> написа: > >Hi Strahil > > > >Hope you are well. I get the following error when I tried to confirm > >reboot: > > > >Error while executing action: Cannot confirm 'Host has been rebooted' > >Host. > >Valid Host statuses are "Non operational", "Maintenance" or > >"Connecting". > > > >And I can't put it in maintenance, only option is "restart" or "stop". > > > >Regards > > > >Nar > > > >On Thu, 6 Aug 2020 at 06:16, Strahil Nikolov > >wrote: > > > >> After rebooting the node, have you "marked" it that it was rebooted ? > >> > >> Best Regards, > >> Strahil Nikolov > >> > >> На 5 август 2020 г. 21:29:04 GMT+03:00, Nardus Geldenhuys < > >> nard...@gmail.com> написа: > >> >Hi oVirt land > >> > > >> >Hope you are well. Got a bit of an issue, actually a big issue. We > >had > >> >some > >> >sort of dip of some sort. All the VM's is still running, but some of > >> >the > >> >hosts is show "Unassigned" or "NonResponsive". So all the hosts was > >> >showing > >> >UP and was fine before our dip. So I did increase > >vdsHeartbeatInSecond > >> >to > >> >240, no luck. > >> > > >> >I still get a timeout on the engine lock even thou I can connect to > >> >that > >> >host from the engine using nc to test to port 54321. I also did > >restart > >> >vdsmd and also rebooted the host with no luck. > >> > > >> > nc -v someserver 54321 > >> >Ncat: Version 7.50 ( https://nmap.org/ncat ) > >> >Ncat: Connected to 172.40.2.172:54321. > >> > > >> >2020-08-05 20:20:34,256+02 ERROR > >> > >>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > >> >(EE-ManagedThreadFactory-engineScheduled-Thread-70) [] EVENT_ID: > >> >VDS_BROKER_COMMAND_FAILURE(10,802), VDSM someserver command Get Host > >> >Capabilities failed: Message timeout which can be caused by > >> >communication > >> >issues > >> > > >> >Any troubleshoot ideas will be gladly appreciated. > >> > > >> >Regards > >> > > >> >Nar > >> > ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/C4HB2J3MH76FI2325Z4AV4VCCEKH4M3S/
[ovirt-users] Re: Unassigned hosts
Restarted vdsmd on host: mom.log: 2020-08-06 07:21:19,053 - mom.GuestManager - INFO - Guest Manager ending 2020-08-06 07:21:20,483 - mom.HostMonitor - INFO - Host Monitor ending 2020-08-06 07:21:24,795 - mom - INFO - MOM starting 2020-08-06 07:21:24,833 - mom - INFO - hypervisor interface vdsmjsonrpcclient 2020-08-06 07:21:24,833 - mom.HostMonitor - INFO - Host Monitor starting 2020-08-06 07:21:24,880 - mom - ERROR - Failed to initialize MOM threads Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 29, in run hypervisor_iface = self.get_hypervisor_interface() File "/usr/lib/python2.7/site-packages/mom/__init__.py", line 217, in get_hypervisor_interface return module.instance(self.config) File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcclientInterface.py", line 96, in instance return JsonRpcVdsmClientInterface() File "/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcclientInterface.py", line 31, in __init__ self._vdsm_api = client.connect(host="localhost") File "/usr/lib/python2.7/site-packages/vdsm/client.py", line 157, in connect raise ConnectionError(host, port, use_tls, timeout, e) ConnectionError: Connection to localhost:54321 with use_tls=True, timeout=60 failed: [Errno 111] Connection refused 2020-08-06 07:21:30,085 - mom - INFO - MOM starting 2020-08-06 07:21:30,122 - mom.HostMonitor - INFO - Host Monitor starting 2020-08-06 07:21:30,123 - mom - INFO - hypervisor interface vdsmjsonrpcclient 2020-08-06 07:21:30,217 - mom.HostMonitor - INFO - HostMonitor is ready 2020-08-06 07:21:30,221 - mom.GuestManager - INFO - Guest Manager starting: multi-thread 2020-08-06 07:21:30,226 - mom.Policy - INFO - Loaded policy '00-defines' 2020-08-06 07:21:30,228 - mom.Policy - INFO - Loaded policy '01-parameters' 2020-08-06 07:21:30,241 - mom.Policy - INFO - Loaded policy '02-balloon' 2020-08-06 07:21:30,263 - mom.Policy - INFO - Loaded policy '03-ksm' 2020-08-06 07:21:30,290 - mom.Policy - INFO - Loaded policy '04-cputune' 2020-08-06 07:21:30,321 - mom.Policy - INFO - Loaded policy '05-iotune' 2020-08-06 07:21:30,321 - mom.PolicyEngine - INFO - Policy Engine starting 2020-08-06 07:21:30,322 - mom.RPCServer - INFO - Using unix socket /var/run/vdsm/mom-vdsm.sock 2020-08-06 07:21:30,323 - mom.RPCServer - INFO - RPC Server starting 2020-08-06 07:21:40,692 - mom.RPCServer - INFO - ping() 2020-08-06 07:21:40,692 - mom.RPCServer - INFO - getStatistics() 2020-08-06 07:21:45,356 - mom.Controllers.KSM - INFO - Updating KSM configuration: pages_to_scan:0 merge_across_nodes:1 run:0 sleep_millisecs:0 2020-08-06 07:21:55,838 - mom.RPCServer - INFO - ping() 2020-08-06 07:21:55,839 - mom.RPCServer - INFO - getStatistics() supervdsm.log: MainProcess|jsonrpc/3::DEBUG::2020-08-05 20:11:14,139::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call ksmTune with ({u'run': 0, u'merge_across_nodes': 1},) {} MainProcess|jsonrpc/3::DEBUG::2020-08-05 20:11:14,139::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return ksmTune with None MainProcess::DEBUG::2020-08-06 07:21:25,279::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call multipath_status with (,) {} MainProcess::DEBUG::2020-08-06 07:21:25,279::logutils::319::root::(_report_stats) ThreadedHandler is ok in the last 40234 seconds (max pending: 3) MainProcess::DEBUG::2020-08-06 07:21:25,279::commands::198::storage.Misc.excCmd::(execCmd) /usr/bin/taskset --cpu-list 0-95 /usr/sbin/dmsetup status --target multipath (cwd None) MainProcess::DEBUG::2020-08-06 07:21:25,283::commands::219::storage.Misc.excCmd::(execCmd) SUCCESS: = ''; = 0 MainProcess::DEBUG::2020-08-06 07:21:25,289::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return multipath_status with {u'T1_58886_2121': [PathStatus(name=u'sdd', status=u'A'), PathStatus(name=u'sdm', status=u'A')], u'T0_someserver_boot_58886_20c2': [PathStatus(name=u'sdi', status=u'A'), PathStatus(name=u'sdr', status=u'A')], u'T0_R4_UCS_MOB1P_DIGIT_58886_20c8': [PathStatus(name=u'sdg', status=u'A'), PathStatus(name=u'sdp', status=u'A')], u'T0_58886_215d': [PathStatus(name=u'sdb', status=u'A'), PathStatus(name=u'sdk', status=u'A')], u'T0_R4_UCS_MOB1P_DIGIT_58886_20c7': [PathStatus(name=u'sdf', status=u'A'), PathStatus(name=u'sdo', status=u'A')], u'T0_58886_20b8': [PathStatus(name=u'sde', status=u'A'), PathStatus(name=u'sdn', status=u'A')], u'T0_58886_208a': [PathStatus(name=u'sdc', status=u'A'), PathStatus(name=u'sdl', status=u'A')], u'T0_58886_2124': [PathStatus(name=u'sdh', status=u'A'), PathStatus(name=u'sdq', status=u'A')], u'T0_58886_215c': [PathStatus(name=u'sda', status=u'A'), PathStatus(name=u'sdj', status=u'A')]} MainProcess|hsm/init::DEBUG::2020-08-06 07:21:25,383::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call hbaRescan with (,) {} MainProcess|hsm/init::DEBUG::2020-08-06 07:21:25,384::commands::198::storage.HBA::(execCmd) /usr/bin/taskset --cpu-list 0-95 /usr/libex
[ovirt-users] Re: Unassigned hosts
Can you fheck for errors on the affected host. Most probably you need the vdsm logs. Best Regards, Strahil Nikolov На 6 август 2020 г. 7:40:23 GMT+03:00, Nardus Geldenhuys написа: >Hi Strahil > >Hope you are well. I get the following error when I tried to confirm >reboot: > >Error while executing action: Cannot confirm 'Host has been rebooted' >Host. >Valid Host statuses are "Non operational", "Maintenance" or >"Connecting". > >And I can't put it in maintenance, only option is "restart" or "stop". > >Regards > >Nar > >On Thu, 6 Aug 2020 at 06:16, Strahil Nikolov >wrote: > >> After rebooting the node, have you "marked" it that it was rebooted ? >> >> Best Regards, >> Strahil Nikolov >> >> На 5 август 2020 г. 21:29:04 GMT+03:00, Nardus Geldenhuys < >> nard...@gmail.com> написа: >> >Hi oVirt land >> > >> >Hope you are well. Got a bit of an issue, actually a big issue. We >had >> >some >> >sort of dip of some sort. All the VM's is still running, but some of >> >the >> >hosts is show "Unassigned" or "NonResponsive". So all the hosts was >> >showing >> >UP and was fine before our dip. So I did increase >vdsHeartbeatInSecond >> >to >> >240, no luck. >> > >> >I still get a timeout on the engine lock even thou I can connect to >> >that >> >host from the engine using nc to test to port 54321. I also did >restart >> >vdsmd and also rebooted the host with no luck. >> > >> > nc -v someserver 54321 >> >Ncat: Version 7.50 ( https://nmap.org/ncat ) >> >Ncat: Connected to 172.40.2.172:54321. >> > >> >2020-08-05 20:20:34,256+02 ERROR >> >>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> >(EE-ManagedThreadFactory-engineScheduled-Thread-70) [] EVENT_ID: >> >VDS_BROKER_COMMAND_FAILURE(10,802), VDSM someserver command Get Host >> >Capabilities failed: Message timeout which can be caused by >> >communication >> >issues >> > >> >Any troubleshoot ideas will be gladly appreciated. >> > >> >Regards >> > >> >Nar >> ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/T73Q4GMQ6JZSBCXYGCEQGI3HOC4SSFNG/
[ovirt-users] Re: Unassigned hosts
Hi Strahil Hope you are well. I get the following error when I tried to confirm reboot: Error while executing action: Cannot confirm 'Host has been rebooted' Host. Valid Host statuses are "Non operational", "Maintenance" or "Connecting". And I can't put it in maintenance, only option is "restart" or "stop". Regards Nar On Thu, 6 Aug 2020 at 06:16, Strahil Nikolov wrote: > After rebooting the node, have you "marked" it that it was rebooted ? > > Best Regards, > Strahil Nikolov > > На 5 август 2020 г. 21:29:04 GMT+03:00, Nardus Geldenhuys < > nard...@gmail.com> написа: > >Hi oVirt land > > > >Hope you are well. Got a bit of an issue, actually a big issue. We had > >some > >sort of dip of some sort. All the VM's is still running, but some of > >the > >hosts is show "Unassigned" or "NonResponsive". So all the hosts was > >showing > >UP and was fine before our dip. So I did increase vdsHeartbeatInSecond > >to > >240, no luck. > > > >I still get a timeout on the engine lock even thou I can connect to > >that > >host from the engine using nc to test to port 54321. I also did restart > >vdsmd and also rebooted the host with no luck. > > > > nc -v someserver 54321 > >Ncat: Version 7.50 ( https://nmap.org/ncat ) > >Ncat: Connected to 172.40.2.172:54321. > > > >2020-08-05 20:20:34,256+02 ERROR > >[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > >(EE-ManagedThreadFactory-engineScheduled-Thread-70) [] EVENT_ID: > >VDS_BROKER_COMMAND_FAILURE(10,802), VDSM someserver command Get Host > >Capabilities failed: Message timeout which can be caused by > >communication > >issues > > > >Any troubleshoot ideas will be gladly appreciated. > > > >Regards > > > >Nar > ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/NMQDDIGOIV7EDXO3EFHVU3ROKU44Y6ZY/
[ovirt-users] Re: Unassigned hosts
After rebooting the node, have you "marked" it that it was rebooted ? Best Regards, Strahil Nikolov На 5 август 2020 г. 21:29:04 GMT+03:00, Nardus Geldenhuys написа: >Hi oVirt land > >Hope you are well. Got a bit of an issue, actually a big issue. We had >some >sort of dip of some sort. All the VM's is still running, but some of >the >hosts is show "Unassigned" or "NonResponsive". So all the hosts was >showing >UP and was fine before our dip. So I did increase vdsHeartbeatInSecond >to >240, no luck. > >I still get a timeout on the engine lock even thou I can connect to >that >host from the engine using nc to test to port 54321. I also did restart >vdsmd and also rebooted the host with no luck. > > nc -v someserver 54321 >Ncat: Version 7.50 ( https://nmap.org/ncat ) >Ncat: Connected to 172.40.2.172:54321. > >2020-08-05 20:20:34,256+02 ERROR >[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >(EE-ManagedThreadFactory-engineScheduled-Thread-70) [] EVENT_ID: >VDS_BROKER_COMMAND_FAILURE(10,802), VDSM someserver command Get Host >Capabilities failed: Message timeout which can be caused by >communication >issues > >Any troubleshoot ideas will be gladly appreciated. > >Regards > >Nar ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QDTX76DFTZYON6QMAFICNDY3GJ6TR2UD/