[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-16 Thread Strahil Nikolov
gt; >> >>> >Apr 08 23:09:39 ovirt-node-01.phoelex.com >> >> >libvirtd[29307]: >> >> >> >>>> >> >2020-04-08 >> >> >> >>>> >> >>> >23:09:39.844+0000: 2

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-15 Thread Strahil Nikolov
;> >> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>08:07:31,438::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) >> >> >>>> >> >>> >ovirt-ho

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-15 Thread Strahil Nikolov
tors) >> >>>> >> >>> >Searching for submonitors in >> >>>> >> >>> >> >>/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker >> >>>> >> >>> > >> >>>> >> >>> >/sub

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-15 Thread Strahil Nikolov
>>> >MainThread::INFO::2020-04-09 >>>> >> >>> >>>> >> >>> >>>> >> >>>> >> >>>> >>>> >>>>08:07:31,441::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-15 Thread Strahil Nikolov
onitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>> >Loaded submonitor storage-domain >> >> >>> > >> >> >>> >MainThread::INFO::2020-04-09 >> >> >>> >>

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-15 Thread Shareef Jalloq
49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) >> >> >>> >Loaded submonitor storage-domain >> >> >>> > >> >> >>> >MainThread::INFO::2020-04-09 >> >> >>> >> >> >>>

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-15 Thread Shareef Jalloq
-health > >> >>> > > >> >>> >MainThread::INFO::2020-04-09 > >> >>> > >> >>> > >> > >> > > >>>08:07:31,444::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonito

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-14 Thread Strahil Nikolov
localhost:54321::DEBUG::2020-04-09 >> >>> >08:07:31,453::concurrent::258::root::(run) START thread >> >> >>> >localhost:54321, started daemon 139992488138496)> (func=> >method >> >>> >Reactor.process_requests of > >

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-14 Thread Shareef Jalloq
; > >>> >MainThread::DEBUG::2020-04-09 > >>> >08:07:31,531::stompclient::294::jsonrpc.AsyncoreClient::(send) > >Sending > >>> >response > >>> > > >>> >MainThread::DEBUG::2020-04-09 > >>> >08:07:31,534::st

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-14 Thread Strahil Nikolov
connect_storage_server) >>> >[{u'status': 0, u'id': u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}] >>> > >>> >MainThread::INFO::2020-04-09 >>> >>> >>08:07:32,814::storage_server::413::ovirt_hosted_engine_ha.lib.storage_server.S

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-14 Thread Shareef Jalloq
'sdUUID=a6cea67d-dbfb-45cf-a775-b4d0d47b26f2',)) >> > >> >MainThread::WARNING::2020-04-09 >> >> >08:07:33,795::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) >> >Can't connect vdsm storage: Command StorageDomain.getI

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-10 Thread Shareef Jalloq
gt; > >Thanks, Shareef. > > > >On Thu, Apr 9, 2020 at 6:12 AM Strahil Nikolov > >wrote: > > > >> On April 9, 2020 1:51:05 AM GMT+03:00, Shareef Jalloq < > >> shar...@jalloq.co.uk> wrote: > >> >Don't know if this is useful or no

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-09 Thread eevans
day, April 9, 2020 12:57 PM To: Shareef Jalloq Cc: eev...@digitaldatatechs.com; Ovirt Users Subject: [ovirt-users] Re: ovirt-engine unresponsive - how to rescue? On April 9, 2020 11:12:30 AM GMT+03:00, Shareef Jalloq wrote: >OK, let's go through this. I'm looking at the node that at least s

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-09 Thread Strahil Nikolov
> > >> >On Wed, Apr 8, 2020 at 11:35 PM Shareef Jalloq > >> >wrote: >> > >> >> Hmmm, virsh tells me the HE is running but it hasn't come up and >the >> >> agent.log is full of the same errors. >> >> >> >> On We

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-09 Thread Shareef Jalloq
t;>> NFS mounts? > >>> > >>> Secondly, why doesn't the node mount the NFS directories at boot? > >Is > >>> that the issue with this particular node? > >>> > >>> On Wed, Apr 8, 2020 at 11:12 PM wrote: > >>> > >>>> Did you try virsh

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-08 Thread Strahil Nikolov
>>> On Wed, Apr 8, 2020 at 11:12 PM wrote: >>> >>>> Did you try virsh list --inactive >>>> >>>> >>>> >>>> Eric Evans >>>> >>>> Digital Data Services LLC. >>>> >>>> 3

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-08 Thread Shareef Jalloq
a Services LLC. >>> >>> 304.660.9080 >>> >>> >>> >>> *From:* Shareef Jalloq >>> *Sent:* Wednesday, April 8, 2020 5:58 PM >>> *To:* Strahil Nikolov >>> *Cc:* Ovirt Users >>> *Subject:* [ovirt-users] Re: ovirt-engi

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-08 Thread Shareef Jalloq
list --inactive >> >> >> >> Eric Evans >> >> Digital Data Services LLC. >> >> 304.660.9080 >> >> >> >> *From:* Shareef Jalloq >> *Sent:* Wednesday, April 8, 2020 5:58 PM >> *To:* Strahil Nikolov >> *Cc:* Ovirt Use

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-08 Thread Shareef Jalloq
*Cc:* Ovirt Users > *Subject:* [ovirt-users] Re: ovirt-engine unresponsive - how to rescue? > > > > I've now shut down the VMs on one host and rebooted it but the agent > service doesn't start. If I run 'hosted-engine --vm-status' I get: > > > > The hosted en

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-08 Thread eevans
Did you try virsh list --inactive Eric Evans Digital Data Services LLC. 304.660.9080 From: Shareef Jalloq Sent: Wednesday, April 8, 2020 5:58 PM To: Strahil Nikolov Cc: Ovirt Users Subject: [ovirt-users] Re: ovirt-engine unresponsive - how to rescue? I've now shut down the VMs

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-08 Thread Shareef Jalloq
I've now shut down the VMs on one host and rebooted it but the agent service doesn't start. If I run 'hosted-engine --vm-status' I get: The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-08 Thread Shareef Jalloq
Right, still down. I've run virsh and it doesn't know anything about the engine vm. I've restarted the broker and agent services and I still get nothing in virsh->list. In the logs under /var/log/ovirt-hosted-engine-ha I see lots of errors: broker.log: MainThread::INFO::2020-04-08

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-08 Thread eevans
If you haven’t got this resolved, log into the host and use ‘saslpasswd’ without the quotes. Then virsh start and use the password you set on the local account. I’m not sure it will work, but has worked for regular vm’s. Eric Evans Digital Data Services LLC. 304.660.9080 From:

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-08 Thread eevans
: Maton, Brett Sent: Wednesday, April 8, 2020 12:09 PM To: Shareef Jalloq Cc: Ovirt Users Subject: [ovirt-users] Re: ovirt-engine unresponsive - how to rescue? First steps, on one of your hosts as root: To get information: hosted-engine --vm-status To start the engine: hosted-engine

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-08 Thread Strahil Nikolov
On April 8, 2020 7:47:20 PM GMT+03:00, "Maton, Brett" wrote: >On the host you tried to restart the engine on: > >Add an alias to virsh (authenticates with virsh_auth.conf) > >alias virsh='virsh -c >qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf' > >Then run virsh: > >virsh >

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-08 Thread Maton, Brett
On the host you tried to restart the engine on: Add an alias to virsh (authenticates with virsh_auth.conf) alias virsh='virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf' Then run virsh: virsh virsh # list IdName State

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-08 Thread Shareef Jalloq
Thanks! The status hangs due to, I guess, the VM being down [root@ovirt-node-01 ~]# hosted-engine --vm-start VM exists and is down, cleaning up and restarting VM in WaitForLaunch but this doesn't seem to do anything. OK, after a while I get a status of it being barfed... --== Host

[ovirt-users] Re: ovirt-engine unresponsive - how to rescue?

2020-04-08 Thread Maton, Brett
First steps, on one of your hosts as root: To get information: hosted-engine --vm-status To start the engine: hosted-engine --vm-start On Wed, 8 Apr 2020 at 17:00, Shareef Jalloq wrote: > So my engine has gone down and I can't ssh into it either. If I try to > log into the web-ui of the