Of course right when I sent this email, I went back over to one of my consoles, 
re-ran "hosted-engine --status", and I saw that it was up. I can confirm my 
hosted engine is now online and healthy.

So to recap: restarting vdsmd solved my problem.

I provided lots of details in the Bugzilla, and I generated an sosreport on two 
of my three systems prior to restarting vdsmd.

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Friday, August 13th, 2021 at 9:31 PM, David White 
<dmwhite...@protonmail.com> wrote:

> I have updated the Bugzilla with all of the details I included below, as well 
> as additional details.
> 

> I figured better to err on the side of providing too many details than not 
> enough.
> 

> For the oVirt list's edification, I will note that restarting vdsmd on all 3 
> hosts did fix the problem -- to an extent. Unfortunately, my hosted-engine is 
> still not starting (although I can now clearly connect to the hosted-engine 
> storage), and I see this output every time I try to start the hosted-engine:
> 

> [root@cha2-storage ~]# hosted-engine --vm-start
> 

> Command VM.getStats with args {'vmID': 
> 'ffd77d79-a699-455e-88e2-f55ee53166ef'} failed:
> 

> (code=1, message=Virtual machine does not exist: {'vmId': 
> 'ffd77d79-a699-455e-88e2-f55ee53166ef'})
> 

> VM in WaitForLaunch
> 

> I'm not sure if that's because I screwed up when I was doing gluster 
> maintenance, or what.
> 

> But at this point, does this mean I have to re-deploy the hosted engine?
> 

> To confirm, if I re-deploy the hosted engine, will all of my regular VMs 
> remain intact? I have over 20 VMs in this environment, and it would be a 
> major deal to have to rebuild all 20+ of those VMs.
> 

> Sent with ProtonMail Secure Email.
> 

> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> 

> On Friday, August 13th, 2021 at 2:41 PM, Nir Soffer nsof...@redhat.com wrote:
> 

> > On Fri, Aug 13, 2021 at 9:13 PM David White via Users users@ovirt.org wrote:
> > 

> > > Hello,
> > > 

> > > It appears that my Manager / hosted-engine isn't working, and I'm unable 
> > > to get it to start.
> > > 

> > > I have a 3-node HCI cluster, but right now, Gluster is only running on 1 
> > > host (so no replication).
> > > 

> > > I was hoping to upgrade / replace the storage on my 2nd host today, but 
> > > aborted that maintenance when I found that I couldn't even get into the 
> > > Manager.
> > > 

> > > The storage is mounted, but here's what I see:
> > > 

> > > [root@cha2-storage dwhite]# hosted-engine --vm-status
> > > 

> > > The hosted engine configuration has not been retrieved from shared 
> > > storage. Please ensure that ovirt-ha-agent is running and the storage 
> > > server is reachable.
> > > 

> > > [root@cha2-storage dwhite]# systemctl status ovirt-ha-agent
> > > 

> > > ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability 
> > > Monitoring Agent
> > > 

> > > Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; 
> > > vendor preset: disabled)
> > > 

> > > Active: active (running) since Fri 2021-08-13 11:10:51 EDT; 2h 44min ago
> > > 

> > > Main PID: 3591872 (ovirt-ha-agent)
> > > 

> > > Tasks: 1 (limit: 409676)
> > > 

> > > Memory: 21.5M
> > > 

> > > CGroup: /system.slice/ovirt-ha-agent.service
> > > 

> > > └─3591872 /usr/libexec/platform-python 
> > > /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent
> > > 

> > > Aug 13 11:10:51 cha2-storage.mgt.barredowlweb.com systemd[1]: Started 
> > > oVirt Hosted Engine High Availability Monitoring Agent.
> > > 

> > > Any time I try to do anything like connect the engine storage, disconnect 
> > > the engine storage, or connect to the console, it just sits there, and 
> > > doesn't do anything, and I eventually have to ctl-c out of it.
> > > 

> > > Maybe I have to be patient? When I ctl-c, I get a trackback error:
> > > 

> > > [root@cha2-storage dwhite]# hosted-engine --console
> > > 

> > > ^CTraceback (most recent call last):
> > > 

> > > File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
> > > 

> > >     "__main__", mod_spec)
> > >     

> > > 

> > > File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
> > > 

> > > exec(code, run_globals)
> > > 

> > > File 
> > > "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/vdsm_helper.py",
> > >  line 214, in <module>
> > > 

> > > [root@cha2-storage dwhite]# args.command(args)
> > > 

> > > File 
> > > "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/vdsm_helper.py",
> > >  line 42, in func
> > > 

> > > f(*args, **kwargs)
> > > 

> > > File 
> > > "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/vdsm_helper.py",
> > >  line 91, in checkVmStatus
> > > 

> > > cli = ohautil.connect_vdsm_json_rpc()
> > > 

> > > File 
> > > "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/util.py", 
> > > line 472, in connect_vdsm_json_rpc
> > > 

> > > __vdsm_json_rpc_connect(logger, timeout)
> > > 

> > > File 
> > > "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/util.py", 
> > > line 395, in __vdsm_json_rpc_connect
> > > 

> > > timeout=timeout)
> > > 

> > > File "/usr/lib/python3.6/site-packages/vdsm/client.py", line 154, in 
> > > connect
> > > 

> > > outgoing_heartbeat=outgoing_heartbeat, nr_retries=nr_retries)
> > > 

> > > File "/usr/lib/python3.6/site-packages/yajsonrpc/stompclient.py", line 
> > > 426, in SimpleClient
> > > 

> > > nr_retries, reconnect_interval)
> > > 

> > > File "/usr/lib/python3.6/site-packages/yajsonrpc/stompclient.py", line 
> > > 448, in StandAloneRpcClient
> > > 

> > > client = StompClient(utils.create_connected_socket(host, port, sslctx),
> > > 

> > > File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 379, in 
> > > create_connected_socket
> > > 

> > > sock.connect((host, port))
> > > 

> > > File "/usr/lib64/python3.6/ssl.py", line 1068, in connect
> > > 

> > > self._real_connect(addr, False)
> > > 

> > > File "/usr/lib64/python3.6/ssl.py", line 1059, in _real_connect
> > > 

> > > self.do_handshake()
> > > 

> > > File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
> > > 

> > > self._sslobj.do_handshake()
> > > 

> > > File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
> > > 

> > > self._sslobj.do_handshake()
> > > 

> > > This is what I see in /var/log/ovirt-hosted-engine-ha/broker.log:
> > > 

> > > MainThread::WARNING::2021-08-11 
> > > 10:24:41,596::storage_broker::100::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(init)
> > >  Can't connect vdsm storage: Connection to storage server failed
> > > 

> > > MainThread::ERROR::2021-08-11 
> > > 10:24:41,596::broker::69::ovirt_hosted_engine_ha.broker.broker.Broker::(run)
> > >  Failed initializing the broker: Connection to storage server failed
> > > 

> > > MainThread::ERROR::2021-08-11 
> > > 10:24:41,598::broker::71::ovirt_hosted_engine_ha.broker.broker.Broker::(run)
> > >  Traceback (most recent call last):
> > > 

> > > File 
> > > "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py",
> > >  line 64, in run
> > > 

> > > self._storage_broker_instance = self._get_storage_broker()
> > > 

> > > File 
> > > "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py",
> > >  line 143, in _get_storage_broker
> > > 

> > > return storage_broker.StorageBroker()
> > > 

> > > File 
> > > "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
> > >  line 97, in init
> > > 

> > > self._backend.connect()
> > > 

> > > File 
> > > "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
> > >  line 375, in connect
> > > 

> > > sserver.connect_storage_server()
> > > 

> > > File 
> > > "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_server.py",
> > >  line 451, in connect_storage_server
> > > 

> > > 'Connection to storage server failed'
> > > 

> > > RuntimeError: Connection to storage server failed
> > > 

> > > MainThread::ERROR::2021-08-11 
> > > 10:24:41,599::broker::72::ovirt_hosted_engine_ha.broker.broker.Broker::(run)
> > >  Trying to restart the broker
> > > 

> > > MainThread::INFO::2021-08-11 
> > > 10:24:42,439::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run)
> > >  ovirt-hosted-engine-ha broker 2.4.7 started
> > > 

> > > MainThread::INFO::2021-08-11 
> > > 10:24:44,442::monitor::45::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors)
> > >  Searching for submonitors in 
> > > /usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/submonitors
> > > 

> > > MainThread::INFO::2021-08-11 
> > > 10:24:44,443::monitor::62::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors)
> > >  Loaded submonitor cpu-load
> > > 

> > > MainThread::INFO::2021-08-11 
> > > 10:24:44,449::monitor::62::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors)
> > >  Loaded submonitor cpu-load-no-engine
> > > 

> > > MainThread::INFO::2021-08-11 
> > > 10:24:44,450::monitor::62::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors)
> > >  Loaded submonitor engine-health
> > > 

> > > MainThread::INFO::2021-08-11 
> > > 10:24:44,451::monitor::62::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors)
> > >  Loaded submonitor mem-free
> > > 

> > > MainThread::INFO::2021-08-11 
> > > 10:24:44,451::monitor::62::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors)
> > >  Loaded submonitor mgmt-bridge
> > > 

> > > MainThread::INFO::2021-08-11 
> > > 10:24:44,452::monitor::62::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors)
> > >  Loaded submonitor network
> > > 

> > > MainThread::INFO::2021-08-11 
> > > 10:24:44,452::monitor::62::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors)
> > >  Loaded submonitor storage-domain
> > > 

> > > MainThread::INFO::2021-08-11 
> > > 10:24:44,452::monitor::63::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors)
> > >  Finished loading submonitors
> > > 

> > > And I see this in /var/log/vdsm/vdsm.log:
> > > 

> > > 2021-08-13 14:08:10,844-0400 ERROR (Reactor thread) 
> > > [ProtocolDetector.AcceptorImpl] Unhandled exception in acceptor 
> > > (protocoldetector:76)
> > > 

> > > Traceback (most recent call last):
> > > 

> > > File "/usr/lib64/python3.6/asyncore.py", line 108, in readwrite
> > > 

> > > File "/usr/lib64/python3.6/asyncore.py", line 417, in handle_read_event
> > > 

> > > File "/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py", line 
> > > 57, in handle_accept
> > > 

> > > File "/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py", line 
> > > 173, in _delegate_call
> > > 

> > > File "/usr/lib/python3.6/site-packages/vdsm/protocoldetector.py", line 
> > > 53, in handle_accept
> > > 

> > > File "/usr/lib64/python3.6/asyncore.py", line 348, in accept
> > > 

> > > File "/usr/lib64/python3.6/socket.py", line 205, in accept
> > > 

> > > OSError: [Errno 24] Too many open files
> > 

> > This may be this bug:
> > 

> > https://bugzilla.redhat.com/show_bug.cgi?id=1926589
> > 

> > Since vdsm will never recover from this error without a reboot, you should
> > 

> > start by restarting vdsmd service on all hosts.
> > 

> > After restarting vdsmd, connecting to the storage server may succeed.
> > 

> > Please also report this bug, we need to understand if this is the same
> > 

> > issue or another issue.
> > 

> > Vdsm should recover from such critical errors by exiting, so leaks
> > 

> > will cause service restarts (maybe every few days) instead of downtime
> > 

> > of the entire system.
> > 

> > Nir

Attachment: publickey - dmwhite823@protonmail.com - 0x320CD582.asc
Description: application/pgp-keys

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/5KWKT6WLEIDE42JF25E5XT34AENXNOJC/

Reply via email to