Re: [ovirt-users] ha-agent and broker continually crashing after 4.2 update
I actually do not agree with Simone here. The fix he talks about adds a call to prepareImage, but your log clearly shows that prepareImage is the call that fails: Jan 12 16:52:36 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',) I have to ask how old the environment is. Was it by any chance installed back in 3.3/3.4 days and upgraded since then? Martin On Mon, Jan 15, 2018 at 10:17 AM, Simone Tiraboschi wrote: > > > On Fri, Jan 12, 2018 at 9:54 PM, Jayme wrote: >> >> recently upgraded to 4.2 and had some problems with engine vm running, got >> that cleared up now my only remaining issue is that now it seems >> ovirt-ha-broker and ovirt-ha-agent are continually crashing on all three of >> my hosts. Everything is up and working fine otherwise, all VMs running and >> hosted engine VM is running along with interface etc. > > > I think it's due to https://bugzilla.redhat.com/show_bug.cgi?id=1527394 with > got recently fixed. > ovirt-hosted-engine-ha-2.2.3 should address it, please let us know if not. > > >> >> >> Jan 12 16:52:34 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH >> prepareImage error=Volume does not exist: >> (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',) >> Jan 12 16:52:34 cultivar0 python: detected unhandled Python exception in >> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' >> Jan 12 16:52:34 cultivar0 abrt-server: Not saving repeating crash in >> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' >> Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service: main process >> exited, code=exited, status=1/FAILURE >> Jan 12 16:52:34 cultivar0 systemd: Unit ovirt-ha-broker.service entered >> failed state. >> Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service failed. >> Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service holdoff time >> over, scheduling restart. >> Jan 12 16:52:34 cultivar0 systemd: Cannot add dependency job for unit >> lvm2-lvmetad.socket, ignoring: Unit is masked. >> Jan 12 16:52:34 cultivar0 systemd: Started oVirt Hosted Engine High >> Availability Communications Broker. >> Jan 12 16:52:34 cultivar0 systemd: Starting oVirt Hosted Engine High >> Availability Communications Broker... >> Jan 12 16:52:36 cultivar0 journal: vdsm storage.TaskManager.Task ERROR >> (Task='73141dec-9d8f-4164-9c4e-67c43a102eff') Unexpected error#012Traceback >> (most recent call last):#012 File >> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in >> _run#012return fn(*args, **kargs)#012 File "", line 2, in >> prepareImage#012 File >> "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in >> method#012ret = func(*args, **kwargs)#012 File >> "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3162, in >> prepareImage#012raise >> se.VolumeDoesNotExist(leafUUID)#012VolumeDoesNotExist: Volume does not >> exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',) >> Jan 12 16:52:36 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH >> prepareImage error=Volume does not exist: >> (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',) >> Jan 12 16:52:36 cultivar0 python: detected unhandled Python exception in >> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' >> Jan 12 16:52:36 cultivar0 abrt-server: Not saving repeating crash in >> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' >> Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service: main process >> exited, code=exited, status=1/FAILURE >> Jan 12 16:52:36 cultivar0 systemd: Unit ovirt-ha-broker.service entered >> failed state. >> Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service failed. >> >> Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service holdoff time >> over, scheduling restart. >> Jan 12 16:52:36 cultivar0 systemd: Cannot add dependency job for unit >> lvm2-lvmetad.socket, ignoring: Unit is masked. >> Jan 12 16:52:36 cultivar0 systemd: Started oVirt Hosted Engine High >> Availability Communications Broker. >> Jan 12 16:52:36 cultivar0 systemd: Starting oVirt Hosted Engine High >> Availability Communications Broker... >> Jan 12 16:52:37 cultivar0 journal: vdsm storage.TaskManager.Task ERROR >> (Task='bc7af1e2-0ab2-4164-ae88-d2bee03500f9') Unexpected error#012Traceback >> (most recent call last):#012 File >> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in >> _run#012return fn(*args, **kargs)#012 File "", line 2, in >> prepareImage#012 File >> "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in >> method#012ret = func(*args, **kwargs)#012 File >> "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3162, in >> prepareImage#012raise >> se.VolumeDoesNotExist(leafUUID)#012VolumeDoesNotExist: Volume does not >> exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',) >> Jan 12 16:52:37 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH >> prepareImage error=Volume does not exist: >> (u'8582bdfc-ef54-47af-9f1e-f
Re: [ovirt-users] ha-agent and broker continually crashing after 4.2 update
On Fri, Jan 12, 2018 at 9:54 PM, Jayme wrote: > recently upgraded to 4.2 and had some problems with engine vm running, got > that cleared up now my only remaining issue is that now it seems > ovirt-ha-broker and ovirt-ha-agent are continually crashing on all three of > my hosts. Everything is up and working fine otherwise, all VMs running and > hosted engine VM is running along with interface etc. > I think it's due to https://bugzilla.redhat.com/show_bug.cgi?id=1527394 with got recently fixed. ovirt-hosted-engine-ha-2.2.3 should address it, please let us know if not. > > Jan 12 16:52:34 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH > prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e- > f5b7ec1f1cf8',) > Jan 12 16:52:34 cultivar0 python: detected unhandled Python exception in > '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' > Jan 12 16:52:34 cultivar0 abrt-server: Not saving repeating crash in > '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' > Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service: main process > exited, code=exited, status=1/FAILURE > Jan 12 16:52:34 cultivar0 systemd: Unit ovirt-ha-broker.service entered > failed state. > Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service failed. > Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service holdoff time > over, scheduling restart. > Jan 12 16:52:34 cultivar0 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Jan 12 16:52:34 cultivar0 systemd: Started oVirt Hosted Engine High > Availability Communications Broker. > Jan 12 16:52:34 cultivar0 systemd: Starting oVirt Hosted Engine High > Availability Communications Broker... > Jan 12 16:52:36 cultivar0 journal: vdsm storage.TaskManager.Task ERROR > (Task='73141dec-9d8f-4164-9c4e-67c43a102eff') Unexpected > error#012Traceback (most recent call last):#012 File > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in > _run#012return fn(*args, **kargs)#012 File "", line 2, in > prepareImage#012 File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", > line 48, in method#012ret = func(*args, **kwargs)#012 File > "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3162, in > prepareImage#012raise > se.VolumeDoesNotExist(leafUUID)#012VolumeDoesNotExist: > Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',) > Jan 12 16:52:36 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH > prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e- > f5b7ec1f1cf8',) > Jan 12 16:52:36 cultivar0 python: detected unhandled Python exception in > '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' > Jan 12 16:52:36 cultivar0 abrt-server: Not saving repeating crash in > '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' > Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service: main process > exited, code=exited, status=1/FAILURE > Jan 12 16:52:36 cultivar0 systemd: Unit ovirt-ha-broker.service entered > failed state. > Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service failed. > > Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service holdoff time > over, scheduling restart. > Jan 12 16:52:36 cultivar0 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Jan 12 16:52:36 cultivar0 systemd: Started oVirt Hosted Engine High > Availability Communications Broker. > Jan 12 16:52:36 cultivar0 systemd: Starting oVirt Hosted Engine High > Availability Communications Broker... > Jan 12 16:52:37 cultivar0 journal: vdsm storage.TaskManager.Task ERROR > (Task='bc7af1e2-0ab2-4164-ae88-d2bee03500f9') Unexpected > error#012Traceback (most recent call last):#012 File > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in > _run#012return fn(*args, **kargs)#012 File "", line 2, in > prepareImage#012 File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", > line 48, in method#012ret = func(*args, **kwargs)#012 File > "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3162, in > prepareImage#012raise > se.VolumeDoesNotExist(leafUUID)#012VolumeDoesNotExist: > Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',) > Jan 12 16:52:37 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH > prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e- > f5b7ec1f1cf8',) > Jan 12 16:52:37 cultivar0 python: detected unhandled Python exception in > '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' > Jan 12 16:52:38 cultivar0 abrt-server: Not saving repeating crash in > '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' > Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service: main process > exited, code=exited, status=1/FAILURE > Jan 12 16:52:38 cultivar0 systemd: Unit ovirt-ha-broker.service entered > failed state. > Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service failed. > Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.servic
[ovirt-users] ha-agent and broker continually crashing after 4.2 update
recently upgraded to 4.2 and had some problems with engine vm running, got that cleared up now my only remaining issue is that now it seems ovirt-ha-broker and ovirt-ha-agent are continually crashing on all three of my hosts. Everything is up and working fine otherwise, all VMs running and hosted engine VM is running along with interface etc. Jan 12 16:52:34 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',) Jan 12 16:52:34 cultivar0 python: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 12 16:52:34 cultivar0 abrt-server: Not saving repeating crash in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service: main process exited, code=exited, status=1/FAILURE Jan 12 16:52:34 cultivar0 systemd: Unit ovirt-ha-broker.service entered failed state. Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service failed. Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service holdoff time over, scheduling restart. Jan 12 16:52:34 cultivar0 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Jan 12 16:52:34 cultivar0 systemd: Started oVirt Hosted Engine High Availability Communications Broker. Jan 12 16:52:34 cultivar0 systemd: Starting oVirt Hosted Engine High Availability Communications Broker... Jan 12 16:52:36 cultivar0 journal: vdsm storage.TaskManager.Task ERROR (Task='73141dec-9d8f-4164-9c4e-67c43a102eff') Unexpected error#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run#012return fn(*args, **kargs)#012 File "", line 2, in prepareImage#012 File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method#012ret = func(*args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3162, in prepareImage#012raise se.VolumeDoesNotExist(leafUUID)#012VolumeDoesNotExist: Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',) Jan 12 16:52:36 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',) Jan 12 16:52:36 cultivar0 python: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 12 16:52:36 cultivar0 abrt-server: Not saving repeating crash in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service: main process exited, code=exited, status=1/FAILURE Jan 12 16:52:36 cultivar0 systemd: Unit ovirt-ha-broker.service entered failed state. Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service failed. Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service holdoff time over, scheduling restart. Jan 12 16:52:36 cultivar0 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Jan 12 16:52:36 cultivar0 systemd: Started oVirt Hosted Engine High Availability Communications Broker. Jan 12 16:52:36 cultivar0 systemd: Starting oVirt Hosted Engine High Availability Communications Broker... Jan 12 16:52:37 cultivar0 journal: vdsm storage.TaskManager.Task ERROR (Task='bc7af1e2-0ab2-4164-ae88-d2bee03500f9') Unexpected error#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run#012return fn(*args, **kargs)#012 File "", line 2, in prepareImage#012 File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method#012ret = func(*args, **kwargs)#012 File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3162, in prepareImage#012raise se.VolumeDoesNotExist(leafUUID)#012VolumeDoesNotExist: Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',) Jan 12 16:52:37 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',) Jan 12 16:52:37 cultivar0 python: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 12 16:52:38 cultivar0 abrt-server: Not saving repeating crash in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service: main process exited, code=exited, status=1/FAILURE Jan 12 16:52:38 cultivar0 systemd: Unit ovirt-ha-broker.service entered failed state. Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service failed. Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service holdoff time over, scheduling restart. Jan 12 16:52:38 cultivar0 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Jan 12 16:52:38 cultivar0 systemd: start request repeated too quickly for ovirt-ha-broker.service Jan 12 16:52:38 cultivar0 systemd: Failed to start oVirt Hosted Engine High Availability Communications Broker. Jan 12 16:52:38 cultivar0