Re: [ovirt-users] ha-agent and broker continually crashing after 4.2 update

2018-01-15 Thread Martin Sivak 
I actually do not agree with Simone here. The fix he talks about adds
a call to prepareImage, but your log clearly shows that prepareImage
is the call that fails:

Jan 12 16:52:36 cultivar0 journal: vdsm storage.Dispatcher ERROR
FINISH prepareImage error=Volume does not exist:
(u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',)

I have to ask how old the environment is. Was it by any chance
installed back in 3.3/3.4 days and upgraded since then?

Martin

On Mon, Jan 15, 2018 at 10:17 AM, Simone Tiraboschi  wrote:
>
>
> On Fri, Jan 12, 2018 at 9:54 PM, Jayme  wrote:
>>
>> recently upgraded to 4.2 and had some problems with engine vm running, got
>> that cleared up now my only remaining issue is that now it seems
>> ovirt-ha-broker and ovirt-ha-agent are continually crashing on all three of
>> my hosts.  Everything is up and working fine otherwise, all VMs running and
>> hosted engine VM is running along with interface etc.
>
>
> I think it's due to https://bugzilla.redhat.com/show_bug.cgi?id=1527394 with
> got recently fixed.
> ovirt-hosted-engine-ha-2.2.3 should address it, please let us know if not.
>
>
>>
>>
>> Jan 12 16:52:34 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH
>> prepareImage error=Volume does not exist:
>> (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',)
>> Jan 12 16:52:34 cultivar0 python: detected unhandled Python exception in
>> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
>> Jan 12 16:52:34 cultivar0 abrt-server: Not saving repeating crash in
>> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
>> Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service: main process
>> exited, code=exited, status=1/FAILURE
>> Jan 12 16:52:34 cultivar0 systemd: Unit ovirt-ha-broker.service entered
>> failed state.
>> Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service failed.
>> Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service holdoff time
>> over, scheduling restart.
>> Jan 12 16:52:34 cultivar0 systemd: Cannot add dependency job for unit
>> lvm2-lvmetad.socket, ignoring: Unit is masked.
>> Jan 12 16:52:34 cultivar0 systemd: Started oVirt Hosted Engine High
>> Availability Communications Broker.
>> Jan 12 16:52:34 cultivar0 systemd: Starting oVirt Hosted Engine High
>> Availability Communications Broker...
>> Jan 12 16:52:36 cultivar0 journal: vdsm storage.TaskManager.Task ERROR
>> (Task='73141dec-9d8f-4164-9c4e-67c43a102eff') Unexpected error#012Traceback
>> (most recent call last):#012  File
>> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
>> _run#012return fn(*args, **kargs)#012  File "", line 2, in
>> prepareImage#012  File
>> "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in
>> method#012ret = func(*args, **kwargs)#012  File
>> "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3162, in
>> prepareImage#012raise
>> se.VolumeDoesNotExist(leafUUID)#012VolumeDoesNotExist: Volume does not
>> exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',)
>> Jan 12 16:52:36 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH
>> prepareImage error=Volume does not exist:
>> (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',)
>> Jan 12 16:52:36 cultivar0 python: detected unhandled Python exception in
>> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
>> Jan 12 16:52:36 cultivar0 abrt-server: Not saving repeating crash in
>> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
>> Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service: main process
>> exited, code=exited, status=1/FAILURE
>> Jan 12 16:52:36 cultivar0 systemd: Unit ovirt-ha-broker.service entered
>> failed state.
>> Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service failed.
>>
>> Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service holdoff time
>> over, scheduling restart.
>> Jan 12 16:52:36 cultivar0 systemd: Cannot add dependency job for unit
>> lvm2-lvmetad.socket, ignoring: Unit is masked.
>> Jan 12 16:52:36 cultivar0 systemd: Started oVirt Hosted Engine High
>> Availability Communications Broker.
>> Jan 12 16:52:36 cultivar0 systemd: Starting oVirt Hosted Engine High
>> Availability Communications Broker...
>> Jan 12 16:52:37 cultivar0 journal: vdsm storage.TaskManager.Task ERROR
>> (Task='bc7af1e2-0ab2-4164-ae88-d2bee03500f9') Unexpected error#012Traceback
>> (most recent call last):#012  File
>> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
>> _run#012return fn(*args, **kargs)#012  File "", line 2, in
>> prepareImage#012  File
>> "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in
>> method#012ret = func(*args, **kwargs)#012  File
>> "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3162, in
>> prepareImage#012raise
>> se.VolumeDoesNotExist(leafUUID)#012VolumeDoesNotExist: Volume does not
>> exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',)
>> Jan 12 16:52:37 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH
>> prepareImage error=Volume does

Re: [ovirt-users] ha-agent and broker continually crashing after 4.2 update

2018-01-15 Thread Simone Tiraboschi 
On Fri, Jan 12, 2018 at 9:54 PM, Jayme  wrote:

> recently upgraded to 4.2 and had some problems with engine vm running, got
> that cleared up now my only remaining issue is that now it seems
> ovirt-ha-broker and ovirt-ha-agent are continually crashing on all three of
> my hosts.  Everything is up and working fine otherwise, all VMs running and
> hosted engine VM is running along with interface etc.
>

I think it's due to https://bugzilla.redhat.com/show_bug.cgi?id=1527394
with got recently fixed.
ovirt-hosted-engine-ha-2.2.3 should address it, please let us know if not.



>
> Jan 12 16:52:34 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH
> prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-
> f5b7ec1f1cf8',)
> Jan 12 16:52:34 cultivar0 python: detected unhandled Python exception in
> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
> Jan 12 16:52:34 cultivar0 abrt-server: Not saving repeating crash in
> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
> Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service: main process
> exited, code=exited, status=1/FAILURE
> Jan 12 16:52:34 cultivar0 systemd: Unit ovirt-ha-broker.service entered
> failed state.
> Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service failed.
> Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service holdoff time
> over, scheduling restart.
> Jan 12 16:52:34 cultivar0 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Jan 12 16:52:34 cultivar0 systemd: Started oVirt Hosted Engine High
> Availability Communications Broker.
> Jan 12 16:52:34 cultivar0 systemd: Starting oVirt Hosted Engine High
> Availability Communications Broker...
> Jan 12 16:52:36 cultivar0 journal: vdsm storage.TaskManager.Task ERROR
> (Task='73141dec-9d8f-4164-9c4e-67c43a102eff') Unexpected
> error#012Traceback (most recent call last):#012  File
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
> _run#012return fn(*args, **kargs)#012  File "", line 2, in
> prepareImage#012  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py",
> line 48, in method#012ret = func(*args, **kwargs)#012  File
> "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3162, in
> prepareImage#012raise 
> se.VolumeDoesNotExist(leafUUID)#012VolumeDoesNotExist:
> Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',)
> Jan 12 16:52:36 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH
> prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-
> f5b7ec1f1cf8',)
> Jan 12 16:52:36 cultivar0 python: detected unhandled Python exception in
> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
> Jan 12 16:52:36 cultivar0 abrt-server: Not saving repeating crash in
> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
> Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service: main process
> exited, code=exited, status=1/FAILURE
> Jan 12 16:52:36 cultivar0 systemd: Unit ovirt-ha-broker.service entered
> failed state.
> Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service failed.
>
> Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service holdoff time
> over, scheduling restart.
> Jan 12 16:52:36 cultivar0 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Jan 12 16:52:36 cultivar0 systemd: Started oVirt Hosted Engine High
> Availability Communications Broker.
> Jan 12 16:52:36 cultivar0 systemd: Starting oVirt Hosted Engine High
> Availability Communications Broker...
> Jan 12 16:52:37 cultivar0 journal: vdsm storage.TaskManager.Task ERROR
> (Task='bc7af1e2-0ab2-4164-ae88-d2bee03500f9') Unexpected
> error#012Traceback (most recent call last):#012  File
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
> _run#012return fn(*args, **kargs)#012  File "", line 2, in
> prepareImage#012  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py",
> line 48, in method#012ret = func(*args, **kwargs)#012  File
> "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3162, in
> prepareImage#012raise 
> se.VolumeDoesNotExist(leafUUID)#012VolumeDoesNotExist:
> Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',)
> Jan 12 16:52:37 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH
> prepareImage error=Volume does not exist: (u'8582bdfc-ef54-47af-9f1e-
> f5b7ec1f1cf8',)
> Jan 12 16:52:37 cultivar0 python: detected unhandled Python exception in
> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
> Jan 12 16:52:38 cultivar0 abrt-server: Not saving repeating crash in
> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
> Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service: main process
> exited, code=exited, status=1/FAILURE
> Jan 12 16:52:38 cultivar0 systemd: Unit ovirt-ha-broker.service entered
> failed state.
> Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service failed.
> Jan 12 16:52:38 cultivar0 systemd:

[ovirt-users] ha-agent and broker continually crashing after 4.2 update

2018-01-12 Thread Jayme 
recently upgraded to 4.2 and had some problems with engine vm running, got
that cleared up now my only remaining issue is that now it seems
ovirt-ha-broker and ovirt-ha-agent are continually crashing on all three of
my hosts.  Everything is up and working fine otherwise, all VMs running and
hosted engine VM is running along with interface etc.

Jan 12 16:52:34 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH
prepareImage error=Volume does not exist:
(u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',)
Jan 12 16:52:34 cultivar0 python: detected unhandled Python exception in
'/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Jan 12 16:52:34 cultivar0 abrt-server: Not saving repeating crash in
'/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service: main process
exited, code=exited, status=1/FAILURE
Jan 12 16:52:34 cultivar0 systemd: Unit ovirt-ha-broker.service entered
failed state.
Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service failed.
Jan 12 16:52:34 cultivar0 systemd: ovirt-ha-broker.service holdoff time
over, scheduling restart.
Jan 12 16:52:34 cultivar0 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Jan 12 16:52:34 cultivar0 systemd: Started oVirt Hosted Engine High
Availability Communications Broker.
Jan 12 16:52:34 cultivar0 systemd: Starting oVirt Hosted Engine High
Availability Communications Broker...
Jan 12 16:52:36 cultivar0 journal: vdsm storage.TaskManager.Task ERROR
(Task='73141dec-9d8f-4164-9c4e-67c43a102eff') Unexpected error#012Traceback
(most recent call last):#012  File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run#012return fn(*args, **kargs)#012  File "", line 2, in
prepareImage#012  File
"/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in
method#012ret = func(*args, **kwargs)#012  File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3162, in
prepareImage#012raise
se.VolumeDoesNotExist(leafUUID)#012VolumeDoesNotExist: Volume does not
exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',)
Jan 12 16:52:36 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH
prepareImage error=Volume does not exist:
(u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',)
Jan 12 16:52:36 cultivar0 python: detected unhandled Python exception in
'/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Jan 12 16:52:36 cultivar0 abrt-server: Not saving repeating crash in
'/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service: main process
exited, code=exited, status=1/FAILURE
Jan 12 16:52:36 cultivar0 systemd: Unit ovirt-ha-broker.service entered
failed state.
Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service failed.

Jan 12 16:52:36 cultivar0 systemd: ovirt-ha-broker.service holdoff time
over, scheduling restart.
Jan 12 16:52:36 cultivar0 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Jan 12 16:52:36 cultivar0 systemd: Started oVirt Hosted Engine High
Availability Communications Broker.
Jan 12 16:52:36 cultivar0 systemd: Starting oVirt Hosted Engine High
Availability Communications Broker...
Jan 12 16:52:37 cultivar0 journal: vdsm storage.TaskManager.Task ERROR
(Task='bc7af1e2-0ab2-4164-ae88-d2bee03500f9') Unexpected error#012Traceback
(most recent call last):#012  File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run#012return fn(*args, **kargs)#012  File "", line 2, in
prepareImage#012  File
"/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in
method#012ret = func(*args, **kwargs)#012  File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3162, in
prepareImage#012raise
se.VolumeDoesNotExist(leafUUID)#012VolumeDoesNotExist: Volume does not
exist: (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',)
Jan 12 16:52:37 cultivar0 journal: vdsm storage.Dispatcher ERROR FINISH
prepareImage error=Volume does not exist:
(u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',)
Jan 12 16:52:37 cultivar0 python: detected unhandled Python exception in
'/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Jan 12 16:52:38 cultivar0 abrt-server: Not saving repeating crash in
'/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service: main process
exited, code=exited, status=1/FAILURE
Jan 12 16:52:38 cultivar0 systemd: Unit ovirt-ha-broker.service entered
failed state.
Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service failed.
Jan 12 16:52:38 cultivar0 systemd: ovirt-ha-broker.service holdoff time
over, scheduling restart.
Jan 12 16:52:38 cultivar0 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Jan 12 16:52:38 cultivar0 systemd: start request repeated too quickly for
ovirt-ha-broker.service
Jan 12 16:52:38 cultivar0 systemd: Failed to start oVirt Hosted Engine High
Availability Communications Broker.
Jan 12 16:52:38