Re: [ovirt-users] oVirt 3.6.2 issue when updating to gluster 3.7.8

[Sahina Bose]

On 02/17/2016 04:57 PM, Luiz Claudio Prazeres Goncalves wrote:


Hi, I realised that gluster 3.7.8 was released for GA. So I updated 
manually using "yum -y install glusterfs*" gluster get's updated 
normally, but  unfortunately things stopped to work completely. 
ovirt_hosted_engine_ha was not able to connect to my gluster storage 
domain (as you can see below). I'm not using ovirt + gluster in 
hyperconverged way. I have 3 external gluster hosts.



What does the mount log on the hypervisor say? Could you attach that?



As a workaround I've executed "*yum*downgrade glusterfs*" and after 
doing this ovirt started to work again...but as a collateral effect 
now I can't migrate my vm's anymore.  On the vdsm logs I can see the 
following errors when trying to manually migrate a any VM Anyone 
knows how to fix it?


Thread-66055::DEBUG::2016-02-17 
11:05:57,497::migration::453::virt.vm::(stop) 
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::stopping migration 
downtime thread


Thread-66055::ERROR::2016-02-17 
11:05:57,497::migration::208::virt.vm::(_recover) 
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::unable to connect to 
server at 'kvm2.brightsquid.com:49152 
': No route to host


Thread-66055::DEBUG::2016-02-17 
11:05:57,497::stompreactor::389::jsonrpc.AsyncoreClient::(send) 
Sending response


Thread-66056::DEBUG::2016-02-17 
11:05:57,498::migration::450::virt.vm::(run) 
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::migration downtime thread 
exiting


Thread-66055::DEBUG::2016-02-17 
11:05:57,540::__init__::206::jsonrpc.Notification::(emit) Sending 
event {"params": {"notify_time": 4327804740, 
"05681896-a76a-4ae1-879e-8fe5d28634e1": {"status": "Migration 
Source"}}, "jsonrpc": "2.0", "method": 
"|virt|VM_status|05681896-a76a-4ae1-879e-8fe5d28634e1"}


Thread-66055::ERROR::2016-02-17 
11:05:57,541::migration::310::virt.vm::(run) 
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::Failedto migrate


Traceback (most recent call last):

  File "/usr/share/vdsm/virt/migration.py", line 294, in run

self._startUnderlyingMigration(time.time())

  File "/usr/share/vdsm/virt/migration.py", line 364, in 
_startUnderlyingMigration


self._perform_migration(duri, muri)

  File "/usr/share/vdsm/virt/migration.py", line 403, in 
_perform_migration


self._vm._dom.migrateToURI3(duri, params, flags)

  File "/usr/share/vdsm/virt/virdomain.py", line 68, in f

ret = attr(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", 
line 124, in wrapper


ret = f(*args, **kwargs)

  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1836, in 
migrateToURI3


if ret == -1: raise libvirtError ('virDomainMigrateToURI3() 
failed', dom=self)


libvirtError: unable to connect to server at 
'kvm2.brightsquid.com:49152 ': No 
route to host


Thread-68::DEBUG::2016-02-17 
11:05:57,652::fileSD::173::Storage.Misc.excCmd::(getReadDelay) 
/usr/bin/taskset --cpu-list 0-39 /usr/bin/dd 
if=/rhev/data-center/mnt/gluster2.brightsquid.com:_home_export_iso/61827b7b-e255-44f5-a791-482a144be29f/dom_md/metadata 
iflag=direct of=/dev/null bs=4096 count=1 (cwd None)


Thread-68::DEBUG::2016-02-17 
11:05:57,662::fileSD::173::Storage.Misc.excCmd::(getReadDelay) 
SUCCESS:  = '0+1 records in\n0+1 records out\n341 bytes (341 B) 
copied, 0.000713967 s, 478 kB/s\n';  = 0











Issue to connect to the gluster storage.
*
Feb 17 01:25:40 kvm2 ovirt-ha-agent: 
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting storage 
server
Feb 17 01:25:40 kvm2 ovirt-ha-agent: 
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting storage 
server
Feb 17 01:25:40 kvm2 journal: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Connection to 
storage server failed' - trying to restart agent
Feb 17 01:25:40 kvm2 ovirt-ha-agent: 
ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Connection to 
storage server failed' - trying to restart agent
Feb 17 01:25:41 kvm2 ovirt-ha-broker: 
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection 
established
Feb 17 01:25:41 kvm2 journal: ovirt-ha-broker 
ovirt_hosted_engine_ha.broker.listener.ConnectionHandler ERROR Error 
handling request, data: 'set-storage-domain FilesystemBackend 
dom_type=glusterfs 
sd_uuid=7d376952-312b-4539-b809-a8fa740f7883'#012Traceback (most 
recent call last):#012  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 166, in handle#012data)#012  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", 
line 299, in _dispatch#012.set_storage_domain(client, sd_type, 
**options)#012  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", 
line 66, in set_storage_domain#012 
 self._backends[client].connect()#012  File 

[ovirt-users] oVirt 3.6.2 issue when updating to gluster 3.7.8

[Luiz Claudio Prazeres Goncalves]

Hi, I realised that gluster 3.7.8 was released for GA. So I updated
manually using "yum -y install glusterfs*" gluster get's updated normally,
but  unfortunately things stopped to work completely.
ovirt_hosted_engine_ha was not able to connect to my gluster storage domain
(as you can see below). I'm not using ovirt + gluster in hyperconverged
way. I have 3 external gluster hosts.

As a workaround I've executed "*yum* downgrade glusterfs*" and after doing
this ovirt started to work again...but as a collateral effect now I can't
migrate my vm's anymore.  On the vdsm logs I can see the following errors
when trying to manually migrate a any VM Anyone knows how to fix it?

Thread-66055::DEBUG::2016-02-17
11:05:57,497::migration::453::virt.vm::(stop)
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::stopping migration downtime
thread

Thread-66055::ERROR::2016-02-17
11:05:57,497::migration::208::virt.vm::(_recover)
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::unable to connect to server at
'kvm2.brightsquid.com:49152': No route to host

Thread-66055::DEBUG::2016-02-17
11:05:57,497::stompreactor::389::jsonrpc.AsyncoreClient::(send) Sending
response

Thread-66056::DEBUG::2016-02-17
11:05:57,498::migration::450::virt.vm::(run)
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::migration downtime thread
exiting

Thread-66055::DEBUG::2016-02-17
11:05:57,540::__init__::206::jsonrpc.Notification::(emit) Sending event
{"params": {"notify_time": 4327804740,
"05681896-a76a-4ae1-879e-8fe5d28634e1": {"status": "Migration Source"}},
"jsonrpc": "2.0", "method":
"|virt|VM_status|05681896-a76a-4ae1-879e-8fe5d28634e1"}

Thread-66055::ERROR::2016-02-17
11:05:57,541::migration::310::virt.vm::(run)
vmId=`05681896-a76a-4ae1-879e-8fe5d28634e1`::Failed to migrate

Traceback (most recent call last):

  File "/usr/share/vdsm/virt/migration.py", line 294, in run

self._startUnderlyingMigration(time.time())

  File "/usr/share/vdsm/virt/migration.py", line 364, in
_startUnderlyingMigration

self._perform_migration(duri, muri)

  File "/usr/share/vdsm/virt/migration.py", line 403, in _perform_migration

self._vm._dom.migrateToURI3(duri, params, flags)

  File "/usr/share/vdsm/virt/virdomain.py", line 68, in f

ret = attr(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line
124, in wrapper

ret = f(*args, **kwargs)

  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1836, in
migrateToURI3

if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed',
dom=self)

libvirtError: unable to connect to server at 'kvm2.brightsquid.com:49152':
No route to host

Thread-68::DEBUG::2016-02-17
11:05:57,652::fileSD::173::Storage.Misc.excCmd::(getReadDelay)
/usr/bin/taskset --cpu-list 0-39 /usr/bin/dd
if=/rhev/data-center/mnt/gluster2.brightsquid.com:_home_export_iso/61827b7b-e255-44f5-a791-482a144be29f/dom_md/metadata
iflag=direct of=/dev/null bs=4096 count=1 (cwd None)

Thread-68::DEBUG::2016-02-17
11:05:57,662::fileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS:
 = '0+1 records in\n0+1 records out\n341 bytes (341 B) copied,
0.000713967 s, 478 kB/s\n';  = 0










Issue to connect to the gluster storage.
*
Feb 17 01:25:40 kvm2 ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting
storage server
Feb 17 01:25:40 kvm2 ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting
storage server
Feb 17 01:25:40 kvm2 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Connection to
storage server failed' - trying to restart agent
Feb 17 01:25:40 kvm2 ovirt-ha-agent:
ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Connection to
storage server failed' - trying to restart agent
Feb 17 01:25:41 kvm2 ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
established
Feb 17 01:25:41 kvm2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.listener.ConnectionHandler ERROR Error
handling request, data: 'set-storage-domain FilesystemBackend
dom_type=glusterfs
sd_uuid=7d376952-312b-4539-b809-a8fa740f7883'#012Traceback (most recent
call last):#012  File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 166, in handle#012data)#012  File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 299, in _dispatch#012.set_storage_domain(client, sd_type,
**options)#012  File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 66, in set_storage_domain#012self._backends[client].connect()#012
 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 456, in connect#012self._dom_type)#012  File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 108, in get_domain_path#012" in {1}".format(sd_uuid,