Re: [ovirt-users] migration failed - "Cannot get interface MTU on 'vdsmbr_...'

2017-03-29 Thread Dan Kenigsberg 
Thanks for following up on this. We need to put a little more effort on

Bug 1400952 - [RFE] Resolve listen IP for graphics attached to Open
vSwitch network

so that the hook is no longer needed.

Please let us know how oVirt+OvS is working for you!


On Wed, Mar 29, 2017 at 6:17 PM, Devin A. Bougie
 wrote:
> Just incase anyone else runs into this, you need to set 
> "migration_ovs_hook_enabled=True" in vdsm.conf.  It seems the vdsm.conf 
> created by "hosted-engine --deploy" did not list all of the options, so I 
> overlooked this one.
>
> Thanks for all the help,
> Devin
>
> On Mar 27, 2017, at 11:10 AM, Devin A. Bougie  
> wrote:
>> Hi, All.  We have a new oVirt 4.1.1 cluster up with the OVS switch type.  
>> Everything seems to be working great, except for live migration.
>>
>> I believe the red flag in vdsm.log on the source is:
>> Cannot get interface MTU on 'vdsmbr_QwORbsw2': No such device (migration:287)
>>
>> Which results from vdsm assigning an arbitrary bridge name to each ovs 
>> bridge.
>>
>> Please see below for more details on the bridges and excerpts from the logs. 
>>  Any help would be greatly appreciated.
>>
>> Many thanks,
>> Devin
>>
>> SOURCE OVS BRIDGES:
>> # ovs-vsctl show
>> 6d96d9a5-e30d-455b-90c7-9e9632574695
>>Bridge "vdsmbr_QwORbsw2"
>>Port "vdsmbr_QwORbsw2"
>>Interface "vdsmbr_QwORbsw2"
>>type: internal
>>Port "vnet0"
>>Interface "vnet0"
>>Port classepublic
>>Interface classepublic
>>type: internal
>>Port "ens1f0"
>>Interface "ens1f0"
>>Bridge "vdsmbr_9P7ZYKWn"
>>Port ovirtmgmt
>>Interface ovirtmgmt
>>type: internal
>>Port "ens1f1"
>>Interface "ens1f1"
>>Port "vdsmbr_9P7ZYKWn"
>>Interface "vdsmbr_9P7ZYKWn"
>>type: internal
>>ovs_version: "2.7.0"
>>
>> DESTINATION OVS BRIDGES:
>> # ovs-vsctl show
>> f66d765d-712a-4c81-b18e-da1acc9cfdde
>>Bridge "vdsmbr_vdpp0dOd"
>>Port "vdsmbr_vdpp0dOd"
>>Interface "vdsmbr_vdpp0dOd"
>>type: internal
>>Port "ens1f0"
>>Interface "ens1f0"
>>Port classepublic
>>Interface classepublic
>>type: internal
>>Bridge "vdsmbr_3sEwEKd1"
>>Port "vdsmbr_3sEwEKd1"
>>Interface "vdsmbr_3sEwEKd1"
>>type: internal
>>Port "ens1f1"
>>Interface "ens1f1"
>>Port ovirtmgmt
>>Interface ovirtmgmt
>>type: internal
>>ovs_version: "2.7.0"
>>
>>
>> SOURCE VDSM LOG:
>> ...
>> 2017-03-27 10:57:02,567-0400 INFO  (jsonrpc/1) [vdsm.api] START migrate 
>> args=(, {u'incomingLimit': 2, u'src': 
>> u'192.168.55.84', u'dstqemu': u'192.168.55.81', u'autoConverge': u'false', 
>> u'tunneled': u'false', u'enableGuestEvents': False, u'dst': 
>> u'lnxvirt01-p55.classe.cornell.edu:54321', u'vmId': 
>> u'cf9c5dbf-3924-47c6-b323-22ac90a1f682', u'abortOnError': u'true', 
>> u'outgoingLimit': 2, u'compressed': u'false', u'maxBandwidth': 5000, 
>> u'method': u'online', 'mode': 'remote'}) kwargs={} (api:37)
>> 2017-03-27 10:57:02,570-0400 INFO  (jsonrpc/1) [vdsm.api] FINISH migrate 
>> return={'status': {'message': 'Migration in progress', 'code': 0}, 
>> 'progress': 0} (api:43)
>> 2017-03-27 10:57:02,570-0400 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC 
>> call VM.migrate succeeded in 0.01 seconds (__init__:515)
>> 2017-03-27 10:57:03,028-0400 INFO  (migsrc/cf9c5dbf) [virt.vm] 
>> (vmId='cf9c5dbf-3924-47c6-b323-22ac90a1f682') Creation of destination VM 
>> took: 0 seconds (migration:455)
>> 2017-03-27 10:57:03,028-0400 INFO  (migsrc/cf9c5dbf) [virt.vm] 
>> (vmId='cf9c5dbf-3924-47c6-b323-22ac90a1f682') starting migration to 
>> qemu+tls://lnxvirt01-p55.classe.cornell.edu/system with miguri 
>> tcp://192.168.55.81 (migration:480)
>> 2017-03-27 10:57:03,224-0400 ERROR (migsrc/cf9c5dbf) [virt.vm] 
>> (vmId='cf9c5dbf-3924-47c6-b323-22ac90a1f682') Cannot get interface MTU on 
>> 'vdsmbr_QwORbsw2': No such device (migration:287)
>> 2017-03-27 10:57:03,322-0400 ERROR (migsrc/cf9c5dbf) [virt.vm] 
>> (vmId='cf9c5dbf-3924-47c6-b323-22ac90a1f682') Failed to migrate 
>> (migration:429)
>> Traceback (most recent call last):
>>  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 411, 
>> in run
>>self._startUnderlyingMigration(time.time())
>>  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 489, 
>> in _startUnderlyingMigration
>>self._perform_with_downtime_thread(duri, muri)
>>  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 555, 
>> in _perform_with_downtime_thread
>>self._perform_migration(duri, muri)
>>  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 528, 
>> in _perform_migration
>>self._vm._dom.migrateToURI3(duri, params, flags)
>>  File

Re: [ovirt-users] migration failed - "Cannot get interface MTU on 'vdsmbr_...'

2017-03-29 Thread Devin A. Bougie 
Just incase anyone else runs into this, you need to set 
"migration_ovs_hook_enabled=True" in vdsm.conf.  It seems the vdsm.conf created 
by "hosted-engine --deploy" did not list all of the options, so I overlooked 
this one.

Thanks for all the help,
Devin

On Mar 27, 2017, at 11:10 AM, Devin A. Bougie  wrote:
> Hi, All.  We have a new oVirt 4.1.1 cluster up with the OVS switch type.  
> Everything seems to be working great, except for live migration.
> 
> I believe the red flag in vdsm.log on the source is:
> Cannot get interface MTU on 'vdsmbr_QwORbsw2': No such device (migration:287)
> 
> Which results from vdsm assigning an arbitrary bridge name to each ovs bridge.
> 
> Please see below for more details on the bridges and excerpts from the logs.  
> Any help would be greatly appreciated.
> 
> Many thanks,
> Devin
> 
> SOURCE OVS BRIDGES:
> # ovs-vsctl show
> 6d96d9a5-e30d-455b-90c7-9e9632574695
>Bridge "vdsmbr_QwORbsw2"
>Port "vdsmbr_QwORbsw2"
>Interface "vdsmbr_QwORbsw2"
>type: internal
>Port "vnet0"
>Interface "vnet0"
>Port classepublic
>Interface classepublic
>type: internal
>Port "ens1f0"
>Interface "ens1f0"
>Bridge "vdsmbr_9P7ZYKWn"
>Port ovirtmgmt
>Interface ovirtmgmt
>type: internal
>Port "ens1f1"
>Interface "ens1f1"
>Port "vdsmbr_9P7ZYKWn"
>Interface "vdsmbr_9P7ZYKWn"
>type: internal
>ovs_version: "2.7.0"
> 
> DESTINATION OVS BRIDGES:
> # ovs-vsctl show
> f66d765d-712a-4c81-b18e-da1acc9cfdde
>Bridge "vdsmbr_vdpp0dOd"
>Port "vdsmbr_vdpp0dOd"
>Interface "vdsmbr_vdpp0dOd"
>type: internal
>Port "ens1f0"
>Interface "ens1f0"
>Port classepublic
>Interface classepublic
>type: internal
>Bridge "vdsmbr_3sEwEKd1"
>Port "vdsmbr_3sEwEKd1"
>Interface "vdsmbr_3sEwEKd1"
>type: internal
>Port "ens1f1"
>Interface "ens1f1"
>Port ovirtmgmt
>Interface ovirtmgmt
>type: internal
>ovs_version: "2.7.0"
> 
> 
> SOURCE VDSM LOG:
> ...
> 2017-03-27 10:57:02,567-0400 INFO  (jsonrpc/1) [vdsm.api] START migrate 
> args=(, {u'incomingLimit': 2, u'src': 
> u'192.168.55.84', u'dstqemu': u'192.168.55.81', u'autoConverge': u'false', 
> u'tunneled': u'false', u'enableGuestEvents': False, u'dst': 
> u'lnxvirt01-p55.classe.cornell.edu:54321', u'vmId': 
> u'cf9c5dbf-3924-47c6-b323-22ac90a1f682', u'abortOnError': u'true', 
> u'outgoingLimit': 2, u'compressed': u'false', u'maxBandwidth': 5000, 
> u'method': u'online', 'mode': 'remote'}) kwargs={} (api:37)
> 2017-03-27 10:57:02,570-0400 INFO  (jsonrpc/1) [vdsm.api] FINISH migrate 
> return={'status': {'message': 'Migration in progress', 'code': 0}, 
> 'progress': 0} (api:43)
> 2017-03-27 10:57:02,570-0400 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC 
> call VM.migrate succeeded in 0.01 seconds (__init__:515)
> 2017-03-27 10:57:03,028-0400 INFO  (migsrc/cf9c5dbf) [virt.vm] 
> (vmId='cf9c5dbf-3924-47c6-b323-22ac90a1f682') Creation of destination VM 
> took: 0 seconds (migration:455)
> 2017-03-27 10:57:03,028-0400 INFO  (migsrc/cf9c5dbf) [virt.vm] 
> (vmId='cf9c5dbf-3924-47c6-b323-22ac90a1f682') starting migration to 
> qemu+tls://lnxvirt01-p55.classe.cornell.edu/system with miguri 
> tcp://192.168.55.81 (migration:480)
> 2017-03-27 10:57:03,224-0400 ERROR (migsrc/cf9c5dbf) [virt.vm] 
> (vmId='cf9c5dbf-3924-47c6-b323-22ac90a1f682') Cannot get interface MTU on 
> 'vdsmbr_QwORbsw2': No such device (migration:287)
> 2017-03-27 10:57:03,322-0400 ERROR (migsrc/cf9c5dbf) [virt.vm] 
> (vmId='cf9c5dbf-3924-47c6-b323-22ac90a1f682') Failed to migrate 
> (migration:429)
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 411, in 
> run
>self._startUnderlyingMigration(time.time())
>  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 489, in 
> _startUnderlyingMigration
>self._perform_with_downtime_thread(duri, muri)
>  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 555, in 
> _perform_with_downtime_thread
>self._perform_migration(duri, muri)
>  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 528, in 
> _perform_migration
>self._vm._dom.migrateToURI3(duri, params, flags)
>  File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 69, in f
>ret = attr(*args, **kwargs)
>  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, 
> in wrapper
>ret = f(*args, **kwargs)
>  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper
>return func(inst, *args, **kwargs)
>  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1939, in 
> migrateToURI3
>if ret == -1: raise

[ovirt-users] migration failed - "Cannot get interface MTU on 'vdsmbr_...'

2017-03-27 Thread Devin A. Bougie 
Hi, All.  We have a new oVirt 4.1.1 cluster up with the OVS switch type.  
Everything seems to be working great, except for live migration.

I believe the red flag in vdsm.log on the source is:
Cannot get interface MTU on 'vdsmbr_QwORbsw2': No such device (migration:287)

Which results from vdsm assigning an arbitrary bridge name to each ovs bridge.

Please see below for more details on the bridges and excerpts from the logs.  
Any help would be greatly appreciated.

Many thanks,
Devin

SOURCE OVS BRIDGES:
# ovs-vsctl show
6d96d9a5-e30d-455b-90c7-9e9632574695
Bridge "vdsmbr_QwORbsw2"
Port "vdsmbr_QwORbsw2"
Interface "vdsmbr_QwORbsw2"
type: internal
Port "vnet0"
Interface "vnet0"
Port classepublic
Interface classepublic
type: internal
Port "ens1f0"
Interface "ens1f0"
Bridge "vdsmbr_9P7ZYKWn"
Port ovirtmgmt
Interface ovirtmgmt
type: internal
Port "ens1f1"
Interface "ens1f1"
Port "vdsmbr_9P7ZYKWn"
Interface "vdsmbr_9P7ZYKWn"
type: internal
ovs_version: "2.7.0"

DESTINATION OVS BRIDGES:
# ovs-vsctl show
f66d765d-712a-4c81-b18e-da1acc9cfdde
Bridge "vdsmbr_vdpp0dOd"
Port "vdsmbr_vdpp0dOd"
Interface "vdsmbr_vdpp0dOd"
type: internal
Port "ens1f0"
Interface "ens1f0"
Port classepublic
Interface classepublic
type: internal
Bridge "vdsmbr_3sEwEKd1"
Port "vdsmbr_3sEwEKd1"
Interface "vdsmbr_3sEwEKd1"
type: internal
Port "ens1f1"
Interface "ens1f1"
Port ovirtmgmt
Interface ovirtmgmt
type: internal
ovs_version: "2.7.0"


SOURCE VDSM LOG:
...
2017-03-27 10:57:02,567-0400 INFO  (jsonrpc/1) [vdsm.api] START migrate 
args=(, {u'incomingLimit': 2, u'src': 
u'192.168.55.84', u'dstqemu': u'192.168.55.81', u'autoConverge': u'false', 
u'tunneled': u'false', u'enableGuestEvents': False, u'dst': 
u'lnxvirt01-p55.classe.cornell.edu:54321', u'vmId': 
u'cf9c5dbf-3924-47c6-b323-22ac90a1f682', u'abortOnError': u'true', 
u'outgoingLimit': 2, u'compressed': u'false', u'maxBandwidth': 5000, u'method': 
u'online', 'mode': 'remote'}) kwargs={} (api:37)
2017-03-27 10:57:02,570-0400 INFO  (jsonrpc/1) [vdsm.api] FINISH migrate 
return={'status': {'message': 'Migration in progress', 'code': 0}, 'progress': 
0} (api:43)
2017-03-27 10:57:02,570-0400 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call 
VM.migrate succeeded in 0.01 seconds (__init__:515)
2017-03-27 10:57:03,028-0400 INFO  (migsrc/cf9c5dbf) [virt.vm] 
(vmId='cf9c5dbf-3924-47c6-b323-22ac90a1f682') Creation of destination VM took: 
0 seconds (migration:455)
2017-03-27 10:57:03,028-0400 INFO  (migsrc/cf9c5dbf) [virt.vm] 
(vmId='cf9c5dbf-3924-47c6-b323-22ac90a1f682') starting migration to 
qemu+tls://lnxvirt01-p55.classe.cornell.edu/system with miguri 
tcp://192.168.55.81 (migration:480)
2017-03-27 10:57:03,224-0400 ERROR (migsrc/cf9c5dbf) [virt.vm] 
(vmId='cf9c5dbf-3924-47c6-b323-22ac90a1f682') Cannot get interface MTU on 
'vdsmbr_QwORbsw2': No such device (migration:287)
2017-03-27 10:57:03,322-0400 ERROR (migsrc/cf9c5dbf) [virt.vm] 
(vmId='cf9c5dbf-3924-47c6-b323-22ac90a1f682') Failed to migrate (migration:429)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 411, in 
run
self._startUnderlyingMigration(time.time())
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 489, in 
_startUnderlyingMigration
self._perform_with_downtime_thread(duri, muri)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 555, in 
_perform_with_downtime_thread
self._perform_migration(duri, muri)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 528, in 
_perform_migration
self._vm._dom.migrateToURI3(duri, params, flags)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 69, in f
ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, 
in wrapper
ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper
return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1939, in 
migrateToURI3
if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', 
dom=self)
libvirtError: Cannot get interface MTU on 'vdsmbr_QwORbsw2': No such device
2017-03-27 10:57:03,435-0400 INFO  (Reactor thread) 
[ProtocolDetector.AcceptorImpl] Accepted connection from ::1:33716 
(protocoldetector:72)
2017-03-27 10:57:03,452-0400 INFO  (Reactor thread) [ProtocolDetector.Detector] 
Detected protocol stomp from ::1:33716 (protocoldetector:127)
2017-03-27 10:57:03,452-0400 INFO  (Reactor thread) [Broker.StompAdapter] 
Processing CONNECT