Re: [ovirt-users] VMs with multiple vdisks don't migrate

[Milan Zamazal]

"fsoyer"  writes:

> I tried to activate the debug mode, but the restart of libvirt crashed
> something on the host : it was no more possible to start any vm on it, and
> migration to it just never started. So I decided to restart it, and to be 
> sure,
> I've restarted all the hosts.
> And... now the migration of all VMs, simple or multi-disks, works ?!? So, 
> there
> was probably something hidden that was resetted or repaired by the global
> restart ! In french, we call that "tomber en marche" ;)

I'm always amazed how many problems in computing are eventually resolved
(and how many new ones introduced) by reboot :-).  I'm glad that it
works for you now.

Regards,
Milan
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] VMs with multiple vdisks don't migrate

[fsoyer]

I Milan,
I tried to activate the debug mode, but the restart of libvirt crashed 
something on the host : it was no more possible to start any vm on it, and 
migration to it just never started. So I decided to restart it, and to be sure, 
I've restarted all the hosts.
And... now the migration of all VMs, simple or multi-disks, works ?!? So, there 
was probably something hidden that was resetted or repaired by the global 
restart ! In french, we call that "tomber en marche" ;)

So : solved. Thank you for the wasted time !

--

Cordialement,

Frank Soyer
Mob. 06 72 28 38 53 - Fix. 05 49 50 52 34

Le Lundi, Février 26, 2018 12:59 CET, Milan Zamazal  a 
écrit:
 "fsoyer"  writes:

> I don't beleive that this is relatd to a host, tests have been done from 
> victor
> source to ginger dest and ginger to victor. I don't see problems on storage
> (gluster 3.12 native managed by ovirt), when VMs with a uniq disk from 20 to
> 250G migrate without error in some seconds and with no downtime.

The host itself may be fine, but libvirt/QEMU running there may expose
problems, perhaps just for some VMs. According to your logs something
is not behaving as expected on the source host during the faulty
migration.

> How ca I enable this libvirt debug mode ?

Set the following options in /etc/libvirt/libvirtd.conf (look for
examples in comments there)

- log_level=1
- log_outputs="1:file:/var/log/libvirt/libvirtd.log"

and restart libvirt. Then /var/log/libvirt/libvirtd.log should contain
the log. It will be huge, so I suggest to enable it only for the time
of reproducing the problem.

> --
>
> Cordialement,
>
> Frank Soyer
>
>  
>
> Le Vendredi, Février 23, 2018 09:56 CET, Milan Zamazal  
> a écrit:
>  Maor Lipchuk  writes:
>
>> I encountered a bug (see [1]) which contains the same error mentioned in
>> your VDSM logs (see [2]), but I doubt it is related.
>
> Indeed, it's not related.
>
> The error in vdsm_victor.log just means that the info gathering call
> tries to access libvirt domain before the incoming migration is
> completed. It's ugly but harmless.
>
>> Milan, maybe you have any advice to troubleshoot the issue? Will the
>> libvirt/qemu logs can help?
>
> It seems there is something wrong on (at least) the source host. There
> are no migration progress messages in the vdsm_ginger.log and there are
> warnings about stale stat samples. That looks like problems with
> calling libvirt – slow and/or stuck calls, maybe due to storage
> problems. The possibly faulty second disk could cause that.
>
> libvirt debug logs could tell us whether that is indeed the problem and
> whether it is caused by storage or something else.
>
>> I would suggest to open a bug on that issue so we can track it more
>> properly.
>>
>> Regards,
>> Maor
>>
>>
>> [1]
>> https://bugzilla.redhat.com/show_bug.cgi?id=1486543 - Migration leads to
>> VM running on 2 Hosts
>>
>> [2]
>> 2018-02-16 09:43:35,236+0100 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer]
>> Internal server error (__init__:577)
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 572,
>> in _handle_request
>> res = method(**params)
>> File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
>> _dynamicMethod
>> result = fn(*methodArgs)
>> File "/usr/share/vdsm/API.py", line 1454, in getAllVmIoTunePolicies
>> io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()
>> File "/usr/share/vdsm/clientIF.py", line 454, in getAllVmIoTunePolicies
>> 'current_values': v.getIoTune()}
>> File "/usr/share/vdsm/virt/vm.py", line 2859, in getIoTune
>> result = self.getIoTuneResponse()
>> File "/usr/share/vdsm/virt/vm.py", line 2878, in getIoTuneResponse
>> res = self._dom.blockIoTune(
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47,
>> in __getattr__
>> % self.vmid)
>> NotConnectedError: VM u'755cf168-de65-42ed-b22f-efe9136f7594' was not
>> started yet or was shut down
>>
>> On Thu, Feb 22, 2018 at 4:22 PM, fsoyer  wrote:
>>
>>> Hi,
>>> Yes, on 2018-02-16 (vdsm logs) I tried with a VM standing on ginger
>>> (192.168.0.6) migrated (or failed to migrate...) to victor (192.168.0.5),
>>> while the engine.log in the first mail on 2018-02-12 was for VMs standing
>>> on victor, migrated (or failed to migrate...) to ginger. Symptoms were
>>> exactly the same, in both directions, and VMs works like a charm before,
>>> and even after (migration "killed" by a poweroff of VMs).
>>> Am I the only one experimenting this problem ?
>>>
>>>
>>> Thanks
>>> --
>>>
>>> Cordialement,
>>>
>>> *Frank Soyer *
>>>
>>>
>>>
>>> Le Jeudi, Février 22, 2018 00:45 CET, Maor Lipchuk 
>>> a écrit:
>>>
>>>
>>> Hi Frank,
>>>
>>> Sorry about the delay repond.
>>> I've been going through the logs you attached, although I could not find
>>> any specific indication why the migration failed because of the disk you
>>> were 

Re: [ovirt-users] VMs with multiple vdisks don't migrate

[Milan Zamazal]

"fsoyer"  writes:

> I don't beleive that this is relatd to a host, tests have been done from 
> victor
> source to ginger dest and ginger to victor. I don't see problems on storage
> (gluster 3.12 native managed by ovirt), when VMs with a uniq disk from 20 to
> 250G migrate without error in some seconds and with no downtime.

The host itself may be fine, but libvirt/QEMU running there may expose
problems, perhaps just for some VMs.  According to your logs something
is not behaving as expected on the source host during the faulty
migration.

> How ca I enable this libvirt debug mode ?

Set the following options in /etc/libvirt/libvirtd.conf (look for
examples in comments there)

- log_level=1
- log_outputs="1:file:/var/log/libvirt/libvirtd.log"

and restart libvirt.  Then /var/log/libvirt/libvirtd.log should contain
the log.  It will be huge, so I suggest to enable it only for the time
of reproducing the problem.

> --
>
> Cordialement,
>
> Frank Soyer 
>
>  
>
> Le Vendredi, Février 23, 2018 09:56 CET, Milan Zamazal  
> a écrit:
>  Maor Lipchuk  writes:
>
>> I encountered a bug (see [1]) which contains the same error mentioned in
>> your VDSM logs (see [2]), but I doubt it is related.
>
> Indeed, it's not related.
>
> The error in vdsm_victor.log just means that the info gathering call
> tries to access libvirt domain before the incoming migration is
> completed. It's ugly but harmless.
>
>> Milan, maybe you have any advice to troubleshoot the issue? Will the
>> libvirt/qemu logs can help?
>
> It seems there is something wrong on (at least) the source host. There
> are no migration progress messages in the vdsm_ginger.log and there are
> warnings about stale stat samples. That looks like problems with
> calling libvirt – slow and/or stuck calls, maybe due to storage
> problems. The possibly faulty second disk could cause that.
>
> libvirt debug logs could tell us whether that is indeed the problem and
> whether it is caused by storage or something else.
>
>> I would suggest to open a bug on that issue so we can track it more
>> properly.
>>
>> Regards,
>> Maor
>>
>>
>> [1]
>> https://bugzilla.redhat.com/show_bug.cgi?id=1486543 - Migration leads to
>> VM running on 2 Hosts
>>
>> [2]
>> 2018-02-16 09:43:35,236+0100 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer]
>> Internal server error (__init__:577)
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 572,
>> in _handle_request
>> res = method(**params)
>> File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
>> _dynamicMethod
>> result = fn(*methodArgs)
>> File "/usr/share/vdsm/API.py", line 1454, in getAllVmIoTunePolicies
>> io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()
>> File "/usr/share/vdsm/clientIF.py", line 454, in getAllVmIoTunePolicies
>> 'current_values': v.getIoTune()}
>> File "/usr/share/vdsm/virt/vm.py", line 2859, in getIoTune
>> result = self.getIoTuneResponse()
>> File "/usr/share/vdsm/virt/vm.py", line 2878, in getIoTuneResponse
>> res = self._dom.blockIoTune(
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47,
>> in __getattr__
>> % self.vmid)
>> NotConnectedError: VM u'755cf168-de65-42ed-b22f-efe9136f7594' was not
>> started yet or was shut down
>>
>> On Thu, Feb 22, 2018 at 4:22 PM, fsoyer  wrote:
>>
>>> Hi,
>>> Yes, on 2018-02-16 (vdsm logs) I tried with a VM standing on ginger
>>> (192.168.0.6) migrated (or failed to migrate...) to victor (192.168.0.5),
>>> while the engine.log in the first mail on 2018-02-12 was for VMs standing
>>> on victor, migrated (or failed to migrate...) to ginger. Symptoms were
>>> exactly the same, in both directions, and VMs works like a charm before,
>>> and even after (migration "killed" by a poweroff of VMs).
>>> Am I the only one experimenting this problem ?
>>>
>>>
>>> Thanks
>>> --
>>>
>>> Cordialement,
>>>
>>> *Frank Soyer *
>>>
>>>
>>>
>>> Le Jeudi, Février 22, 2018 00:45 CET, Maor Lipchuk 
>>> a écrit:
>>>
>>>
>>> Hi Frank,
>>>
>>> Sorry about the delay repond.
>>> I've been going through the logs you attached, although I could not find
>>> any specific indication why the migration failed because of the disk you
>>> were mentionning.
>>> Does this VM run with both disks on the target host without migration?
>>>
>>> Regards,
>>> Maor
>>>
>>>
>>> On Fri, Feb 16, 2018 at 11:03 AM, fsoyer  wrote:

 Hi Maor,
 sorry for the double post, I've change the email adress of my account and
 supposed that I'd need to re-post it.
 And thank you for your time. Here are the logs. I added a vdisk to an
 existing VM : it no more migrates, needing to poweroff it after minutes.
 Then simply deleting the second disk makes migrate it in exactly 9s without
 problem !
 https://gist.github.com/fgth/4707446331d201eef574ac31b6e89561
 

Re: [ovirt-users] VMs with multiple vdisks don't migrate

[fsoyer]

Hi,
I don't beleive that this is relatd to a host, tests have been done from victor 
source to ginger dest and ginger to victor. I don't see problems on storage 
(gluster 3.12 native managed by ovirt), when VMs with a uniq disk from 20 to 
250G migrate without error in some seconds and with no downtime.
How ca I enable this libvirt debug mode ?

--

Cordialement,

Frank Soyer

 

Le Vendredi, Février 23, 2018 09:56 CET, Milan Zamazal  a 
écrit:
 Maor Lipchuk  writes:

> I encountered a bug (see [1]) which contains the same error mentioned in
> your VDSM logs (see [2]), but I doubt it is related.

Indeed, it's not related.

The error in vdsm_victor.log just means that the info gathering call
tries to access libvirt domain before the incoming migration is
completed. It's ugly but harmless.

> Milan, maybe you have any advice to troubleshoot the issue? Will the
> libvirt/qemu logs can help?

It seems there is something wrong on (at least) the source host. There
are no migration progress messages in the vdsm_ginger.log and there are
warnings about stale stat samples. That looks like problems with
calling libvirt – slow and/or stuck calls, maybe due to storage
problems. The possibly faulty second disk could cause that.

libvirt debug logs could tell us whether that is indeed the problem and
whether it is caused by storage or something else.

> I would suggest to open a bug on that issue so we can track it more
> properly.
>
> Regards,
> Maor
>
>
> [1]
> https://bugzilla.redhat.com/show_bug.cgi?id=1486543 - Migration leads to
> VM running on 2 Hosts
>
> [2]
> 2018-02-16 09:43:35,236+0100 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer]
> Internal server error (__init__:577)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 572,
> in _handle_request
> res = method(**params)
> File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
> _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/API.py", line 1454, in getAllVmIoTunePolicies
> io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()
> File "/usr/share/vdsm/clientIF.py", line 454, in getAllVmIoTunePolicies
> 'current_values': v.getIoTune()}
> File "/usr/share/vdsm/virt/vm.py", line 2859, in getIoTune
> result = self.getIoTuneResponse()
> File "/usr/share/vdsm/virt/vm.py", line 2878, in getIoTuneResponse
> res = self._dom.blockIoTune(
> File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47,
> in __getattr__
> % self.vmid)
> NotConnectedError: VM u'755cf168-de65-42ed-b22f-efe9136f7594' was not
> started yet or was shut down
>
> On Thu, Feb 22, 2018 at 4:22 PM, fsoyer  wrote:
>
>> Hi,
>> Yes, on 2018-02-16 (vdsm logs) I tried with a VM standing on ginger
>> (192.168.0.6) migrated (or failed to migrate...) to victor (192.168.0.5),
>> while the engine.log in the first mail on 2018-02-12 was for VMs standing
>> on victor, migrated (or failed to migrate...) to ginger. Symptoms were
>> exactly the same, in both directions, and VMs works like a charm before,
>> and even after (migration "killed" by a poweroff of VMs).
>> Am I the only one experimenting this problem ?
>>
>>
>> Thanks
>> --
>>
>> Cordialement,
>>
>> *Frank Soyer *
>>
>>
>>
>> Le Jeudi, Février 22, 2018 00:45 CET, Maor Lipchuk 
>> a écrit:
>>
>>
>> Hi Frank,
>>
>> Sorry about the delay repond.
>> I've been going through the logs you attached, although I could not find
>> any specific indication why the migration failed because of the disk you
>> were mentionning.
>> Does this VM run with both disks on the target host without migration?
>>
>> Regards,
>> Maor
>>
>>
>> On Fri, Feb 16, 2018 at 11:03 AM, fsoyer  wrote:
>>>
>>> Hi Maor,
>>> sorry for the double post, I've change the email adress of my account and
>>> supposed that I'd need to re-post it.
>>> And thank you for your time. Here are the logs. I added a vdisk to an
>>> existing VM : it no more migrates, needing to poweroff it after minutes.
>>> Then simply deleting the second disk makes migrate it in exactly 9s without
>>> problem !
>>> https://gist.github.com/fgth/4707446331d201eef574ac31b6e89561
>>> https://gist.github.com/fgth/f8de9c22664aee53722af676bff8719d
>>>
>>> --
>>>
>>> Cordialement,
>>>
>>> *Frank Soyer *
>>> Le Mercredi, Février 14, 2018 11:04 CET, Maor Lipchuk <
>>> mlipc...@redhat.com> a écrit:
>>>
>>>
>>> Hi Frank,
>>>
>>> I already replied on your last email.
>>> Can you provide the VDSM logs from the time of the migration failure for
>>> both hosts:
>>> ginger.local.systea.f r and v
>>> ictor.local.systea.fr
>>>
>>> Thanks,
>>> Maor
>>>
>>> On Wed, Feb 14, 2018 at 11:23 AM, fsoyer  wrote:

 Hi all,
 I discovered yesterday a problem when migrating VM with more than one
 vdisk.
 On our test servers (oVirt4.1, shared storage with Gluster), I created 2
 VMs needed 

Re: [ovirt-users] VMs with multiple vdisks don't migrate

[Milan Zamazal]

Maor Lipchuk  writes:

> I encountered a bug (see [1]) which contains the same error mentioned in
> your VDSM logs (see [2]), but I doubt it is related.

Indeed, it's not related.

The error in vdsm_victor.log just means that the info gathering call
tries to access libvirt domain before the incoming migration is
completed.  It's ugly but harmless.

> Milan, maybe you have any advice to troubleshoot the issue? Will the
> libvirt/qemu logs can help?

It seems there is something wrong on (at least) the source host.  There
are no migration progress messages in the vdsm_ginger.log and there are
warnings about stale stat samples.  That looks like problems with
calling libvirt – slow and/or stuck calls, maybe due to storage
problems.  The possibly faulty second disk could cause that.

libvirt debug logs could tell us whether that is indeed the problem and
whether it is caused by storage or something else.

> I would suggest to open a bug on that issue so we can track it more
> properly.
>
> Regards,
> Maor
>
>
> [1]
> https://bugzilla.redhat.com/show_bug.cgi?id=1486543 -  Migration leads to
> VM running on 2 Hosts
>
> [2]
> 2018-02-16 09:43:35,236+0100 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer]
> Internal server error (__init__:577)
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 572,
> in _handle_request
> res = method(**params)
>   File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
> _dynamicMethod
> result = fn(*methodArgs)
>   File "/usr/share/vdsm/API.py", line 1454, in getAllVmIoTunePolicies
> io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()
>   File "/usr/share/vdsm/clientIF.py", line 454, in getAllVmIoTunePolicies
> 'current_values': v.getIoTune()}
>   File "/usr/share/vdsm/virt/vm.py", line 2859, in getIoTune
> result = self.getIoTuneResponse()
>   File "/usr/share/vdsm/virt/vm.py", line 2878, in getIoTuneResponse
> res = self._dom.blockIoTune(
>   File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47,
> in __getattr__
> % self.vmid)
> NotConnectedError: VM u'755cf168-de65-42ed-b22f-efe9136f7594' was not
> started yet or was shut down
>
> On Thu, Feb 22, 2018 at 4:22 PM, fsoyer  wrote:
>
>> Hi,
>> Yes, on 2018-02-16 (vdsm logs) I tried with a VM standing on ginger
>> (192.168.0.6) migrated (or failed to migrate...) to victor (192.168.0.5),
>> while the engine.log in the first mail on 2018-02-12 was for VMs standing
>> on victor, migrated (or failed to migrate...) to ginger. Symptoms were
>> exactly the same, in both directions, and VMs works like a charm before,
>> and even after (migration "killed" by a poweroff of VMs).
>> Am I the only one experimenting this problem ?
>>
>>
>> Thanks
>> --
>>
>> Cordialement,
>>
>> *Frank Soyer *
>>
>>
>>
>> Le Jeudi, Février 22, 2018 00:45 CET, Maor Lipchuk 
>> a écrit:
>>
>>
>> Hi Frank,
>>
>> Sorry about the delay repond.
>> I've been going through the logs you attached, although I could not find
>> any specific indication why the migration failed because of the disk you
>> were mentionning.
>> Does this VM run with both disks on the target host without migration?
>>
>> Regards,
>> Maor
>>
>>
>> On Fri, Feb 16, 2018 at 11:03 AM, fsoyer  wrote:
>>>
>>> Hi Maor,
>>> sorry for the double post, I've change the email adress of my account and
>>> supposed that I'd need to re-post it.
>>> And thank you for your time. Here are the logs. I added a vdisk to an
>>> existing VM : it no more migrates, needing to poweroff it after minutes.
>>> Then simply deleting the second disk makes migrate it in exactly 9s without
>>> problem !
>>> https://gist.github.com/fgth/4707446331d201eef574ac31b6e89561
>>> https://gist.github.com/fgth/f8de9c22664aee53722af676bff8719d
>>>
>>> --
>>>
>>> Cordialement,
>>>
>>> *Frank Soyer *
>>> Le Mercredi, Février 14, 2018 11:04 CET, Maor Lipchuk <
>>> mlipc...@redhat.com> a écrit:
>>>
>>>
>>> Hi Frank,
>>>
>>> I already replied on your last email.
>>> Can you provide the VDSM logs from the time of the migration failure for
>>> both hosts:
>>>   ginger.local.systea.f r and v
>>> ictor.local.systea.fr
>>>
>>> Thanks,
>>> Maor
>>>
>>> On Wed, Feb 14, 2018 at 11:23 AM, fsoyer  wrote:

 Hi all,
 I discovered yesterday a problem when migrating VM with more than one
 vdisk.
 On our test servers (oVirt4.1, shared storage with Gluster), I created 2
 VMs needed for a test, from a template with a 20G vdisk. On this VMs I
 added a 100G vdisk (for this tests I didn't want to waste time to extend
 the existing vdisks... But I lost time finally...). The VMs with the 2
 vdisks works well.
 Now I saw some updates waiting on the host. I tried to put it in
 maintenance... But it stopped on the two VM. They were marked "migrating",
 but no more accessible. Other 

Re: [ovirt-users] VMs with multiple vdisks don't migrate

[Maor Lipchuk]

I encountered a bug (see [1]) which contains the same error mentioned in
your VDSM logs (see [2]), but I doubt it is related.
Milan, maybe you have any advice to troubleshoot the issue? Will the
libvirt/qemu logs can help?
I would suggest to open a bug on that issue so we can track it more
properly.

Regards,
Maor


[1]
https://bugzilla.redhat.com/show_bug.cgi?id=1486543 -  Migration leads to
VM running on 2 Hosts

[2]
2018-02-16 09:43:35,236+0100 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer]
Internal server error (__init__:577)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 572,
in _handle_request
res = method(**params)
  File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
_dynamicMethod
result = fn(*methodArgs)
  File "/usr/share/vdsm/API.py", line 1454, in getAllVmIoTunePolicies
io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()
  File "/usr/share/vdsm/clientIF.py", line 454, in getAllVmIoTunePolicies
'current_values': v.getIoTune()}
  File "/usr/share/vdsm/virt/vm.py", line 2859, in getIoTune
result = self.getIoTuneResponse()
  File "/usr/share/vdsm/virt/vm.py", line 2878, in getIoTuneResponse
res = self._dom.blockIoTune(
  File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47,
in __getattr__
% self.vmid)
NotConnectedError: VM u'755cf168-de65-42ed-b22f-efe9136f7594' was not
started yet or was shut down

On Thu, Feb 22, 2018 at 4:22 PM, fsoyer  wrote:

> Hi,
> Yes, on 2018-02-16 (vdsm logs) I tried with a VM standing on ginger
> (192.168.0.6) migrated (or failed to migrate...) to victor (192.168.0.5),
> while the engine.log in the first mail on 2018-02-12 was for VMs standing
> on victor, migrated (or failed to migrate...) to ginger. Symptoms were
> exactly the same, in both directions, and VMs works like a charm before,
> and even after (migration "killed" by a poweroff of VMs).
> Am I the only one experimenting this problem ?
>
>
> Thanks
> --
>
> Cordialement,
>
> *Frank Soyer *
>
>
>
> Le Jeudi, Février 22, 2018 00:45 CET, Maor Lipchuk 
> a écrit:
>
>
> Hi Frank,
>
> Sorry about the delay repond.
> I've been going through the logs you attached, although I could not find
> any specific indication why the migration failed because of the disk you
> were mentionning.
> Does this VM run with both disks on the target host without migration?
>
> Regards,
> Maor
>
>
> On Fri, Feb 16, 2018 at 11:03 AM, fsoyer  wrote:
>>
>> Hi Maor,
>> sorry for the double post, I've change the email adress of my account and
>> supposed that I'd need to re-post it.
>> And thank you for your time. Here are the logs. I added a vdisk to an
>> existing VM : it no more migrates, needing to poweroff it after minutes.
>> Then simply deleting the second disk makes migrate it in exactly 9s without
>> problem !
>> https://gist.github.com/fgth/4707446331d201eef574ac31b6e89561
>> https://gist.github.com/fgth/f8de9c22664aee53722af676bff8719d
>>
>> --
>>
>> Cordialement,
>>
>> *Frank Soyer *
>> Le Mercredi, Février 14, 2018 11:04 CET, Maor Lipchuk <
>> mlipc...@redhat.com> a écrit:
>>
>>
>> Hi Frank,
>>
>> I already replied on your last email.
>> Can you provide the VDSM logs from the time of the migration failure for
>> both hosts:
>>   ginger.local.systea.f r and v
>> ictor.local.systea.fr
>>
>> Thanks,
>> Maor
>>
>> On Wed, Feb 14, 2018 at 11:23 AM, fsoyer  wrote:
>>>
>>> Hi all,
>>> I discovered yesterday a problem when migrating VM with more than one
>>> vdisk.
>>> On our test servers (oVirt4.1, shared storage with Gluster), I created 2
>>> VMs needed for a test, from a template with a 20G vdisk. On this VMs I
>>> added a 100G vdisk (for this tests I didn't want to waste time to extend
>>> the existing vdisks... But I lost time finally...). The VMs with the 2
>>> vdisks works well.
>>> Now I saw some updates waiting on the host. I tried to put it in
>>> maintenance... But it stopped on the two VM. They were marked "migrating",
>>> but no more accessible. Other (small) VMs with only 1 vdisk was migrated
>>> without problem at the same time.
>>> I saw that a kvm process for the (big) VMs was launched on the source
>>> AND destination host, but after tens of minutes, the migration and the VMs
>>> was always freezed. I tried to cancel the migration for the VMs : failed.
>>> The only way to stop it was to poweroff the VMs : the kvm process died on
>>> the 2 hosts and the GUI alerted on a failed migration.
>>> In doubt, I tried to delete the second vdisk on one of this VMs : it
>>> migrates then without error ! And no access problem.
>>> I tried to extend the first vdisk of the second VM, the delete the
>>> second vdisk : it migrates now without problem !
>>>
>>> So after another test with a VM with 2 vdisks, I can say that this
>>> blocked the migration process :(
>>>
>>> In engine.log, for a VMs with 1 

Re: [ovirt-users] VMs with multiple vdisks don't migrate

[Maor Lipchuk]

Hi Frank,

Sorry about the delay repond.
I've been going through the logs you attached, although I could not find
any specific indication why the migration failed because of the disk you
were mentionning.
Does this VM run with both disks on the target host without migration?

Regards,
Maor


On Fri, Feb 16, 2018 at 11:03 AM, fsoyer  wrote:

> Hi Maor,
> sorry for the double post, I've change the email adress of my account and
> supposed that I'd need to re-post it.
> And thank you for your time. Here are the logs. I added a vdisk to an
> existing VM : it no more migrates, needing to poweroff it after minutes.
> Then simply deleting the second disk makes migrate it in exactly 9s without
> problem !
> https://gist.github.com/fgth/4707446331d201eef574ac31b6e89561
> https://gist.github.com/fgth/f8de9c22664aee53722af676bff8719d
>
> --
>
> Cordialement,
>
> *Frank Soyer *
> Le Mercredi, Février 14, 2018 11:04 CET, Maor Lipchuk 
> a écrit:
>
>
> Hi Frank,
>
> I already replied on your last email.
> Can you provide the VDSM logs from the time of the migration failure for
> both hosts:
>   ginger.local.systea.f r and v
> ictor.local.systea.fr
>
> Thanks,
> Maor
>
> On Wed, Feb 14, 2018 at 11:23 AM, fsoyer  wrote:
>>
>> Hi all,
>> I discovered yesterday a problem when migrating VM with more than one
>> vdisk.
>> On our test servers (oVirt4.1, shared storage with Gluster), I created 2
>> VMs needed for a test, from a template with a 20G vdisk. On this VMs I
>> added a 100G vdisk (for this tests I didn't want to waste time to extend
>> the existing vdisks... But I lost time finally...). The VMs with the 2
>> vdisks works well.
>> Now I saw some updates waiting on the host. I tried to put it in
>> maintenance... But it stopped on the two VM. They were marked "migrating",
>> but no more accessible. Other (small) VMs with only 1 vdisk was migrated
>> without problem at the same time.
>> I saw that a kvm process for the (big) VMs was launched on the source AND
>> destination host, but after tens of minutes, the migration and the VMs was
>> always freezed. I tried to cancel the migration for the VMs : failed. The
>> only way to stop it was to poweroff the VMs : the kvm process died on the 2
>> hosts and the GUI alerted on a failed migration.
>> In doubt, I tried to delete the second vdisk on one of this VMs : it
>> migrates then without error ! And no access problem.
>> I tried to extend the first vdisk of the second VM, the delete the second
>> vdisk : it migrates now without problem !
>>
>> So after another test with a VM with 2 vdisks, I can say that this
>> blocked the migration process :(
>>
>> In engine.log, for a VMs with 1 vdisk migrating well, we see :
>>
>> 2018-02-12 16:46:29,705+01 INFO  
>> [org.ovirt.engine.core.bll.MigrateVmToServerCommand]
>> (default task-28) [2f712024-5982-46a8-82c8-fd8293da5725] Lock Acquired
>> to object 
>> 'EngineLock:{exclusiveLocks='[3f57e669-5e4c-4d10-85cc-d573004a099d=VM]',
>> sharedLocks=''}'
>> 2018-02-12 16:46:29,955+01 INFO  
>> [org.ovirt.engine.core.bll.MigrateVmToServerCommand]
>> (org.ovirt.thread.pool-6-thread-32) [2f712024-5982-46a8-82c8-fd8293da5725]
>> Running command: MigrateVmToServerCommand internal: false. Entities
>> affected :  ID: 3f57e669-5e4c-4d10-85cc-d573004a099d Type: VMAction
>> group MIGRATE_VM with role type USER
>> 2018-02-12 16:46:30,261+01 INFO  
>> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
>> (org.ovirt.thread.pool-6-thread-32) [2f712024-5982-46a8-82c8-fd8293da5725]
>> START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true',
>> hostId='ce3938b1-b23f-4d22-840a-f17d7cd87bb1',
>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d', srcHost='192.168.0.6',
>> dstVdsId='d569c2dd-8f30-4878-8aea-858db285cf69', dstHost='
>> 192.168.0.5:54321', migrationMethod='ONLINE', tunnelMigration='false',
>> migrationDowntime='0', autoConverge='true', migrateCompressed='false',
>> consoleAddress='null', maxBandwidth='500', enableGuestEvents='true',
>> maxIncomingMigrations='2', maxOutgoingMigrations='2',
>> convergenceSchedule='[init=[{name=setDowntime, params=[100]}],
>> stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2,
>> action={name=setDowntime, params=[200]}}, {limit=3,
>> action={name=setDowntime, params=[300]}}, {limit=4,
>> action={name=setDowntime, params=[400]}}, {limit=6,
>> action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort,
>> params=[]}}]]'}), log id: 14f61ee0
>> 2018-02-12 16:46:30,262+01 INFO  [org.ovirt.engine.core.vdsbro
>> ker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-32)
>> [2f712024-5982-46a8-82c8-fd8293da5725] START,
>> MigrateBrokerVDSCommand(HostName = victor.local.systea.fr,
>> MigrateVDSCommandParameters:{runAsync='true',
>> hostId='ce3938b1-b23f-4d22-840a-f17d7cd87bb1',
>> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d', srcHost='192.168.0.6',
>> 

Re: [ovirt-users] VMs with multiple vdisks don't migrate

[Maor Lipchuk]

Hi Frank,

I already replied on your last email.
Can you provide the VDSM logs from the time of the migration failure for
both hosts:
  ginger.local.systea.f r and v
ictor.local.systea.fr

Thanks,
Maor

On Wed, Feb 14, 2018 at 11:23 AM, fsoyer  wrote:

> Hi all,
> I discovered yesterday a problem when migrating VM with more than one
> vdisk.
> On our test servers (oVirt4.1, shared storage with Gluster), I created 2
> VMs needed for a test, from a template with a 20G vdisk. On this VMs I
> added a 100G vdisk (for this tests I didn't want to waste time to extend
> the existing vdisks... But I lost time finally...). The VMs with the 2
> vdisks works well.
> Now I saw some updates waiting on the host. I tried to put it in
> maintenance... But it stopped on the two VM. They were marked "migrating",
> but no more accessible. Other (small) VMs with only 1 vdisk was migrated
> without problem at the same time.
> I saw that a kvm process for the (big) VMs was launched on the source AND
> destination host, but after tens of minutes, the migration and the VMs was
> always freezed. I tried to cancel the migration for the VMs : failed. The
> only way to stop it was to poweroff the VMs : the kvm process died on the 2
> hosts and the GUI alerted on a failed migration.
> In doubt, I tried to delete the second vdisk on one of this VMs : it
> migrates then without error ! And no access problem.
> I tried to extend the first vdisk of the second VM, the delete the second
> vdisk : it migrates now without problem !
>
> So after another test with a VM with 2 vdisks, I can say that this blocked
> the migration process :(
>
> In engine.log, for a VMs with 1 vdisk migrating well, we see :
>
> 2018-02-12 16:46:29,705+01 INFO  
> [org.ovirt.engine.core.bll.MigrateVmToServerCommand]
> (default task-28) [2f712024-5982-46a8-82c8-fd8293da5725] Lock Acquired to
> object 
> 'EngineLock:{exclusiveLocks='[3f57e669-5e4c-4d10-85cc-d573004a099d=VM]',
> sharedLocks=''}'
> 2018-02-12 16:46:29,955+01 INFO  
> [org.ovirt.engine.core.bll.MigrateVmToServerCommand]
> (org.ovirt.thread.pool-6-thread-32) [2f712024-5982-46a8-82c8-fd8293da5725]
> Running command: MigrateVmToServerCommand internal: false. Entities
> affected :  ID: 3f57e669-5e4c-4d10-85cc-d573004a099d Type: VMAction group
> MIGRATE_VM with role type USER
> 2018-02-12 16:46:30,261+01 INFO  
> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
> (org.ovirt.thread.pool-6-thread-32) [2f712024-5982-46a8-82c8-fd8293da5725]
> START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true',
> hostId='ce3938b1-b23f-4d22-840a-f17d7cd87bb1',
> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d', srcHost='192.168.0.6',
> dstVdsId='d569c2dd-8f30-4878-8aea-858db285cf69', dstHost='
> 192.168.0.5:54321', migrationMethod='ONLINE', tunnelMigration='false',
> migrationDowntime='0', autoConverge='true', migrateCompressed='false',
> consoleAddress='null', maxBandwidth='500', enableGuestEvents='true',
> maxIncomingMigrations='2', maxOutgoingMigrations='2',
> convergenceSchedule='[init=[{name=setDowntime, params=[100]}],
> stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2,
> action={name=setDowntime, params=[200]}}, {limit=3,
> action={name=setDowntime, params=[300]}}, {limit=4,
> action={name=setDowntime, params=[400]}}, {limit=6,
> action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort,
> params=[]}}]]'}), log id: 14f61ee0
> 2018-02-12 16:46:30,262+01 INFO  [org.ovirt.engine.core.
> vdsbroker.vdsbroker.MigrateBrokerVDSCommand] 
> (org.ovirt.thread.pool-6-thread-32)
> [2f712024-5982-46a8-82c8-fd8293da5725] START, MigrateBrokerVDSCommand(HostName
> = victor.local.systea.fr, MigrateVDSCommandParameters:{runAsync='true',
> hostId='ce3938b1-b23f-4d22-840a-f17d7cd87bb1',
> vmId='3f57e669-5e4c-4d10-85cc-d573004a099d', srcHost='192.168.0.6',
> dstVdsId='d569c2dd-8f30-4878-8aea-858db285cf69', dstHost='
> 192.168.0.5:54321', migrationMethod='ONLINE', tunnelMigration='false',
> migrationDowntime='0', autoConverge='true', migrateCompressed='false',
> consoleAddress='null', maxBandwidth='500', enableGuestEvents='true',
> maxIncomingMigrations='2', maxOutgoingMigrations='2',
> convergenceSchedule='[init=[{name=setDowntime, params=[100]}],
> stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2,
> action={name=setDowntime, params=[200]}}, {limit=3,
> action={name=setDowntime, params=[300]}}, {limit=4,
> action={name=setDowntime, params=[400]}}, {limit=6,
> action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort,
> params=[]}}]]'}), log id: 775cd381
> 2018-02-12 16:46:30,277+01 INFO  [org.ovirt.engine.core.
> vdsbroker.vdsbroker.MigrateBrokerVDSCommand] 
> (org.ovirt.thread.pool-6-thread-32)
> [2f712024-5982-46a8-82c8-fd8293da5725] FINISH, MigrateBrokerVDSCommand,
> log id: 775cd381
> 2018-02-12 16:46:30,285+01 INFO  
> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
> 

[ovirt-users] VMs with multiple vdisks don't migrate

[fsoyer]

Hi all,
I discovered yesterday a problem when migrating VM with more than one vdisk.
On our test servers (oVirt4.1, shared storage with Gluster), I created 2 VMs 
needed for a test, from a template with a 20G vdisk. On this VMs I added a 100G 
vdisk (for this tests I didn't want to waste time to extend the existing 
vdisks... But I lost time finally...). The VMs with the 2 vdisks works well.
Now I saw some updates waiting on the host. I tried to put it in maintenance... 
But it stopped on the two VM. They were marked "migrating", but no more 
accessible. Other (small) VMs with only 1 vdisk was migrated without problem at 
the same time.
I saw that a kvm process for the (big) VMs was launched on the source AND 
destination host, but after tens of minutes, the migration and the VMs was 
always freezed. I tried to cancel the migration for the VMs : failed. The only 
way to stop it was to poweroff the VMs : the kvm process died on the 2 hosts 
and the GUI alerted on a failed migration.
In doubt, I tried to delete the second vdisk on one of this VMs : it migrates 
then without error ! And no access problem.
I tried to extend the first vdisk of the second VM, the delete the second vdisk 
: it migrates now without problem !   

So after another test with a VM with 2 vdisks, I can say that this blocked the 
migration process :(

In engine.log, for a VMs with 1 vdisk migrating well, we see :2018-02-12 
16:46:29,705+01 INFO  [org.ovirt.engine.core.bll.MigrateVmToServerCommand] 
(default task-28) [2f712024-5982-46a8-82c8-fd8293da5725] Lock Acquired to 
object 'EngineLock:{exclusiveLocks='[3f57e669-5e4c-4d10-85cc-d573004a099d=VM]', 
sharedLocks=''}'
2018-02-12 16:46:29,955+01 INFO  
[org.ovirt.engine.core.bll.MigrateVmToServerCommand] 
(org.ovirt.thread.pool-6-thread-32) [2f712024-5982-46a8-82c8-fd8293da5725] 
Running command: MigrateVmToServerCommand internal: false. Entities affected :  
ID: 3f57e669-5e4c-4d10-85cc-d573004a099d Type: VMAction group MIGRATE_VM with 
role type USER
2018-02-12 16:46:30,261+01 INFO  
[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] 
(org.ovirt.thread.pool-6-thread-32) [2f712024-5982-46a8-82c8-fd8293da5725] 
START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', 
hostId='ce3938b1-b23f-4d22-840a-f17d7cd87bb1', 
vmId='3f57e669-5e4c-4d10-85cc-d573004a099d', srcHost='192.168.0.6', 
dstVdsId='d569c2dd-8f30-4878-8aea-858db285cf69', dstHost='192.168.0.5:54321', 
migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', 
autoConverge='true', migrateCompressed='false', consoleAddress='null', 
maxBandwidth='500', enableGuestEvents='true', maxIncomingMigrations='2', 
maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, 
params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, 
{limit=2, action={name=setDowntime, params=[200]}}, {limit=3, 
action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, 
params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, 
action={name=abort, params=[]}}]]'}), log id: 14f61ee0
2018-02-12 16:46:30,262+01 INFO  
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] 
(org.ovirt.thread.pool-6-thread-32) [2f712024-5982-46a8-82c8-fd8293da5725] 
START, MigrateBrokerVDSCommand(HostName = victor.local.systea.fr, 
MigrateVDSCommandParameters:{runAsync='true', 
hostId='ce3938b1-b23f-4d22-840a-f17d7cd87bb1', 
vmId='3f57e669-5e4c-4d10-85cc-d573004a099d', srcHost='192.168.0.6', 
dstVdsId='d569c2dd-8f30-4878-8aea-858db285cf69', dstHost='192.168.0.5:54321', 
migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', 
autoConverge='true', migrateCompressed='false', consoleAddress='null', 
maxBandwidth='500', enableGuestEvents='true', maxIncomingMigrations='2', 
maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, 
params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, 
{limit=2, action={name=setDowntime, params=[200]}}, {limit=3, 
action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, 
params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, 
action={name=abort, params=[]}}]]'}), log id: 775cd381
2018-02-12 16:46:30,277+01 INFO  
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] 
(org.ovirt.thread.pool-6-thread-32) [2f712024-5982-46a8-82c8-fd8293da5725] 
FINISH, MigrateBrokerVDSCommand, log id: 775cd381
2018-02-12 16:46:30,285+01 INFO  
[org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] 
(org.ovirt.thread.pool-6-thread-32) [2f712024-5982-46a8-82c8-fd8293da5725] 
FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 14f61ee0
2018-02-12 16:46:30,301+01 INFO  
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] 
(org.ovirt.thread.pool-6-thread-32) [2f712024-5982-46a8-82c8-fd8293da5725] 
EVENT_ID: VM_MIGRATION_START(62), Correlation ID: 
2f712024-5982-46a8-82c8-fd8293da5725, Job ID: