Re: [ovirt-users] live migration with openvswitch

[Milan Zamazal]

Michal Skrivanek  writes:

>> > I'm afraid that we are not yet ready to backport it to 4.0 - we found
>> > out that as it is, it break migration for vmfex and external network
>> > providers; it also breaks when a buggy Engine db does not send a
>> > displayNetwork. But we plan to fix these issues quite soon.
>
> which “buggy” engine? There were changes in parameters, most of these issues
> are not relevant anymore since we ditched <3.6 though.
> Again it’s ok as long as it is clearly mentioned like "3.6 engine sends it in
> such and such parameter, we can drop it once we support 4.0+"

I think Edward means the problem when there is no display (and
migration) network set for a cluster in Engine.  This may happen due to
a former bug in Engine db scripts.  Vdsm apparently falls back on
ovirtmgmt in most cases so the problem is typically unnoticed.  But when
you look for displayNetwork explicitly in Vdsm, it's not there.

The bug may affect 4.0 installations until a db upgrade fix is created
and backported.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] live migration with openvswitch

[Michal Skrivanek]

> On 16 Sep 2016, at 14:36, Michal Skrivanek  
> wrote:
> 
> 
>> On 15 Sep 2016, at 21:46, Edward Haas > > wrote:
>> 
>> 
>> 
>> On Thu, Sep 15, 2016 at 1:30 PM, Michal Skrivanek 
>> > wrote:
>> 
>> > On 15 Sep 2016, at 10:11, Dan Kenigsberg > > > wrote:
>> >
>> > On Wed, Sep 14, 2016 at 03:04:14PM +0200, Michal Skrivanek wrote:
>> >>
>> >>> On 09 Sep 2016, at 13:09, Edward Haas > >>> > wrote:
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Sep 8, 2016 at 11:27 AM, Pavel Levshin > >>>  > >>> >> wrote:
>> >>> Hi.
>> >>>
>> >>> I'm trying to learn Ovirt 4 and have a problem with it.
>> >>>
>> >>> My cluster consists of 3 nodes. I use Openvswitch for network 
>> >>> connectivity. I have a HostedEngine and one additional VM in the cluster.
>> >>>
>> >>> When I try to migrate the VM to another node, it fails. From vdsm and 
>> >>> libvirtd logs I see that proper network interface on destination node 
>> >>> cannot be found. Libvirt tries to find Openvswitch bridge with name like 
>> >>> "vdsmbr_AOYiPtcT". It exists on source node, but it is unique on every 
>> >>> node, because it contains random part. Additionally, it changes on every 
>> >>> reboot.
>> >>>
>> >>> How this is supposed to work?
>> >>>
>> >>> --
>> >>> Pavel Levshin
>> >>>
>> >>>
>> >>>
>> >>> Hi Pavel,
>> >>>
>> >>> VM migration is supported on the master branch, however it has not been 
>> >>> ported to 4.0 yet.
>> >>
>> >>> You can either build VDSM from source (from master branch) or try to 
>> >>> apply this patch on what you have:
>> >>> https://gerrit.ovirt.org/#/c/59645  
>> >>> >
>> >>
>> >> That’s quite a horrible solution right now. I certainly would not like to 
>> >> see it in 4.0 (given the hacks around display).
>> 
>> What is horrible exactly?
>> It's not too late to propose other solutions.
> 
> if OVS is the next great feature it should fit into the code accordingly. 
> I.e. using hooks only when it’s absolutely necessary and as a temporary 
> measure only until the respective proper RFEs are implemented and available. 
> E.g. when there is a libvirt support missing we can add a qemu command line 
> parameter ourselves bypassing libvirt but we always should have a clear plan 
> (i.e. a bug) to move away from there as soon as the support is 
> there(requested back then when we went with the hack)
> 
> Such things should be reviewed as soon as we get to a similar area, so while 
> modifying libvirt-hook.sh we can see the original reason for the hook is not 
> valid anymore as everything is addressed and the hacky code should have been 
> removed
> It was easy to see that because there is a clear comment about dependent bugs 
> and issues (though missed by all the reviewers, unfortunately!)
> Your new code doesn’t have anything like that and I have no idea what kind of 
> API or behavior we actually need, whether appropriate requests has been filed 
> on e.g. libvirt. That makes it very hard to revisit in the future by the next 
> random person.
> 
>> 
>> Display uses libvirt to resolve a network name to an IP address for it to 
>> bound to. But that works only for linux bridges.
>> That is limiting, especially now that we do not have a Linux bridge, but 
>> something else.
> 
> that’s ok, whatever needs to be done. But then please make sure you’re not 
> breaking existing features, at least again not without a plan(==bug) to fix 
> it.
> 
>> 
>> >> Do we have a bug/plan to improve it?
>> >
>> > We have Bug 1362495 - [OVS] - Add support for live migration
>> > to track that.

oh, and yes, that’s exactly the tracking I wanted to make sure exists. There’s 
just no link in the gerrit commit itself so I didn’t find it (but I wasn’t 
really looking hard either;-)

Thanks,
michal

>> >
>> > I'm afraid that we are not yet ready to backport it to 4.0 - we found
>> > out that as it is, it break migration for vmfex and external network
>> > providers; it also breaks when a buggy Engine db does not send a
>> > displayNetwork. But we plan to fix these issues quite soon.
> 
> which “buggy” engine? There were changes in parameters, most of these issues 
> are not relevant anymore since we ditched <3.6 though.
> Again it’s ok as long as it is clearly mentioned like "3.6 engine sends it in 
> such and such parameter, we can drop it once we support 4.0+"
> 
>> >
>> > The hacks arround display are an actual imporovement. For "legacy"
>> > switchType, we maintain an on-host libvirt-side database of all networks
>> > only to keep libvirt happy. Having a database copy has all the known
>> > troubles of mismatches and being out of sync. 

Re: [ovirt-users] live migration with openvswitch

[Michal Skrivanek]

> On 15 Sep 2016, at 21:46, Edward Haas  wrote:
> 
> 
> 
> On Thu, Sep 15, 2016 at 1:30 PM, Michal Skrivanek 
> > wrote:
> 
> > On 15 Sep 2016, at 10:11, Dan Kenigsberg  > > wrote:
> >
> > On Wed, Sep 14, 2016 at 03:04:14PM +0200, Michal Skrivanek wrote:
> >>
> >>> On 09 Sep 2016, at 13:09, Edward Haas  >>> > wrote:
> >>>
> >>>
> >>>
> >>> On Thu, Sep 8, 2016 at 11:27 AM, Pavel Levshin  >>>   >>> >> wrote:
> >>> Hi.
> >>>
> >>> I'm trying to learn Ovirt 4 and have a problem with it.
> >>>
> >>> My cluster consists of 3 nodes. I use Openvswitch for network 
> >>> connectivity. I have a HostedEngine and one additional VM in the cluster.
> >>>
> >>> When I try to migrate the VM to another node, it fails. From vdsm and 
> >>> libvirtd logs I see that proper network interface on destination node 
> >>> cannot be found. Libvirt tries to find Openvswitch bridge with name like 
> >>> "vdsmbr_AOYiPtcT". It exists on source node, but it is unique on every 
> >>> node, because it contains random part. Additionally, it changes on every 
> >>> reboot.
> >>>
> >>> How this is supposed to work?
> >>>
> >>> --
> >>> Pavel Levshin
> >>>
> >>>
> >>>
> >>> Hi Pavel,
> >>>
> >>> VM migration is supported on the master branch, however it has not been 
> >>> ported to 4.0 yet.
> >>
> >>> You can either build VDSM from source (from master branch) or try to 
> >>> apply this patch on what you have:
> >>> https://gerrit.ovirt.org/#/c/59645  
> >>> >
> >>
> >> That’s quite a horrible solution right now. I certainly would not like to 
> >> see it in 4.0 (given the hacks around display).
> 
> What is horrible exactly?
> It's not too late to propose other solutions.

if OVS is the next great feature it should fit into the code accordingly. I.e. 
using hooks only when it’s absolutely necessary and as a temporary measure only 
until the respective proper RFEs are implemented and available. E.g. when there 
is a libvirt support missing we can add a qemu command line parameter ourselves 
bypassing libvirt but we always should have a clear plan (i.e. a bug) to move 
away from there as soon as the support is there(requested back then when we 
went with the hack)

Such things should be reviewed as soon as we get to a similar area, so while 
modifying libvirt-hook.sh we can see the original reason for the hook is not 
valid anymore as everything is addressed and the hacky code should have been 
removed
It was easy to see that because there is a clear comment about dependent bugs 
and issues (though missed by all the reviewers, unfortunately!)
Your new code doesn’t have anything like that and I have no idea what kind of 
API or behavior we actually need, whether appropriate requests has been filed 
on e.g. libvirt. That makes it very hard to revisit in the future by the next 
random person.

> 
> Display uses libvirt to resolve a network name to an IP address for it to 
> bound to. But that works only for linux bridges.
> That is limiting, especially now that we do not have a Linux bridge, but 
> something else.

that’s ok, whatever needs to be done. But then please make sure you’re not 
breaking existing features, at least again not without a plan(==bug) to fix it.

> 
> >> Do we have a bug/plan to improve it?
> >
> > We have Bug 1362495 - [OVS] - Add support for live migration
> > to track that.
> >
> > I'm afraid that we are not yet ready to backport it to 4.0 - we found
> > out that as it is, it break migration for vmfex and external network
> > providers; it also breaks when a buggy Engine db does not send a
> > displayNetwork. But we plan to fix these issues quite soon.

which “buggy” engine? There were changes in parameters, most of these issues 
are not relevant anymore since we ditched <3.6 though.
Again it’s ok as long as it is clearly mentioned like "3.6 engine sends it in 
such and such parameter, we can drop it once we support 4.0+"

> >
> > The hacks arround display are an actual imporovement. For "legacy"
> > switchType, we maintain an on-host libvirt-side database of all networks
> > only to keep libvirt happy. Having a database copy has all the known
> > troubles of mismatches and being out of sync. For "ovs" switchType, we
> > do not (we don't use a bridge, but a port group so there's no natural
> > way to define our network in libvirt). Modifying the listening address
> > on destination is the flexible and quick way to do it - I wish we had
> > the libvirt migrate hook years ago.
> 
> doesn’t it prevent seamless virti-viewer console connection?
> 
> The end result is the same, we listen on the address of a specific network.
> Previously it contained a 

Re: [ovirt-users] live migration with openvswitch

[Edward Haas]

On Thu, Sep 15, 2016 at 1:30 PM, Michal Skrivanek <
michal.skriva...@redhat.com> wrote:

>
> > On 15 Sep 2016, at 10:11, Dan Kenigsberg  wrote:
> >
> > On Wed, Sep 14, 2016 at 03:04:14PM +0200, Michal Skrivanek wrote:
> >>
> >>> On 09 Sep 2016, at 13:09, Edward Haas  wrote:
> >>>
> >>>
> >>>
> >>> On Thu, Sep 8, 2016 at 11:27 AM, Pavel Levshin  > wrote:
> >>> Hi.
> >>>
> >>> I'm trying to learn Ovirt 4 and have a problem with it.
> >>>
> >>> My cluster consists of 3 nodes. I use Openvswitch for network
> connectivity. I have a HostedEngine and one additional VM in the cluster.
> >>>
> >>> When I try to migrate the VM to another node, it fails. From vdsm and
> libvirtd logs I see that proper network interface on destination node
> cannot be found. Libvirt tries to find Openvswitch bridge with name like
> "vdsmbr_AOYiPtcT". It exists on source node, but it is unique on every
> node, because it contains random part. Additionally, it changes on every
> reboot.
> >>>
> >>> How this is supposed to work?
> >>>
> >>> --
> >>> Pavel Levshin
> >>>
> >>>
> >>>
> >>> Hi Pavel,
> >>>
> >>> VM migration is supported on the master branch, however it has not
> been ported to 4.0 yet.
> >>
> >>> You can either build VDSM from source (from master branch) or try to
> apply this patch on what you have:
> >>> https://gerrit.ovirt.org/#/c/59645  >
> >>
> >> That’s quite a horrible solution right now. I certainly would not like
> to see it in 4.0 (given the hacks around display).
>

What is horrible exactly?
It's not too late to propose other solutions.

Display uses libvirt to resolve a network name to an IP address for it to
bound to. But that works only for linux bridges.
That is limiting, especially now that we do not have a Linux bridge, but
something else.

>> Do we have a bug/plan to improve it?
> >
> > We have Bug 1362495 - [OVS] - Add support for live migration
> > to track that.
> >
> > I'm afraid that we are not yet ready to backport it to 4.0 - we found
> > out that as it is, it break migration for vmfex and external network
> > providers; it also breaks when a buggy Engine db does not send a
> > displayNetwork. But we plan to fix these issues quite soon.
> >
> > The hacks arround display are an actual imporovement. For "legacy"
> > switchType, we maintain an on-host libvirt-side database of all networks
> > only to keep libvirt happy. Having a database copy has all the known
> > troubles of mismatches and being out of sync. For "ovs" switchType, we
> > do not (we don't use a bridge, but a port group so there's no natural
> > way to define our network in libvirt). Modifying the listening address
> > on destination is the flexible and quick way to do it - I wish we had
> > the libvirt migrate hook years ago.
>
> doesn’t it prevent seamless virti-viewer console connection?
>

The end result is the same, we listen on the address of a specific network.
Previously it contained a network name and libvirt converted it to the
correct IP it should bind to, now vdsm resolves it.

also the “TODO” in the code about multiple graphics is worrying (we fully
> support it and are considering to make it a default)
>

Supported where? virt networking code in VDSM which creates an interface
for domxml does not support it at the moment.
Or am I missing something?

If we have an idea of what API would work well? we should raise or
> contribute that to libvirt. Surely it takes time but it is the only way how
> to improve the code eventually.
>

If using libvirt can allow us to drop some persisted data and logic from
vdsm, then it makes sense, but I do not think this is the case.
As it stands today, depending on libvirt persisted data is limiting us, at
least in the networking area. I also do not see the advantage of using it
as a DB.

Thaks,
Edy.


>
> Thanks,
> michal
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] live migration with openvswitch

[Michal Skrivanek]

> On 15 Sep 2016, at 10:11, Dan Kenigsberg  wrote:
> 
> On Wed, Sep 14, 2016 at 03:04:14PM +0200, Michal Skrivanek wrote:
>> 
>>> On 09 Sep 2016, at 13:09, Edward Haas  wrote:
>>> 
>>> 
>>> 
>>> On Thu, Sep 8, 2016 at 11:27 AM, Pavel Levshin >> > wrote:
>>> Hi.
>>> 
>>> I'm trying to learn Ovirt 4 and have a problem with it.
>>> 
>>> My cluster consists of 3 nodes. I use Openvswitch for network connectivity. 
>>> I have a HostedEngine and one additional VM in the cluster.
>>> 
>>> When I try to migrate the VM to another node, it fails. From vdsm and 
>>> libvirtd logs I see that proper network interface on destination node 
>>> cannot be found. Libvirt tries to find Openvswitch bridge with name like 
>>> "vdsmbr_AOYiPtcT". It exists on source node, but it is unique on every 
>>> node, because it contains random part. Additionally, it changes on every 
>>> reboot.
>>> 
>>> How this is supposed to work?
>>> 
>>> --
>>> Pavel Levshin
>>> 
>>> 
>>> 
>>> Hi Pavel,
>>> 
>>> VM migration is supported on the master branch, however it has not been 
>>> ported to 4.0 yet.
>> 
>>> You can either build VDSM from source (from master branch) or try to apply 
>>> this patch on what you have:
>>> https://gerrit.ovirt.org/#/c/59645 
>> 
>> That’s quite a horrible solution right now. I certainly would not like to 
>> see it in 4.0 (given the hacks around display). 
>> Do we have a bug/plan to improve it?
> 
> We have Bug 1362495 - [OVS] - Add support for live migration
> to track that.
> 
> I'm afraid that we are not yet ready to backport it to 4.0 - we found
> out that as it is, it break migration for vmfex and external network
> providers; it also breaks when a buggy Engine db does not send a
> displayNetwork. But we plan to fix these issues quite soon.
> 
> The hacks arround display are an actual imporovement. For "legacy"
> switchType, we maintain an on-host libvirt-side database of all networks
> only to keep libvirt happy. Having a database copy has all the known
> troubles of mismatches and being out of sync. For "ovs" switchType, we
> do not (we don't use a bridge, but a port group so there's no natural
> way to define our network in libvirt). Modifying the listening address
> on destination is the flexible and quick way to do it - I wish we had
> the libvirt migrate hook years ago.

doesn’t it prevent seamless virti-viewer console connection?
also the “TODO” in the code about multiple graphics is worrying (we fully 
support it and are considering to make it a default)
If we have an idea of what API would work well? we should raise or contribute 
that to libvirt. Surely it takes time but it is the only way how to improve the 
code eventually.

Thanks,
michal

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] live migration with openvswitch

[Dan Kenigsberg]

On Wed, Sep 14, 2016 at 03:04:14PM +0200, Michal Skrivanek wrote:
> 
> > On 09 Sep 2016, at 13:09, Edward Haas  wrote:
> > 
> > 
> > 
> > On Thu, Sep 8, 2016 at 11:27 AM, Pavel Levshin  > > wrote:
> > Hi.
> > 
> > I'm trying to learn Ovirt 4 and have a problem with it.
> > 
> > My cluster consists of 3 nodes. I use Openvswitch for network connectivity. 
> > I have a HostedEngine and one additional VM in the cluster.
> > 
> > When I try to migrate the VM to another node, it fails. From vdsm and 
> > libvirtd logs I see that proper network interface on destination node 
> > cannot be found. Libvirt tries to find Openvswitch bridge with name like 
> > "vdsmbr_AOYiPtcT". It exists on source node, but it is unique on every 
> > node, because it contains random part. Additionally, it changes on every 
> > reboot.
> > 
> > How this is supposed to work?
> > 
> > --
> > Pavel Levshin
> > 
> > 
> > 
> > Hi Pavel,
> > 
> > VM migration is supported on the master branch, however it has not been 
> > ported to 4.0 yet.
> 
> > You can either build VDSM from source (from master branch) or try to apply 
> > this patch on what you have:
> > https://gerrit.ovirt.org/#/c/59645 
> 
> That’s quite a horrible solution right now. I certainly would not like to see 
> it in 4.0 (given the hacks around display). 
> Do we have a bug/plan to improve it?

We have Bug 1362495 - [OVS] - Add support for live migration
to track that.

I'm afraid that we are not yet ready to backport it to 4.0 - we found
out that as it is, it break migration for vmfex and external network
providers; it also breaks when a buggy Engine db does not send a
displayNetwork. But we plan to fix these issues quite soon.

The hacks arround display are an actual imporovement. For "legacy"
switchType, we maintain an on-host libvirt-side database of all networks
only to keep libvirt happy. Having a database copy has all the known
troubles of mismatches and being out of sync. For "ovs" switchType, we
do not (we don't use a bridge, but a port group so there's no natural
way to define our network in libvirt). Modifying the listening address
on destination is the flexible and quick way to do it - I wish we had
the libvirt migrate hook years ago.


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] live migration with openvswitch

[Michal Skrivanek]

> On 09 Sep 2016, at 13:09, Edward Haas  wrote:
> 
> 
> 
> On Thu, Sep 8, 2016 at 11:27 AM, Pavel Levshin  > wrote:
> Hi.
> 
> I'm trying to learn Ovirt 4 and have a problem with it.
> 
> My cluster consists of 3 nodes. I use Openvswitch for network connectivity. I 
> have a HostedEngine and one additional VM in the cluster.
> 
> When I try to migrate the VM to another node, it fails. From vdsm and 
> libvirtd logs I see that proper network interface on destination node cannot 
> be found. Libvirt tries to find Openvswitch bridge with name like 
> "vdsmbr_AOYiPtcT". It exists on source node, but it is unique on every node, 
> because it contains random part. Additionally, it changes on every reboot.
> 
> How this is supposed to work?
> 
> --
> Pavel Levshin
> 
> 
> 
> Hi Pavel,
> 
> VM migration is supported on the master branch, however it has not been 
> ported to 4.0 yet.

> You can either build VDSM from source (from master branch) or try to apply 
> this patch on what you have:
> https://gerrit.ovirt.org/#/c/59645 

That’s quite a horrible solution right now. I certainly would not like to see 
it in 4.0 (given the hacks around display). 
Do we have a bug/plan to improve it?

Thanks,
michal

> 
> (note that you'll need to restart vdsm service for this to take affect)
> 
> Thanks,
> Edy.
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] live migration with openvswitch

[Edward Haas]

On Thu, Sep 8, 2016 at 11:27 AM, Pavel Levshin  wrote:

> Hi.
>
> I'm trying to learn Ovirt 4 and have a problem with it.
>
> My cluster consists of 3 nodes. I use Openvswitch for network
> connectivity. I have a HostedEngine and one additional VM in the cluster.
>
> When I try to migrate the VM to another node, it fails. From vdsm and
> libvirtd logs I see that proper network interface on destination node
> cannot be found. Libvirt tries to find Openvswitch bridge with name like
> "vdsmbr_AOYiPtcT". It exists on source node, but it is unique on every
> node, because it contains random part. Additionally, it changes on every
> reboot.
>
> How this is supposed to work?
>
> --
> Pavel Levshin
>
>
>
Hi Pavel,

VM migration is supported on the master branch, however it has not been
ported to 4.0 yet.
You can either build VDSM from source (from master branch) or try to apply
this patch on what you have:
https://gerrit.ovirt.org/#/c/59645

(note that you'll need to restart vdsm service for this to take affect)

Thanks,
Edy.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users