Re: [ovirt-users] massive simultaneous vms migrations ?

[Martin Sivak]

> I did find this parameter (only) in /etc/vdsm/vdsm.conf.rpmnew (??)

.rpmnew files are telling you that RPM could not update the configuration
file automatically during package upgrade, because there was a manual
change in it. You can use a tool like rpmconf or just copy it over
vdsm.conf if you want the defaults.

See for example:

https://ask.fedoraproject.org/en/question/25722/what-are-rpmnew-files/
https://www.redhat.com/archives/rhl-list/2003-December/msg04713.html

Best regards

--
Martin Sivak
SLA / oVirt


On Wed, Apr 19, 2017 at 11:14 AM, Nelson Lameiras <
nelson.lamei...@lyra-network.com> wrote:

> hello pavel,
>
> Thanks for you answer.
> I did find this parameter (only) in /etc/vdsm/vdsm.conf.rpmnew (??)
>
> Parameter is commented with value 2 so my guess is that it is not used...
> So this brings a few more questions :
>
> - Since parameter is commented, default value must be used... can we be
> sure that 2 is default value?
> I do found strange that migrations are limited to 2, I have the feeling
> that more than two are simultaneously being migrated (but I'm maybe wrong),
> how to be sure?
>
> cordialement, regards,
>
> <https://www.lyra-network.com/>
> Nelson LAMEIRAS
> Ingénieur Systèmes et Réseaux / Systems and Networks engineer
> Tel: +33 5 32 09 09 70 <+33%205%2032%2009%2009%2070>
> nelson.lamei...@lyra-network.com
> www.lyra-network.com | www.payzen.eu <https://payzen.eu>
> <https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA>
> <https://www.linkedin.com/company/lyra-network_2>
> <https://twitter.com/LyraNetwork>
> <https://payzen.eu>
> --
> Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE
>
>
> --
> *From: *"Pavel Gashev" <p...@acronis.com>
> *To: *users@ovirt.org, "nelson lameiras" <nelson.lamei...@lyra-network.com
> >
> *Sent: *Tuesday, April 18, 2017 7:16:08 PM
> *Subject: *Re: [ovirt-users] massive simultaneous vms migrations ?
>
> VDSM has the following config option:
>
> # Maximum concurrent outgoing migrations
> # max_outgoing_migrations = 2
>
> On Tue, 2017-04-18 at 18:48 +0200, Nelson Lameiras wrote:
>
> hello,
>
> When putting a host on "maintenance mode", all vms start migrating to
> other hosts.
>
> We have some hosts that have 60 vms. So this will create a 60 vms
> migrating simultaneously.
> Some vms are under so much heavy loads that migration fails often (our
> guess is that massive simultaneous migrations does not help migration
> convergence) - even with "suspend workload if needed" migraton policy.
>
> - Does oVirt really launches 60 simultaneous migrations or is there a
> queuing system ?
> - If there is a queuing system, is there a way to configure a maximum
> number of simultaneous migrations ?
>
> I did see a "migration bandwidth limit", but this is quite what we are
> looking for.
>
> my setup:
> ovirt-engine +hosted engine 4.1.1
> hosts : centos 7.3 fully updated.
>
> for full context to understand this question : 2 times in the past, when
> trying to put a host in maintenance, host stopped responding during massive
> migrations and was fenced by engine. It's still unclear why host stopped
> responding, but we think that migrating 60+ vms simultaneously puts a heavy
> strain on storage ? So we would like to better control migration process in
> order to better understand what's happening. This scenario is "production
> only" since our labs do not contain nearly as much vm with such heavy
> loads. So rather than trying to reproduce, we are trying to avoid ;)
>
> cordialement, regards,
>
> <https://www.lyra-network.com/>
> Nelson LAMEIRAS
> Ingénieur Systèmes et Réseaux / Systems and Networks engineer
> Tel: +33 5 32 09 09 70 <+33%205%2032%2009%2009%2070>
> nelson.lamei...@lyra-network.com
> www.lyra-network.com | www.payzen.eu <https://payzen.eu>
> <https://www.youtube.com/channel/UCrVl1CO_Jlu3KbiRH-tQ_vA>
> <https://www.linkedin.com/company/lyra-network_2>
> <https://twitter.com/LyraNetwork>
> <https://payzen.eu>
> --
> Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE
>
> ___
> Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] massive simultaneous vms migrations ?

[Konstantin Shalygin]

We have many issues migration with 1G and 18-25 vms.

Is slow, stuck, failed. Switched to 10G and set migration limit to 
5000Mbps (actually this is don't work, but if don't set this field, 
limit is 1000Mbps!) - 25vms migrate ~ 30seconds in total.



On 04/19/2017 07:41 PM, Nelson Lameiras wrote:

1000 Mbps full duplex


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] massive simultaneous vms migrations ?

[Nelson Lameiras]

1000 Mbps full duplex 

cordialement, regards, 


Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 
nelson.lamei...@lyra-network.com 

www.lyra-network.com | www.payzen.eu 





Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE 



From: "Konstantin Shalygin" <k0...@k0ste.ru> 
To: "Nelson Lameiras" <nelson.lamei...@lyra-network.com> 
Cc: users@ovirt.org 
Sent: Wednesday, April 19, 2017 11:25:27 AM 
Subject: Re: [ovirt-users] massive simultaneous vms migrations ? 



I mean what is your hardware? 1G? 40G? 

On 04/19/2017 04:16 PM, Nelson Lameiras wrote: 



I'm using ovirtmgmt network for migrations.

I'm guetting the vibe that using a dedicated network for migration is "good 
practice"...

cordialement, regards, 


Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 nelson.lamei...@lyra-network.com www.lyra-network.com | 
www.payzen.eu Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE

- Original Message -
From: "Konstantin Shalygin" <k0...@k0ste.ru> To: users@ovirt.org , "Nelson 
Lameiras" <nelson.lamei...@lyra-network.com> Sent: Wednesday, April 19, 2017 
3:15:29 AM
Subject: Re: Re: [ovirt-users] massive simultaneous vms migrations ?

Hello.

What is your Migration Network? 

BQ_BEGIN

We have some hosts that have 60 vms. So this will create a 60 vms migrating 
simultaneously.
Some vms are under so much heavy loads that migration fails often (our guess is 
that massive simultaneous migrations does not help migration convergence) - 
even with "suspend workload if needed" migraton policy. 



BQ_END


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] massive simultaneous vms migrations ?

[Konstantin Shalygin]

I mean what is your hardware? 1G? 40G?


On 04/19/2017 04:16 PM, Nelson Lameiras wrote:

I'm using ovirtmgmt network for migrations.

I'm guetting the vibe that using a dedicated network for migration is "good 
practice"...

cordialement, regards,


Nelson LAMEIRAS
Ingénieur Systèmes et Réseaux / Systems and Networks engineer
Tel: +33 5 32 09 09 70
nelson.lamei...@lyra-network.com

www.lyra-network.com | www.payzen.eu





Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE

- Original Message -
From: "Konstantin Shalygin" <k0...@k0ste.ru>
To: users@ovirt.org, "Nelson Lameiras" <nelson.lamei...@lyra-network.com>
Sent: Wednesday, April 19, 2017 3:15:29 AM
Subject: Re: Re: [ovirt-users] massive simultaneous vms migrations ?

Hello.

What is your Migration Network?



We have some hosts that have 60 vms. So this will create a 60 vms migrating 
simultaneously.
Some vms are under so much heavy loads that migration fails often (our guess is that 
massive simultaneous migrations does not help migration convergence) - even with 
"suspend workload if needed" migraton policy.


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] massive simultaneous vms migrations ?

[Nelson Lameiras]

Hi yaniv, 

Thanks for your response. My answers below. 

cordialement, regards, 


Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 
nelson.lamei...@lyra-network.com 

www.lyra-network.com | www.payzen.eu 





Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE 



From: "Yaniv Kaul" <yk...@redhat.com> 
To: "Nelson Lameiras" <nelson.lamei...@lyra-network.com> 
Cc: "ovirt users" <users@ovirt.org> 
Sent: Wednesday, April 19, 2017 9:13:27 AM 
Subject: Re: [ovirt-users] massive simultaneous vms migrations ? 



On Tue, Apr 18, 2017 at 7:48 PM, Nelson Lameiras < 
nelson.lamei...@lyra-network.com > wrote: 



hello, 

When putting a host on "maintenance mode", all vms start migrating to other 
hosts. 

We have some hosts that have 60 vms. So this will create a 60 vms migrating 
simultaneously. 
Some vms are under so much heavy loads that migration fails often (our guess is 
that massive simultaneous migrations does not help migration convergence) - 
even with "suspend workload if needed" migraton policy. 

- Does oVirt really launches 60 simultaneous migrations or is there a queuing 
system ? 
- If there is a queuing system, is there a way to configure a maximum number of 
simultaneous migrations ? 

I did see a "migration bandwidth limit", but this is quite what we are looking 
for. 



What migration policy are you using? 
-- "suspend workkiad if needed" 
Are you using a dedicated migration network, or the ovirtmgmt network? 
-- ovirtmgmt network 



BQ_BEGIN


my setup: 
ovirt-engine +hosted engine 4.1.1 
hosts : centos 7.3 fully updated. 

for full context to understand this question : 2 times in the past, when trying 
to put a host in maintenance, host stopped responding during massive migrations 
and was fenced by engine. It's still unclear why host stopped responding, but 
we think that migrating 60+ vms simultaneously puts a heavy strain on storage ? 
So we would like to better control migration process in order to better 
understand what's happening. This scenario is "production only" since our labs 
do not contain nearly as much vm with such heavy loads. So rather than trying 
to reproduce, we are trying to avoid ;) 

BQ_END

If you could open a bug with relevant logs on the host not responding, that'd 
be great. 
-- Too late, we made the "precipitaded" mistake of reinstalling server (beause 
of other reasons), so all traces are lost. Next time I will make sure to keep 
traces. 
Live migration doesn't touch the storage (disks re not moving anywhere), but it 
does stress the network. I doubt it, but perhaps you over-saturate the 
ovirtmgmt network. 
-- This makes sens. It's a long shot though. Maybe it would be possible to 
create a VLAN dedicated to migrations in same physical network et use QOS to 
reserve always some bandwith to oVirt administration ? (maybe opening can of 
worms here) 
Y. 

BQ_BEGIN


cordialement, regards, 


Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 

nelson.lamei...@lyra-network.com 

www.lyra-network.com | www.payzen.eu 





Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE 


___ 
Users mailing list 
Users@ovirt.org 
http://lists.ovirt.org/mailman/listinfo/users 


BQ_END


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] massive simultaneous vms migrations ?

[Nelson Lameiras]

I'm using ovirtmgmt network for migrations.

I'm guetting the vibe that using a dedicated network for migration is "good 
practice"...

cordialement, regards, 


Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 
nelson.lamei...@lyra-network.com 

www.lyra-network.com | www.payzen.eu 





Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE

- Original Message -
From: "Konstantin Shalygin" <k0...@k0ste.ru>
To: users@ovirt.org, "Nelson Lameiras" <nelson.lamei...@lyra-network.com>
Sent: Wednesday, April 19, 2017 3:15:29 AM
Subject: Re: Re: [ovirt-users] massive simultaneous vms migrations ?

Hello.

What is your Migration Network?


> We have some hosts that have 60 vms. So this will create a 60 vms migrating 
> simultaneously.
> Some vms are under so much heavy loads that migration fails often (our guess 
> is that massive simultaneous migrations does not help migration convergence) 
> - even with "suspend workload if needed" migraton policy.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] massive simultaneous vms migrations ?

[Nelson Lameiras]

hello pavel, 

Thanks for you answer. 
I did find this parameter (only) in /etc/vdsm/vdsm.conf.rpmnew (??) 

Parameter is commented with value 2 so my guess is that it is not used... So 
this brings a few more questions : 

- Since parameter is commented, default value must be used... can we be sure 
that 2 is default value? 
I do found strange that migrations are limited to 2, I have the feeling that 
more than two are simultaneously being migrated (but I'm maybe wrong), how to 
be sure? 

cordialement, regards, 


Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 
nelson.lamei...@lyra-network.com 

www.lyra-network.com | www.payzen.eu 





Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE 



From: "Pavel Gashev" <p...@acronis.com> 
To: users@ovirt.org, "nelson lameiras" <nelson.lamei...@lyra-network.com> 
Sent: Tuesday, April 18, 2017 7:16:08 PM 
Subject: Re: [ovirt-users] massive simultaneous vms migrations ? 

VDSM has the following config option: 

# Maximum concurrent outgoing migrations 
# max_outgoing_migrations = 2 

On Tue, 2017-04-18 at 18:48 +0200, Nelson Lameiras wrote: 



hello, 

When putting a host on "maintenance mode", all vms start migrating to other 
hosts. 

We have some hosts that have 60 vms. So this will create a 60 vms migrating 
simultaneously. 
Some vms are under so much heavy loads that migration fails often (our guess is 
that massive simultaneous migrations does not help migration convergence) - 
even with "suspend workload if needed" migraton policy. 

- Does oVirt really launches 60 simultaneous migrations or is there a queuing 
system ? 
- If there is a queuing system, is there a way to configure a maximum number of 
simultaneous migrations ? 

I did see a "migration bandwidth limit", but this is quite what we are looking 
for. 

my setup: 
ovirt-engine +hosted engine 4.1.1 
hosts : centos 7.3 fully updated. 

for full context to understand this question : 2 times in the past, when trying 
to put a host in maintenance, host stopped responding during massive migrations 
and was fenced by engine. It's still unclear why host stopped responding, but 
we think that migrating 60+ vms simultaneously puts a heavy strain on storage ? 
So we would like to better control migration process in order to better 
understand what's happening. This scenario is "production only" since our labs 
do not contain nearly as much vm with such heavy loads. So rather than trying 
to reproduce, we are trying to avoid ;) 

cordialement, regards, 




Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 
nelson.lamei...@lyra-network.com 

www.lyra-network.com | www.payzen.eu 









Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE 

___
Users mailing list Users@ovirt.org 
http://lists.ovirt.org/mailman/listinfo/users 



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] massive simultaneous vms migrations ?

[Yaniv Kaul]

On Tue, Apr 18, 2017 at 7:48 PM, Nelson Lameiras <
nelson.lamei...@lyra-network.com> wrote:

> hello,
>
> When putting a host on "maintenance mode", all vms start migrating to
> other hosts.
>
> We have some hosts that have 60 vms. So this will create a 60 vms
> migrating simultaneously.
> Some vms are under so much heavy loads that migration fails often (our
> guess is that massive simultaneous migrations does not help migration
> convergence) - even with "suspend workload if needed" migraton policy.
>
> - Does oVirt really launches 60 simultaneous migrations or is there a
> queuing system ?
> - If there is a queuing system, is there a way to configure a maximum
> number of simultaneous migrations ?
>
> I did see a "migration bandwidth limit", but this is quite what we are
> looking for.
>

What migration policy are you using?
Are you using a dedicated migration network, or the ovirtmgmt network?


>
> my setup:
> ovirt-engine +hosted engine 4.1.1
> hosts : centos 7.3 fully updated.
>
> for full context to understand this question : 2 times in the past, when
> trying to put a host in maintenance, host stopped responding during massive
> migrations and was fenced by engine. It's still unclear why host stopped
> responding, but we think that migrating 60+ vms simultaneously puts a heavy
> strain on storage ? So we would like to better control migration process in
> order to better understand what's happening. This scenario is "production
> only" since our labs do not contain nearly as much vm with such heavy
> loads. So rather than trying to reproduce, we are trying to avoid ;)
>

If you could open a bug with relevant logs on the host not responding,
that'd be great.
Live migration doesn't touch the storage (disks re not moving anywhere),
but it does stress the network. I doubt it, but perhaps you over-saturate
the ovirtmgmt network.
Y.


>
> cordialement, regards,
>
> 
> Nelson LAMEIRAS
> Ingénieur Systèmes et Réseaux / Systems and Networks engineer
> Tel: +33 5 32 09 09 70 <+33%205%2032%2009%2009%2070>
> nelson.lamei...@lyra-network.com
> www.lyra-network.com | www.payzen.eu 
> 
> 
> 
> 
> --
> Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] massive simultaneous vms migrations ?

[Konstantin Shalygin]

Hello.

What is your Migration Network?



We have some hosts that have 60 vms. So this will create a 60 vms migrating 
simultaneously.
Some vms are under so much heavy loads that migration fails often (our guess is that 
massive simultaneous migrations does not help migration convergence) - even with 
"suspend workload if needed" migraton policy.


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] massive simultaneous vms migrations ?

[Pavel Gashev]

VDSM has the following config option:

# Maximum concurrent outgoing migrations
# max_outgoing_migrations = 2

On Tue, 2017-04-18 at 18:48 +0200, Nelson Lameiras wrote:
hello,

When putting a host on "maintenance mode", all vms start migrating to other 
hosts.

We have some hosts that have 60 vms. So this will create a 60 vms migrating 
simultaneously.
Some vms are under so much heavy loads that migration fails often (our guess is 
that massive simultaneous migrations does not help migration convergence) - 
even with "suspend workload if needed" migraton policy.

- Does oVirt really launches 60 simultaneous migrations or is there a queuing 
system ?
- If there is a queuing system, is there a way to configure a maximum number of 
simultaneous migrations ?

I did see a "migration bandwidth limit", but this is quite what we are looking 
for.

my setup:
ovirt-engine +hosted engine 4.1.1
hosts : centos 7.3 fully updated.

for full context to understand this question : 2 times in the past, when trying 
to put a host in maintenance, host stopped responding during massive migrations 
and was fenced by engine. It's still unclear why host stopped responding, but 
we think that migrating 60+ vms simultaneously puts a heavy strain on storage ? 
So we would like to better control migration process in order to better 
understand what's happening. This scenario is "production only" since our labs 
do not contain nearly as much vm with such heavy loads. So rather than trying 
to reproduce, we are trying to avoid ;)

cordialement, regards,

[cid:1492535709.17056.4.camel@acronis.com]

Nelson LAMEIRAS
Ingénieur Systèmes et Réseaux / Systems and Networks engineer
Tel: +33 5 32 09 09 70
nelson.lamei...@lyra-network.com

www.lyra-network.com | 
www.payzen.eu


[cid:1492535709.17056.3.camel@acronis.com]
[cid:1492535709.17056.2.camel@acronis.com] 

[cid:1492535709.17056.1.camel@acronis.com] 

[cid:1492535709.17056.0.camel@acronis.com] 



Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] massive simultaneous vms migrations ?

[Nelson Lameiras]

hello, 

When putting a host on "maintenance mode", all vms start migrating to other 
hosts. 

We have some hosts that have 60 vms. So this will create a 60 vms migrating 
simultaneously. 
Some vms are under so much heavy loads that migration fails often (our guess is 
that massive simultaneous migrations does not help migration convergence) - 
even with "suspend workload if needed" migraton policy. 

- Does oVirt really launches 60 simultaneous migrations or is there a queuing 
system ? 
- If there is a queuing system, is there a way to configure a maximum number of 
simultaneous migrations ? 

I did see a "migration bandwidth limit", but this is quite what we are looking 
for. 

my setup: 
ovirt-engine +hosted engine 4.1.1 
hosts : centos 7.3 fully updated. 

for full context to understand this question : 2 times in the past, when trying 
to put a host in maintenance, host stopped responding during massive migrations 
and was fenced by engine. It's still unclear why host stopped responding, but 
we think that migrating 60+ vms simultaneously puts a heavy strain on storage ? 
So we would like to better control migration process in order to better 
understand what's happening. This scenario is "production only" since our labs 
do not contain nearly as much vm with such heavy loads. So rather than trying 
to reproduce, we are trying to avoid ;) 

cordialement, regards, 


Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 
nelson.lamei...@lyra-network.com 

www.lyra-network.com | www.payzen.eu 





Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE 

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users