Re: help/advise needed: Private gateway vs. new physcial network issue

[Andrija Panic]

I managed to solve it, thanks Simon:

Steps, give or take:
- add new physical network via CM
- enable new physical network via CM
- manually define VLAN as isolation method in DB (add new ROW
in physical_network_isolation_methods table)
- duplicate existing Guest network row from table
physical_network_traffic_types (new uuid, and other changes as needed, to
reflect new KVM label "bond0") - so this network becomes another Guest
network...
- Edit tags on both guest physical networks (via GUI)
- Updated tags all Network Offerings inside DB (to match bond0.950 as main
vxlan guest carrier network) - maybe not needed, maybe yes...
- found "hidden" DB record in network_offering table called
"System-Private-Gateway-Network-Offering" and this is where I needed to set
tag to match the tag on second Guest network (bond0)


after this all works as expected

Very purely documented (but I'm assuming much easire when you create 2
Guest networks during adding Zone in begining)

I will see to update docs, once I'm clear on this procedure in more
details...

Thanks a lot Simon !

On 3 May 2017 at 23:34, Andrija Panic  wrote:

> Ok, thanks, I need to read a bit on tagging networks, first time I
> encounter this.
>
> agent.properties only has 1 guest interface definition (
> guest.network.device=bond0.950), so I will see how this behaves...
>
> Thanks for input Simon
>
> On 3 May 2017 at 23:19, Simon Weller  wrote:
>
>> We deploy with 2 physical interfaces. 1 is for vxlan guest networks and
>> the other is a trunk interfaces for public, mgmt and private gateways. We
>> found that tagging was necessary, or the incorrect interface can be
>> selected because both have guest networks.
>>
>>
>> 
>> From: Andrija Panic 
>> Sent: Wednesday, May 3, 2017 4:09 PM
>> To: dev@cloudstack.apache.org
>> Cc: us...@cloudstack.apache.org
>> Subject: Re: help/advise needed: Private gateway vs. new physcial network
>> issue
>>
>> Hi Simon,
>>
>> not at all. We use tags only for storage and compute(service)/disk
>> offerings...
>>
>> But,
>>
>> I just found out, even when I change recird in DB record, change KVM label
>> from bond0.950 to bond0, then disable/enable zone, and even restart mgmt
>> servers, still ACS provision vlan 999 on top of bond0.950 although I
>> selected bond0.
>>
>>
>> Her is funny thing: when I changed agent.properties
>> file guest.network.device=bond0.950 to bond0, then it worked (at least
>> proper PIF selected)...but again this can't be done on production in my
>> case
>>
>> It would be interesting to know (Cloudops and others) if you guys use same
>> physical network to carrrie guest private networks (vlans or vxlans?) AND
>> these new vlans for PRIV.GTW. We use vxlans gor guest traffic...
>>
>>
>> Thanks Simon,
>>
>> Andrija
>>
>> On 3 May 2017 at 23:01, Simon Weller  wrote:
>>
>> > Andrija,
>> >
>> >
>> > Do you have any network tagging setup for your vpc network offerings
>> that
>> > correspond to your zone network tags?
>> >
>> > 
>> > From: Andrija Panic 
>> > Sent: Wednesday, May 3, 2017 3:46 PM
>> > To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
>> > Subject: help/advise needed: Private gateway vs. new physcial network
>> issue
>> >
>> > Hi all,
>> >
>> > I'm trying to to test Private Gateway on our production (actually on DEV
>> > first :) ) setup, of ACS 4.5,
>> > but I'm hitting some strange issues during actual creation of PV GTW.
>> >
>> > My setup is the following:
>> >
>> > ACS 4.5, advanced zone KVM (ubuntu 14)
>> > mgmt network: KVM label/name: cloudbr0
>> > sec. stor.network KMV label/name: cloudbr2
>> > guest network KVM label/name: bond0.950 (we use vxlans, so this is
>> > apropriate...)
>> > public network KVM label/name: cloudbr3
>> >
>> > This above is all fine, but when adding PRIV.GTW, ACS tries to provision
>> > new vlan interface (later with bridge...) on top of selected physical
>> > interface (from the list above) - which in my case is impossible, as it
>> > seems.
>> >
>> > So I decided to add addional Physical Network (name: bond0), so I expect
>> > ACS will provision i.e. bond0.999 vlan interface for one PRIV.GTW for
>> > testing purposes (vlan 999)
>> >
>> > PROBLEM:
>> > - in running zone, I need to disable it, then I use CloudMonkey to add
>> > zone:
>> > * create physicalnetwork name=bond0 broadcastdomainrange=zone
>> > zoneid=d27f6354-a715-40c7-8322-a31091f97699 isolationmethod=vlan
>> > Afterwards I do enable the zone: update physicalnetwork state=Enabled
>> > id=3424e392-e0a1-4c21-81d9-db69acbe6c8e
>> >
>> > First command above, does NOT update DB table
>> > cloud.physical_network_isolation_methods
>> > with new record, so when you list network it dont mentions
>> > isolation_method.
>> > OK, I edit DB directly, and create new row referencing new network by
>> ID,
>> > and vlan set as isolation 

Re: help/advise needed: Private gateway vs. new physcial network issue

[Andrija Panic]

Ok, thanks, I need to read a bit on tagging networks, first time I
encounter this.

agent.properties only has 1 guest interface definition (
guest.network.device=bond0.950), so I will see how this behaves...

Thanks for input Simon

On 3 May 2017 at 23:19, Simon Weller  wrote:

> We deploy with 2 physical interfaces. 1 is for vxlan guest networks and
> the other is a trunk interfaces for public, mgmt and private gateways. We
> found that tagging was necessary, or the incorrect interface can be
> selected because both have guest networks.
>
>
> 
> From: Andrija Panic 
> Sent: Wednesday, May 3, 2017 4:09 PM
> To: dev@cloudstack.apache.org
> Cc: us...@cloudstack.apache.org
> Subject: Re: help/advise needed: Private gateway vs. new physcial network
> issue
>
> Hi Simon,
>
> not at all. We use tags only for storage and compute(service)/disk
> offerings...
>
> But,
>
> I just found out, even when I change recird in DB record, change KVM label
> from bond0.950 to bond0, then disable/enable zone, and even restart mgmt
> servers, still ACS provision vlan 999 on top of bond0.950 although I
> selected bond0.
>
>
> Her is funny thing: when I changed agent.properties
> file guest.network.device=bond0.950 to bond0, then it worked (at least
> proper PIF selected)...but again this can't be done on production in my
> case
>
> It would be interesting to know (Cloudops and others) if you guys use same
> physical network to carrrie guest private networks (vlans or vxlans?) AND
> these new vlans for PRIV.GTW. We use vxlans gor guest traffic...
>
>
> Thanks Simon,
>
> Andrija
>
> On 3 May 2017 at 23:01, Simon Weller  wrote:
>
> > Andrija,
> >
> >
> > Do you have any network tagging setup for your vpc network offerings that
> > correspond to your zone network tags?
> >
> > 
> > From: Andrija Panic 
> > Sent: Wednesday, May 3, 2017 3:46 PM
> > To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
> > Subject: help/advise needed: Private gateway vs. new physcial network
> issue
> >
> > Hi all,
> >
> > I'm trying to to test Private Gateway on our production (actually on DEV
> > first :) ) setup, of ACS 4.5,
> > but I'm hitting some strange issues during actual creation of PV GTW.
> >
> > My setup is the following:
> >
> > ACS 4.5, advanced zone KVM (ubuntu 14)
> > mgmt network: KVM label/name: cloudbr0
> > sec. stor.network KMV label/name: cloudbr2
> > guest network KVM label/name: bond0.950 (we use vxlans, so this is
> > apropriate...)
> > public network KVM label/name: cloudbr3
> >
> > This above is all fine, but when adding PRIV.GTW, ACS tries to provision
> > new vlan interface (later with bridge...) on top of selected physical
> > interface (from the list above) - which in my case is impossible, as it
> > seems.
> >
> > So I decided to add addional Physical Network (name: bond0), so I expect
> > ACS will provision i.e. bond0.999 vlan interface for one PRIV.GTW for
> > testing purposes (vlan 999)
> >
> > PROBLEM:
> > - in running zone, I need to disable it, then I use CloudMonkey to add
> > zone:
> > * create physicalnetwork name=bond0 broadcastdomainrange=zone
> > zoneid=d27f6354-a715-40c7-8322-a31091f97699 isolationmethod=vlan
> > Afterwards I do enable the zone: update physicalnetwork state=Enabled
> > id=3424e392-e0a1-4c21-81d9-db69acbe6c8e
> >
> > First command above, does NOT update DB table
> > cloud.physical_network_isolation_methods
> > with new record, so when you list network it dont mentions
> > isolation_method.
> > OK, I edit DB directly, and create new row referencing new network by ID,
> > and vlan set as isolation method.
> >
> > BTW, table cloud.physical_network_traffic_types is not populated, which
> I
> > assume is OK/good since I don't want any normal traffci
> > (mgmt/guest.public/storage) to go over this physical net - but again this
> > might be the root of problems ? Since the only guest network is on PIF
> > bond0.950
> >
> > When I try to create PRIV.GTW, ACS does some magic, and again tries to
> > provision vlan 999 interface (example vlan from above) on bond0.950
> (guest
> > network) (bond0.950.999)
> >
> > I checked the logs (attached below) and it does trie to provision GTW on
> > new physical network really.
> >
> > I'm assuming, that maybe since no values for new bond0 network inside
> table
> > cloud.physical_network_traffic_types is populated, that than ACS fails
> > back
> > to only available guest network, and that is bond0.950 - also I recall we
> > need to define KVM label so the ACS will actaully know on which interface
> > to use... (which is missing from DB for new bond0 network, as
> explained...)
> >
> > I checked the logs, and didn't see any intersting stuff really (perhaps
> I'm
> > missing something...)
> > https://pastebin.com/MZXrK31M
> [https://pastebin.com/i/facebook.png]
>
> PRIV.GTW created on 

Re: help/advise needed: Private gateway vs. new physcial network issue

[Simon Weller]

We deploy with 2 physical interfaces. 1 is for vxlan guest networks and the 
other is a trunk interfaces for public, mgmt and private gateways. We found 
that tagging was necessary, or the incorrect interface can be selected because 
both have guest networks.



From: Andrija Panic 
Sent: Wednesday, May 3, 2017 4:09 PM
To: dev@cloudstack.apache.org
Cc: us...@cloudstack.apache.org
Subject: Re: help/advise needed: Private gateway vs. new physcial network issue

Hi Simon,

not at all. We use tags only for storage and compute(service)/disk
offerings...

But,

I just found out, even when I change recird in DB record, change KVM label
from bond0.950 to bond0, then disable/enable zone, and even restart mgmt
servers, still ACS provision vlan 999 on top of bond0.950 although I
selected bond0.


Her is funny thing: when I changed agent.properties
file guest.network.device=bond0.950 to bond0, then it worked (at least
proper PIF selected)...but again this can't be done on production in my case

It would be interesting to know (Cloudops and others) if you guys use same
physical network to carrrie guest private networks (vlans or vxlans?) AND
these new vlans for PRIV.GTW. We use vxlans gor guest traffic...


Thanks Simon,

Andrija

On 3 May 2017 at 23:01, Simon Weller  wrote:

> Andrija,
>
>
> Do you have any network tagging setup for your vpc network offerings that
> correspond to your zone network tags?
>
> 
> From: Andrija Panic 
> Sent: Wednesday, May 3, 2017 3:46 PM
> To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
> Subject: help/advise needed: Private gateway vs. new physcial network issue
>
> Hi all,
>
> I'm trying to to test Private Gateway on our production (actually on DEV
> first :) ) setup, of ACS 4.5,
> but I'm hitting some strange issues during actual creation of PV GTW.
>
> My setup is the following:
>
> ACS 4.5, advanced zone KVM (ubuntu 14)
> mgmt network: KVM label/name: cloudbr0
> sec. stor.network KMV label/name: cloudbr2
> guest network KVM label/name: bond0.950 (we use vxlans, so this is
> apropriate...)
> public network KVM label/name: cloudbr3
>
> This above is all fine, but when adding PRIV.GTW, ACS tries to provision
> new vlan interface (later with bridge...) on top of selected physical
> interface (from the list above) - which in my case is impossible, as it
> seems.
>
> So I decided to add addional Physical Network (name: bond0), so I expect
> ACS will provision i.e. bond0.999 vlan interface for one PRIV.GTW for
> testing purposes (vlan 999)
>
> PROBLEM:
> - in running zone, I need to disable it, then I use CloudMonkey to add
> zone:
> * create physicalnetwork name=bond0 broadcastdomainrange=zone
> zoneid=d27f6354-a715-40c7-8322-a31091f97699 isolationmethod=vlan
> Afterwards I do enable the zone: update physicalnetwork state=Enabled
> id=3424e392-e0a1-4c21-81d9-db69acbe6c8e
>
> First command above, does NOT update DB table
> cloud.physical_network_isolation_methods
> with new record, so when you list network it dont mentions
> isolation_method.
> OK, I edit DB directly, and create new row referencing new network by ID,
> and vlan set as isolation method.
>
> BTW, table cloud.physical_network_traffic_types is not populated, which I
> assume is OK/good since I don't want any normal traffci
> (mgmt/guest.public/storage) to go over this physical net - but again this
> might be the root of problems ? Since the only guest network is on PIF
> bond0.950
>
> When I try to create PRIV.GTW, ACS does some magic, and again tries to
> provision vlan 999 interface (example vlan from above) on bond0.950 (guest
> network) (bond0.950.999)
>
> I checked the logs (attached below) and it does trie to provision GTW on
> new physical network really.
>
> I'm assuming, that maybe since no values for new bond0 network inside table
> cloud.physical_network_traffic_types is populated, that than ACS fails
> back
> to only available guest network, and that is bond0.950 - also I recall we
> need to define KVM label so the ACS will actaully know on which interface
> to use... (which is missing from DB for new bond0 network, as explained...)
>
> I checked the logs, and didn't see any intersting stuff really (perhaps I'm
> missing something...)
> https://pastebin.com/MZXrK31M
[https://pastebin.com/i/facebook.png]

PRIV.GTW created on wrong PIF - Pastebin.com
pastebin.com



> [https://pastebin.com/i/facebook.png]
>
> PRIV.GTW created on wrong PIF - Pastebin.com >
> pastebin.com
>
>
>
>
> I would really appreciate any help, since I dont know which direction to go
> now...
>
>
>
>
>
> --
>
> Andrija Panić
>



--

Andrija Panić

Re: help/advise needed: Private gateway vs. new physcial network issue

[Andrija Panic]

Hi Simon,

not at all. We use tags only for storage and compute(service)/disk
offerings...

But,

I just found out, even when I change recird in DB record, change KVM label
from bond0.950 to bond0, then disable/enable zone, and even restart mgmt
servers, still ACS provision vlan 999 on top of bond0.950 although I
selected bond0.


Her is funny thing: when I changed agent.properties
file guest.network.device=bond0.950 to bond0, then it worked (at least
proper PIF selected)...but again this can't be done on production in my case

It would be interesting to know (Cloudops and others) if you guys use same
physical network to carrrie guest private networks (vlans or vxlans?) AND
these new vlans for PRIV.GTW. We use vxlans gor guest traffic...


Thanks Simon,

Andrija

On 3 May 2017 at 23:01, Simon Weller  wrote:

> Andrija,
>
>
> Do you have any network tagging setup for your vpc network offerings that
> correspond to your zone network tags?
>
> 
> From: Andrija Panic 
> Sent: Wednesday, May 3, 2017 3:46 PM
> To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
> Subject: help/advise needed: Private gateway vs. new physcial network issue
>
> Hi all,
>
> I'm trying to to test Private Gateway on our production (actually on DEV
> first :) ) setup, of ACS 4.5,
> but I'm hitting some strange issues during actual creation of PV GTW.
>
> My setup is the following:
>
> ACS 4.5, advanced zone KVM (ubuntu 14)
> mgmt network: KVM label/name: cloudbr0
> sec. stor.network KMV label/name: cloudbr2
> guest network KVM label/name: bond0.950 (we use vxlans, so this is
> apropriate...)
> public network KVM label/name: cloudbr3
>
> This above is all fine, but when adding PRIV.GTW, ACS tries to provision
> new vlan interface (later with bridge...) on top of selected physical
> interface (from the list above) - which in my case is impossible, as it
> seems.
>
> So I decided to add addional Physical Network (name: bond0), so I expect
> ACS will provision i.e. bond0.999 vlan interface for one PRIV.GTW for
> testing purposes (vlan 999)
>
> PROBLEM:
> - in running zone, I need to disable it, then I use CloudMonkey to add
> zone:
> * create physicalnetwork name=bond0 broadcastdomainrange=zone
> zoneid=d27f6354-a715-40c7-8322-a31091f97699 isolationmethod=vlan
> Afterwards I do enable the zone: update physicalnetwork state=Enabled
> id=3424e392-e0a1-4c21-81d9-db69acbe6c8e
>
> First command above, does NOT update DB table
> cloud.physical_network_isolation_methods
> with new record, so when you list network it dont mentions
> isolation_method.
> OK, I edit DB directly, and create new row referencing new network by ID,
> and vlan set as isolation method.
>
> BTW, table cloud.physical_network_traffic_types is not populated, which I
> assume is OK/good since I don't want any normal traffci
> (mgmt/guest.public/storage) to go over this physical net - but again this
> might be the root of problems ? Since the only guest network is on PIF
> bond0.950
>
> When I try to create PRIV.GTW, ACS does some magic, and again tries to
> provision vlan 999 interface (example vlan from above) on bond0.950 (guest
> network) (bond0.950.999)
>
> I checked the logs (attached below) and it does trie to provision GTW on
> new physical network really.
>
> I'm assuming, that maybe since no values for new bond0 network inside table
> cloud.physical_network_traffic_types is populated, that than ACS fails
> back
> to only available guest network, and that is bond0.950 - also I recall we
> need to define KVM label so the ACS will actaully know on which interface
> to use... (which is missing from DB for new bond0 network, as explained...)
>
> I checked the logs, and didn't see any intersting stuff really (perhaps I'm
> missing something...)
> https://pastebin.com/MZXrK31M
> [https://pastebin.com/i/facebook.png]
>
> PRIV.GTW created on wrong PIF - Pastebin.com >
> pastebin.com
>
>
>
>
> I would really appreciate any help, since I dont know which direction to go
> now...
>
>
>
>
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Re: help/advise needed: Private gateway vs. new physcial network issue

[Simon Weller]

Andrija,


Do you have any network tagging setup for your vpc network offerings that 
correspond to your zone network tags?


From: Andrija Panic 
Sent: Wednesday, May 3, 2017 3:46 PM
To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
Subject: help/advise needed: Private gateway vs. new physcial network issue

Hi all,

I'm trying to to test Private Gateway on our production (actually on DEV
first :) ) setup, of ACS 4.5,
but I'm hitting some strange issues during actual creation of PV GTW.

My setup is the following:

ACS 4.5, advanced zone KVM (ubuntu 14)
mgmt network: KVM label/name: cloudbr0
sec. stor.network KMV label/name: cloudbr2
guest network KVM label/name: bond0.950 (we use vxlans, so this is
apropriate...)
public network KVM label/name: cloudbr3

This above is all fine, but when adding PRIV.GTW, ACS tries to provision
new vlan interface (later with bridge...) on top of selected physical
interface (from the list above) - which in my case is impossible, as it
seems.

So I decided to add addional Physical Network (name: bond0), so I expect
ACS will provision i.e. bond0.999 vlan interface for one PRIV.GTW for
testing purposes (vlan 999)

PROBLEM:
- in running zone, I need to disable it, then I use CloudMonkey to add zone:
* create physicalnetwork name=bond0 broadcastdomainrange=zone
zoneid=d27f6354-a715-40c7-8322-a31091f97699 isolationmethod=vlan
Afterwards I do enable the zone: update physicalnetwork state=Enabled
id=3424e392-e0a1-4c21-81d9-db69acbe6c8e

First command above, does NOT update DB table
cloud.physical_network_isolation_methods
with new record, so when you list network it dont mentions isolation_method.
OK, I edit DB directly, and create new row referencing new network by ID,
and vlan set as isolation method.

BTW, table cloud.physical_network_traffic_types is not populated, which I
assume is OK/good since I don't want any normal traffci
(mgmt/guest.public/storage) to go over this physical net - but again this
might be the root of problems ? Since the only guest network is on PIF
bond0.950

When I try to create PRIV.GTW, ACS does some magic, and again tries to
provision vlan 999 interface (example vlan from above) on bond0.950 (guest
network) (bond0.950.999)

I checked the logs (attached below) and it does trie to provision GTW on
new physical network really.

I'm assuming, that maybe since no values for new bond0 network inside table
cloud.physical_network_traffic_types is populated, that than ACS fails back
to only available guest network, and that is bond0.950 - also I recall we
need to define KVM label so the ACS will actaully know on which interface
to use... (which is missing from DB for new bond0 network, as explained...)

I checked the logs, and didn't see any intersting stuff really (perhaps I'm
missing something...)
https://pastebin.com/MZXrK31M
[https://pastebin.com/i/facebook.png]

PRIV.GTW created on wrong PIF - Pastebin.com
pastebin.com




I would really appreciate any help, since I dont know which direction to go
now...





--

Andrija Panić

proper place to put migrations

[Nathan Johnson]

I have created a JIRA bug here that will require the use of a migration to  
fix.

https://issues.apache.org/jira/browse/CLOUDSTACK-9902

What is the most appropriate branch to fork from to submit a PR in this  
case, and what is the most appropriate migration script to edit (or create  
a new one)?  Is there going to be a new 4.9.x release?  If so, what is the  
next one called?  4.9.2.1 or 4.9.3.0 ?  Or should this go against master  
and just get added to schema-4920to41000.sql?

Thanks in advance

Nathan Johnson
R Engineer
Education Networks of America

help/advise needed: Private gateway vs. new physcial network issue

[Andrija Panic]

Hi all,

I'm trying to to test Private Gateway on our production (actually on DEV
first :) ) setup, of ACS 4.5,
but I'm hitting some strange issues during actual creation of PV GTW.

My setup is the following:

ACS 4.5, advanced zone KVM (ubuntu 14)
mgmt network: KVM label/name: cloudbr0
sec. stor.network KMV label/name: cloudbr2
guest network KVM label/name: bond0.950 (we use vxlans, so this is
apropriate...)
public network KVM label/name: cloudbr3

This above is all fine, but when adding PRIV.GTW, ACS tries to provision
new vlan interface (later with bridge...) on top of selected physical
interface (from the list above) - which in my case is impossible, as it
seems.

So I decided to add addional Physical Network (name: bond0), so I expect
ACS will provision i.e. bond0.999 vlan interface for one PRIV.GTW for
testing purposes (vlan 999)

PROBLEM:
- in running zone, I need to disable it, then I use CloudMonkey to add zone:
* create physicalnetwork name=bond0 broadcastdomainrange=zone
zoneid=d27f6354-a715-40c7-8322-a31091f97699 isolationmethod=vlan
Afterwards I do enable the zone: update physicalnetwork state=Enabled
id=3424e392-e0a1-4c21-81d9-db69acbe6c8e

First command above, does NOT update DB table
cloud.physical_network_isolation_methods
with new record, so when you list network it dont mentions isolation_method.
OK, I edit DB directly, and create new row referencing new network by ID,
and vlan set as isolation method.

BTW, table cloud.physical_network_traffic_types is not populated, which I
assume is OK/good since I don't want any normal traffci
(mgmt/guest.public/storage) to go over this physical net - but again this
might be the root of problems ? Since the only guest network is on PIF
bond0.950

When I try to create PRIV.GTW, ACS does some magic, and again tries to
provision vlan 999 interface (example vlan from above) on bond0.950 (guest
network) (bond0.950.999)

I checked the logs (attached below) and it does trie to provision GTW on
new physical network really.

I'm assuming, that maybe since no values for new bond0 network inside table
cloud.physical_network_traffic_types is populated, that than ACS fails back
to only available guest network, and that is bond0.950 - also I recall we
need to define KVM label so the ACS will actaully know on which interface
to use... (which is missing from DB for new bond0 network, as explained...)

I checked the logs, and didn't see any intersting stuff really (perhaps I'm
missing something...)
https://pastebin.com/MZXrK31M

I would really appreciate any help, since I dont know which direction to go
now...





-- 

Andrija Panić

Re: Private gateways - experience, anyone really using it ?

[Andrija Panic]

Thanks everyone for swift comments :)

I see (in ACS 4.5 ATM, to be soon upgraded to 4.8) there are some issues
when creating (additional) physical networks (with CM), i.e. we set
isolation method to be VLAN, but DB field is not written to at all, so when
adding PV GTW, ACS fails back to add vlan interface on top of completely
another physical interface because of this problem (now fixing manually in
DB). Hope I will have PV GTW working, once I resolve this...

thx again



On 3 May 2017 at 17:39, Will Stevens  wrote:

> Interesting.  I wonder if the limitations we are seeing are actually a UI
> artifact and not actually an implementation artifact.  I will have to look
> into that.  Thanks...  :)
>
> *Will STEVENS*
> Lead Developer
>
> 
>
> On Wed, May 3, 2017 at 11:32 AM, Simon Weller  wrote:
>
> > So we manage static routes within our middleware and UI (we don't use the
> > ACS native UI), so I don't think we've experienced this.
> >
> > 
> > From: williamstev...@gmail.com  on behalf of
> > Will Stevens 
> > Sent: Wednesday, May 3, 2017 10:26 AM
> > To: dev@cloudstack.apache.org
> > Cc: us...@cloudstack.apache.org
> > Subject: Re: Private gateways - experience, anyone really using it ?
> >
> > @sweller: I would be interested in the PG fixes you have done.  Anything
> > related to overlapping routes?
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > 
> > CloudOps | Managed Private Cloud | Managed Public Cloud | Cloud Building
> |
> > Cloud Architecture | Cloud Migration
> > goo.gl
> > We provide private, public and hybrid cloud solutions for businesses
> > seeking to scale, and for enterprises making their move to the Cloud.
> >
> >
> >
> >
> > On Wed, May 3, 2017 at 11:23 AM, Simon Weller  wrote:
> >
> > > We use private gateways extensively with KVM and redundant VRs. We have
> > > our own ACS 4.8 branch (https://github.com/myENA/
> > > cloudstack/tree/release/ENA-4.8) with quite a number of backported
> fixes
> > > that relate to PGs.
> > >
> > > As far as I'm aware, vxlan is not supported on PGs today. We would like
> > to
> > > see it though. Today we use vlans and connect a customer lan to a PG
> via
> > > MPLS.
> > >
> > >
> > > 
> > > From: Andrija Panic 
> > > Sent: Wednesday, May 3, 2017 10:09 AM
> > > To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
> > > Subject: Private gateways - experience, anyone really using it ?
> > >
> > > Hi all,
> > >
> > > I'm interested to know if anyone is using private gatewyas in
> production,
> > > and what are your experience with it, any undocumented limitations,
> etc ?
> > >
> > > I dont see iif t supports vxlans (perhaps will try to test it myself)
> > >
> > > I really would appreciate any feedback.
> > >
> > > Thanks,
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
>



-- 

Andrija Panić

Re: Private gateways - experience, anyone really using it ?

[Will Stevens]

Interesting.  I wonder if the limitations we are seeing are actually a UI
artifact and not actually an implementation artifact.  I will have to look
into that.  Thanks...  :)

*Will STEVENS*
Lead Developer



On Wed, May 3, 2017 at 11:32 AM, Simon Weller  wrote:

> So we manage static routes within our middleware and UI (we don't use the
> ACS native UI), so I don't think we've experienced this.
>
> 
> From: williamstev...@gmail.com  on behalf of
> Will Stevens 
> Sent: Wednesday, May 3, 2017 10:26 AM
> To: dev@cloudstack.apache.org
> Cc: us...@cloudstack.apache.org
> Subject: Re: Private gateways - experience, anyone really using it ?
>
> @sweller: I would be interested in the PG fixes you have done.  Anything
> related to overlapping routes?
>
> *Will STEVENS*
> Lead Developer
>
> 
> CloudOps | Managed Private Cloud | Managed Public Cloud | Cloud Building |
> Cloud Architecture | Cloud Migration
> goo.gl
> We provide private, public and hybrid cloud solutions for businesses
> seeking to scale, and for enterprises making their move to the Cloud.
>
>
>
>
> On Wed, May 3, 2017 at 11:23 AM, Simon Weller  wrote:
>
> > We use private gateways extensively with KVM and redundant VRs. We have
> > our own ACS 4.8 branch (https://github.com/myENA/
> > cloudstack/tree/release/ENA-4.8) with quite a number of backported fixes
> > that relate to PGs.
> >
> > As far as I'm aware, vxlan is not supported on PGs today. We would like
> to
> > see it though. Today we use vlans and connect a customer lan to a PG via
> > MPLS.
> >
> >
> > 
> > From: Andrija Panic 
> > Sent: Wednesday, May 3, 2017 10:09 AM
> > To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
> > Subject: Private gateways - experience, anyone really using it ?
> >
> > Hi all,
> >
> > I'm interested to know if anyone is using private gatewyas in production,
> > and what are your experience with it, any undocumented limitations, etc ?
> >
> > I dont see iif t supports vxlans (perhaps will try to test it myself)
> >
> > I really would appreciate any feedback.
> >
> > Thanks,
> >
> > --
> >
> > Andrija Panić
> >
>

Re: Private gateways - experience, anyone really using it ?

[Simon Weller]

So we manage static routes within our middleware and UI (we don't use the ACS 
native UI), so I don't think we've experienced this.


From: williamstev...@gmail.com  on behalf of Will 
Stevens 
Sent: Wednesday, May 3, 2017 10:26 AM
To: dev@cloudstack.apache.org
Cc: us...@cloudstack.apache.org
Subject: Re: Private gateways - experience, anyone really using it ?

@sweller: I would be interested in the PG fixes you have done.  Anything
related to overlapping routes?

*Will STEVENS*
Lead Developer


CloudOps | Managed Private Cloud | Managed Public Cloud | Cloud Building | 
Cloud Architecture | Cloud Migration
goo.gl
We provide private, public and hybrid cloud solutions for businesses seeking to 
scale, and for enterprises making their move to the Cloud.




On Wed, May 3, 2017 at 11:23 AM, Simon Weller  wrote:

> We use private gateways extensively with KVM and redundant VRs. We have
> our own ACS 4.8 branch (https://github.com/myENA/
> cloudstack/tree/release/ENA-4.8) with quite a number of backported fixes
> that relate to PGs.
>
> As far as I'm aware, vxlan is not supported on PGs today. We would like to
> see it though. Today we use vlans and connect a customer lan to a PG via
> MPLS.
>
>
> 
> From: Andrija Panic 
> Sent: Wednesday, May 3, 2017 10:09 AM
> To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
> Subject: Private gateways - experience, anyone really using it ?
>
> Hi all,
>
> I'm interested to know if anyone is using private gatewyas in production,
> and what are your experience with it, any undocumented limitations, etc ?
>
> I dont see iif t supports vxlans (perhaps will try to test it myself)
>
> I really would appreciate any feedback.
>
> Thanks,
>
> --
>
> Andrija Panić
>

Re: Private gateways - experience, anyone really using it ?

[Will Stevens]

@sweller: I would be interested in the PG fixes you have done.  Anything
related to overlapping routes?

*Will STEVENS*
Lead Developer



On Wed, May 3, 2017 at 11:23 AM, Simon Weller  wrote:

> We use private gateways extensively with KVM and redundant VRs. We have
> our own ACS 4.8 branch (https://github.com/myENA/
> cloudstack/tree/release/ENA-4.8) with quite a number of backported fixes
> that relate to PGs.
>
> As far as I'm aware, vxlan is not supported on PGs today. We would like to
> see it though. Today we use vlans and connect a customer lan to a PG via
> MPLS.
>
>
> 
> From: Andrija Panic 
> Sent: Wednesday, May 3, 2017 10:09 AM
> To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
> Subject: Private gateways - experience, anyone really using it ?
>
> Hi all,
>
> I'm interested to know if anyone is using private gatewyas in production,
> and what are your experience with it, any undocumented limitations, etc ?
>
> I dont see iif t supports vxlans (perhaps will try to test it myself)
>
> I really would appreciate any feedback.
>
> Thanks,
>
> --
>
> Andrija Panić
>

Re: Private gateways - experience, anyone really using it ?

[Simon Weller]

We use private gateways extensively with KVM and redundant VRs. We have our own 
ACS 4.8 branch (https://github.com/myENA/cloudstack/tree/release/ENA-4.8) with 
quite a number of backported fixes that relate to PGs.

As far as I'm aware, vxlan is not supported on PGs today. We would like to see 
it though. Today we use vlans and connect a customer lan to a PG via MPLS.



From: Andrija Panic 
Sent: Wednesday, May 3, 2017 10:09 AM
To: us...@cloudstack.apache.org; dev@cloudstack.apache.org
Subject: Private gateways - experience, anyone really using it ?

Hi all,

I'm interested to know if anyone is using private gatewyas in production,
and what are your experience with it, any undocumented limitations, etc ?

I dont see iif t supports vxlans (perhaps will try to test it myself)

I really would appreciate any feedback.

Thanks,

--

Andrija Panić

Re: Private gateways - experience, anyone really using it ?

[Will Stevens]

​We have been using it quite a bit in production over the last 3+ years.
It works pretty well.  It is a pretty simple implementation, so it is
pretty stable.  I know my networking guys have some frustrations with the
limitations on the static routes.  No overlapping routes and things like
that which make things harder for them.  I have not gotten into the code to
try to see how hard that stuff is to 'fix'.

We are not using VXLAN (yet), so I can't really comment on anything on that
front.

In general, it is a very useful piece of kit in ACS and we leverage it
often when we have to do more complicated configurations, especially when
offering managed network services.​

Hope that helps...

*Will STEVENS*
Lead Developer



On Wed, May 3, 2017 at 11:09 AM, Andrija Panic 
wrote:

> Hi all,
>
> I'm interested to know if anyone is using private gatewyas in production,
> and what are your experience with it, any undocumented limitations, etc ?
>
> I dont see iif t supports vxlans (perhaps will try to test it myself)
>
> I really would appreciate any feedback.
>
> Thanks,
>
> --
>
> Andrija Panić
>

Private gateways - experience, anyone really using it ?

[Andrija Panic]

Hi all,

I'm interested to know if anyone is using private gatewyas in production,
and what are your experience with it, any undocumented limitations, etc ?

I dont see iif t supports vxlans (perhaps will try to test it myself)

I really would appreciate any feedback.

Thanks,

-- 

Andrija Panić

Re: Very slow Virtual Router provisioning with 4.9.2.0

[Rene Moser]

Thanks Remi for the hint and Daan for pick it up! That is why I like
open source software development and this project ;)

On 05/03/2017 02:49 PM, Daan Hoogland wrote:
> Happy to pick this up, Remi. I'm travelling now but will look at both on
> Friday.

Re: Very slow Virtual Router provisioning with 4.9.2.0

[Daan Hoogland]

Happy to pick this up, Remi. I'm travelling now but will look at both on
Friday.

Biligual auto correct use.  Read at your own risico

On 3 May 2017 2:25 pm, "Remi Bergsma"  wrote:

> Always happy to share, but I won’t have time to work on porting this to
> CloudStack any time soon.
>
> Regards, Remi
>
>
> On 03/05/2017, 13:44, "Rohit Yadav"  wrote:
>
> Hi Remi, thanks for sharing. We would love to have those changes (for
> 4.9+), looking forward to your pull requests.
>
>
> Regards.
>
> 
> From: Remi Bergsma 
> Sent: 03 May 2017 16:58:18
> To: dev@cloudstack.apache.org
> Subject: Re: Very slow Virtual Router provisioning with 4.9.2.0
>
> Hi,
>
> The patches I talked about:
>
> 1) Iptables speed improvement
> https://github.com/apache/cloudstack/pull/1482
> Was reverted due to a licensing issue.
>
> 2) Passwd speed improvement
> https://github.com/MissionCriticalCloudOldRepos/cosmic-core/pull/138
>
> By now, these are rather old patches so they need some work before
> they apply to CloudStack again.
>
> Regards, Remi
>
>
>
> On 03/05/2017, 12:49, "Jeff Hair"  wrote:
>
> Hi Remi,
>
> Do you have a link to the PR that was reverted? And also possibly
> the code
> that makes the password updating more efficient?
>
> Jeff
>
> On Wed, May 3, 2017 at 10:36 AM, Remi Bergsma <
> rberg...@schubergphilis.com>
> wrote:
>
> > Hi Wido,
> >
> > When we had similar issues last year, we found that for example
> comparing
> > the iptables rules one-by-one is 1000x slower than simply
> loading them all
> > at once. Boris rewrote this part in our Cosmic fork, may be
> worth looking
> > into this again. The PR to CloudStack was merged, but reverted
> later, can't
> > remember why. We run it in production ever since. Also feeding
> passwords to
> > the passwd server is very inefficient (it operates like a
> snowball and gets
> > slower once you have more VMs). That we also fixed in Cosmic,
> not sure if
> > that patch made it upstream. Wrote it about a year ago already.
> >
> > We tested applying 10K iptables rules in just a couple of
> seconds. 1000
> > VMs takes a few minutes to deploy.
> >
> > Generally speaking I'd suggest looking at the logs to find what
> takes long
> > or is executed a lot of times. Iptables and passwd are two to
> look at.
> >
> > If you want I can lookup the patches. Not handy on my phone now
> ;-)
> >
> > Regards, Remi
> > 
> > From: Wido den Hollander 
> > Sent: Tuesday, May 2, 2017 7:57:08 PM
> > To: dev@cloudstack.apache.org
> > Subject: Very slow Virtual Router provisioning with 4.9.2.0
> >
> > Hi,
> >
> > Last night I upgraded a CloudStack 4.5.2 setup to 4.9.2.0. All
> went well,
> > but the VR provisioning is terribly slow which causes all kinds
> of problems.
> >
> > The vr_cfg.sh and update_config.py scripts start to run. Restart
> dnsmasq,
> > add metadata, etc.
> >
> > But for just 1800 hosts this can take up to 2 hours and that
> causes
> > timeouts in the management server and other problems.
> >
> > 2 hours is just very, very slow. So I am starting to wonder if
> something
> > is wrong here.
> >
> > Did anybody else see this?
> >
> > Running Basic Networking with CloudStack 4.9.2.0
> >
> > Wido
> >
>
>
>
>
> rohit.ya...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
>
>

Re: ovf file parser

[Will Stevens]

Cool. Let me know if you have questions.

My instinct is that we probably want to keep the Ova manipulation in the
context of vmware since I don't believe it will be used outside that
context. Trying to manipulate the ovf files with generic tools may prove to
be more complicated to manage going forward as it is almost guaranteed to
require some 'hacks' to make it work. If we can avoid those by using the
vim jars, it may be worth it. I have not reviewed anything on the vim jars
side, so I don't know how good of an option that is.

Are there key benefits to not using the vim jars?

Cheers,

Will

On May 3, 2017 3:34 AM, "Abhinandan Prateek" <
abhinandan.prat...@shapeblue.com> wrote:

> Hi Will,
>
>I am improving the multiple disk OVA feature. As part of revamp I am
> moving out some OVF manipulation code from the vmware hypervisor plugin
> context to secondary storage component. The existing code was using vim25
> and managed objects to query and rewrite the OVF file. I have rewritten
> that, using standard java w3c dom parser.
>
>The overall flow is mostly similar and as below:
> 1. Decompress OVA and read the OVF file. OVF file will give information
> about various disks
> 3. Create the regular cloudstack template out for the boot disk and
> rewrite the OVF file, minus the information about other disks.
> 4. For each additional disk create data disk templates and capture the
> relationship in db.
> 5. This can then be followed by creating the multi-disk cloudstack VM.
>
> Essentially I am rewriting the original OVF file after removing the File
> and Disk information that refers to the other disks.  Given that the the
> VMWare is picky, I think it will require some more cleanup and massaging.
> Your inputs will definitely help.
>
> Overall I think the two pieces, the tool that you have and the cloudstack
> multi disk OVA functionality can nicely complement each other. Will post my
> learning here.
>
> Thanks and regards,
> -abhi
>
>
>
>
> On 02/05/17, 6:05 PM, "williamstev...@gmail.com on behalf of Will
> Stevens" 
> wrote:
>
> >Hey Abhinandan,
> >First, can you give us a bit more context regarding what you are doing so
> >we can highlight potential areas to watch out for?  I have done some OVF
> >parsing/modification and there are a bunch of gotchas to be aware of.  I
> >will try to outline some of the ones I found.  I have not tried to use the
> >vim25.jar, so I can't really help on that front.
> >
> >In my use case, I was exporting VMs via the ovftool from a source VMware
> >environment, and I was migrating them to an ACS managed VMware
> >environment.  In doing so, I also wanted to support VMs with multiple
> disks
> >using a Root volume and multiple Data volumes, as well as change the nic
> >type (vmxnet3), assign static IPs, etc...  I have not had the time to open
> >source my migration tool, but it is on my todo list.
> >
> >My general flow was:
> >- Export the VM with ovftool
> >- Extract the resulting OVA into its parts (OVF, VMDKs, Manifest)
> >- Duplicate the OVF file, once per VMDK
> >- Modify a OVF file to be specific for each of the VMDKs (one OVF per
> VMDK)
> >- Take each VMDK and the corresponding OVF and recompress them back into
> an
> >OVA
> >- Treat the first OVA as a template and the rest as data disks
> >
> >My initial (naive) approach was to just treat the OVF as a well behaved
> XML
> >file and use standard XML libs (in my case in Python) to parse and
> >manipulate the OVF file.  This approach had a few pitfalls which I will
> >outline here.
> >
> >VMware is VERY picky about the format of the OVF file, if the file is not
> >perfect, VMware won't import it (or at least the VM won't launch).  There
> >were two main items which caused me issues.
> >
> >a) The  tag MUST have all of the namespace definitions even if
> >they are not used in the file.  This is something that most XML parsers
> are
> >confused by.  Most XML parsers will only include the namespaces used in
> the
> >file when the file is saved.  I had to ensure that the resulting OVF files
> >had all of the original namespace definitions for the file to import
> >correctly.  If I remember correctly, they even had to be in the right
> >order.  I did this by changing the resulting file after saving it with the
> >XML lib.
> >
> >b) VMware uses namespaces which actually collide with each other.  For
> >example, both the default namespace and the 'ovf' namespace share the same
> >URL.  Again, XML libraries don't understand this, so I had to manage that
> >manually.  Luckily, the way VMware handles these namespaces is relatively
> >consistent, so I was able to find a workaround.  Basically, the default
> >namespace will apply to all of the elements, and the 'ovf' namespace will
> >be applied only in the attributes.  Because of this I was able to just use
> >the 'ovf' namespace and then after exporting the file, I did a find
> replace
> >from '
> 

Re: Very slow Virtual Router provisioning with 4.9.2.0

[Remi Bergsma]

Always happy to share, but I won’t have time to work on porting this to 
CloudStack any time soon.

Regards, Remi


On 03/05/2017, 13:44, "Rohit Yadav"  wrote:

Hi Remi, thanks for sharing. We would love to have those changes (for 
4.9+), looking forward to your pull requests.


Regards.


From: Remi Bergsma 
Sent: 03 May 2017 16:58:18
To: dev@cloudstack.apache.org
Subject: Re: Very slow Virtual Router provisioning with 4.9.2.0

Hi,

The patches I talked about:

1) Iptables speed improvement
https://github.com/apache/cloudstack/pull/1482
Was reverted due to a licensing issue.

2) Passwd speed improvement
https://github.com/MissionCriticalCloudOldRepos/cosmic-core/pull/138

By now, these are rather old patches so they need some work before they 
apply to CloudStack again.

Regards, Remi



On 03/05/2017, 12:49, "Jeff Hair"  wrote:

Hi Remi,

Do you have a link to the PR that was reverted? And also possibly the 
code
that makes the password updating more efficient?

Jeff

On Wed, May 3, 2017 at 10:36 AM, Remi Bergsma 

wrote:

> Hi Wido,
>
> When we had similar issues last year, we found that for example 
comparing
> the iptables rules one-by-one is 1000x slower than simply loading 
them all
> at once. Boris rewrote this part in our Cosmic fork, may be worth 
looking
> into this again. The PR to CloudStack was merged, but reverted later, 
can't
> remember why. We run it in production ever since. Also feeding 
passwords to
> the passwd server is very inefficient (it operates like a snowball 
and gets
> slower once you have more VMs). That we also fixed in Cosmic, not 
sure if
> that patch made it upstream. Wrote it about a year ago already.
>
> We tested applying 10K iptables rules in just a couple of seconds. 
1000
> VMs takes a few minutes to deploy.
>
> Generally speaking I'd suggest looking at the logs to find what takes 
long
> or is executed a lot of times. Iptables and passwd are two to look at.
>
> If you want I can lookup the patches. Not handy on my phone now ;-)
>
> Regards, Remi
> 
> From: Wido den Hollander 
> Sent: Tuesday, May 2, 2017 7:57:08 PM
> To: dev@cloudstack.apache.org
> Subject: Very slow Virtual Router provisioning with 4.9.2.0
>
> Hi,
>
> Last night I upgraded a CloudStack 4.5.2 setup to 4.9.2.0. All went 
well,
> but the VR provisioning is terribly slow which causes all kinds of 
problems.
>
> The vr_cfg.sh and update_config.py scripts start to run. Restart 
dnsmasq,
> add metadata, etc.
>
> But for just 1800 hosts this can take up to 2 hours and that causes
> timeouts in the management server and other problems.
>
> 2 hours is just very, very slow. So I am starting to wonder if 
something
> is wrong here.
>
> Did anybody else see this?
>
> Running Basic Networking with CloudStack 4.9.2.0
>
> Wido
>




rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: Very slow Virtual Router provisioning with 4.9.2.0

[Rohit Yadav]

Hi Remi, thanks for sharing. We would love to have those changes (for 4.9+), 
looking forward to your pull requests.


Regards.


From: Remi Bergsma 
Sent: 03 May 2017 16:58:18
To: dev@cloudstack.apache.org
Subject: Re: Very slow Virtual Router provisioning with 4.9.2.0

Hi,

The patches I talked about:

1) Iptables speed improvement
https://github.com/apache/cloudstack/pull/1482
Was reverted due to a licensing issue.

2) Passwd speed improvement
https://github.com/MissionCriticalCloudOldRepos/cosmic-core/pull/138

By now, these are rather old patches so they need some work before they apply 
to CloudStack again.

Regards, Remi



On 03/05/2017, 12:49, "Jeff Hair"  wrote:

Hi Remi,

Do you have a link to the PR that was reverted? And also possibly the code
that makes the password updating more efficient?

Jeff

On Wed, May 3, 2017 at 10:36 AM, Remi Bergsma 
wrote:

> Hi Wido,
>
> When we had similar issues last year, we found that for example comparing
> the iptables rules one-by-one is 1000x slower than simply loading them all
> at once. Boris rewrote this part in our Cosmic fork, may be worth looking
> into this again. The PR to CloudStack was merged, but reverted later, 
can't
> remember why. We run it in production ever since. Also feeding passwords 
to
> the passwd server is very inefficient (it operates like a snowball and 
gets
> slower once you have more VMs). That we also fixed in Cosmic, not sure if
> that patch made it upstream. Wrote it about a year ago already.
>
> We tested applying 10K iptables rules in just a couple of seconds. 1000
> VMs takes a few minutes to deploy.
>
> Generally speaking I'd suggest looking at the logs to find what takes long
> or is executed a lot of times. Iptables and passwd are two to look at.
>
> If you want I can lookup the patches. Not handy on my phone now ;-)
>
> Regards, Remi
> 
> From: Wido den Hollander 
> Sent: Tuesday, May 2, 2017 7:57:08 PM
> To: dev@cloudstack.apache.org
> Subject: Very slow Virtual Router provisioning with 4.9.2.0
>
> Hi,
>
> Last night I upgraded a CloudStack 4.5.2 setup to 4.9.2.0. All went well,
> but the VR provisioning is terribly slow which causes all kinds of 
problems.
>
> The vr_cfg.sh and update_config.py scripts start to run. Restart dnsmasq,
> add metadata, etc.
>
> But for just 1800 hosts this can take up to 2 hours and that causes
> timeouts in the management server and other problems.
>
> 2 hours is just very, very slow. So I am starting to wonder if something
> is wrong here.
>
> Did anybody else see this?
>
> Running Basic Networking with CloudStack 4.9.2.0
>
> Wido
>




rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: Very slow Virtual Router provisioning with 4.9.2.0

[Remi Bergsma]

Hi,

The patches I talked about:

1) Iptables speed improvement
https://github.com/apache/cloudstack/pull/1482 
Was reverted due to a licensing issue. 

2) Passwd speed improvement
https://github.com/MissionCriticalCloudOldRepos/cosmic-core/pull/138 

By now, these are rather old patches so they need some work before they apply 
to CloudStack again.

Regards, Remi



On 03/05/2017, 12:49, "Jeff Hair"  wrote:

Hi Remi,

Do you have a link to the PR that was reverted? And also possibly the code
that makes the password updating more efficient?

Jeff

On Wed, May 3, 2017 at 10:36 AM, Remi Bergsma 
wrote:

> Hi Wido,
>
> When we had similar issues last year, we found that for example comparing
> the iptables rules one-by-one is 1000x slower than simply loading them all
> at once. Boris rewrote this part in our Cosmic fork, may be worth looking
> into this again. The PR to CloudStack was merged, but reverted later, 
can't
> remember why. We run it in production ever since. Also feeding passwords 
to
> the passwd server is very inefficient (it operates like a snowball and 
gets
> slower once you have more VMs). That we also fixed in Cosmic, not sure if
> that patch made it upstream. Wrote it about a year ago already.
>
> We tested applying 10K iptables rules in just a couple of seconds. 1000
> VMs takes a few minutes to deploy.
>
> Generally speaking I'd suggest looking at the logs to find what takes long
> or is executed a lot of times. Iptables and passwd are two to look at.
>
> If you want I can lookup the patches. Not handy on my phone now ;-)
>
> Regards, Remi
> 
> From: Wido den Hollander 
> Sent: Tuesday, May 2, 2017 7:57:08 PM
> To: dev@cloudstack.apache.org
> Subject: Very slow Virtual Router provisioning with 4.9.2.0
>
> Hi,
>
> Last night I upgraded a CloudStack 4.5.2 setup to 4.9.2.0. All went well,
> but the VR provisioning is terribly slow which causes all kinds of 
problems.
>
> The vr_cfg.sh and update_config.py scripts start to run. Restart dnsmasq,
> add metadata, etc.
>
> But for just 1800 hosts this can take up to 2 hours and that causes
> timeouts in the management server and other problems.
>
> 2 hours is just very, very slow. So I am starting to wonder if something
> is wrong here.
>
> Did anybody else see this?
>
> Running Basic Networking with CloudStack 4.9.2.0
>
> Wido
>

Re: Very slow Virtual Router provisioning with 4.9.2.0

[Remi Bergsma]

Hi Wido,

When we had similar issues last year, we found that for example comparing the 
iptables rules one-by-one is 1000x slower than simply loading them all at once. 
Boris rewrote this part in our Cosmic fork, may be worth looking into this 
again. The PR to CloudStack was merged, but reverted later, can't remember why. 
We run it in production ever since. Also feeding passwords to the passwd server 
is very inefficient (it operates like a snowball and gets slower once you have 
more VMs). That we also fixed in Cosmic, not sure if that patch made it 
upstream. Wrote it about a year ago already.

We tested applying 10K iptables rules in just a couple of seconds. 1000 VMs 
takes a few minutes to deploy.

Generally speaking I'd suggest looking at the logs to find what takes long or 
is executed a lot of times. Iptables and passwd are two to look at.

If you want I can lookup the patches. Not handy on my phone now ;-)

Regards, Remi

From: Wido den Hollander 
Sent: Tuesday, May 2, 2017 7:57:08 PM
To: dev@cloudstack.apache.org
Subject: Very slow Virtual Router provisioning with 4.9.2.0

Hi,

Last night I upgraded a CloudStack 4.5.2 setup to 4.9.2.0. All went well, but 
the VR provisioning is terribly slow which causes all kinds of problems.

The vr_cfg.sh and update_config.py scripts start to run. Restart dnsmasq, add 
metadata, etc.

But for just 1800 hosts this can take up to 2 hours and that causes timeouts in 
the management server and other problems.

2 hours is just very, very slow. So I am starting to wonder if something is 
wrong here.

Did anybody else see this?

Running Basic Networking with CloudStack 4.9.2.0

Wido

Re: Very slow Virtual Router provisioning with 4.9.2.0

[Jayapal Uradi]

Another reason of slow can be VR configuration(persistent VR configuration 
design).  When one component config apply, whole VR configuration apply is 
executed. Due to this the VR boot up time will increase.

Thanks,
Jayapal


> On May 3, 2017, at 1:55 PM, Marc-Aurèle Brothier  wrote:
> 
> Hi Wido,
> 
> Well for us, it's not a version problem, it's simply a design problem. This
> VR is very problematic during any upgrade of cloudstack (which I perform
> every week almost on our platform), same goes for the secondary storage VMs
> which scans all templates. We've planned on our roadmap to get rid of the
> system vms. The VR is really a SPoF.
> 
> On Tue, May 2, 2017 at 7:57 PM, Wido den Hollander  wrote:
> 
>> Hi,
>> 
>> Last night I upgraded a CloudStack 4.5.2 setup to 4.9.2.0. All went well,
>> but the VR provisioning is terribly slow which causes all kinds of problems.
>> 
>> The vr_cfg.sh and update_config.py scripts start to run. Restart dnsmasq,
>> add metadata, etc.
>> 
>> But for just 1800 hosts this can take up to 2 hours and that causes
>> timeouts in the management server and other problems.
>> 
>> 2 hours is just very, very slow. So I am starting to wonder if something
>> is wrong here.
>> 
>> Did anybody else see this?
>> 
>> Running Basic Networking with CloudStack 4.9.2.0
>> 
>> Wido
>> 




DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infected mails.

Re: Very slow Virtual Router provisioning with 4.9.2.0

[Marc-Aurèle Brothier]

Hi Wido,

Well for us, it's not a version problem, it's simply a design problem. This
VR is very problematic during any upgrade of cloudstack (which I perform
every week almost on our platform), same goes for the secondary storage VMs
which scans all templates. We've planned on our roadmap to get rid of the
system vms. The VR is really a SPoF.

On Tue, May 2, 2017 at 7:57 PM, Wido den Hollander  wrote:

> Hi,
>
> Last night I upgraded a CloudStack 4.5.2 setup to 4.9.2.0. All went well,
> but the VR provisioning is terribly slow which causes all kinds of problems.
>
> The vr_cfg.sh and update_config.py scripts start to run. Restart dnsmasq,
> add metadata, etc.
>
> But for just 1800 hosts this can take up to 2 hours and that causes
> timeouts in the management server and other problems.
>
> 2 hours is just very, very slow. So I am starting to wonder if something
> is wrong here.
>
> Did anybody else see this?
>
> Running Basic Networking with CloudStack 4.9.2.0
>
> Wido
>

Re: ovf file parser

[Abhinandan Prateek]

Hi Will,

   I am improving the multiple disk OVA feature. As part of revamp I am moving 
out some OVF manipulation code from the vmware hypervisor plugin context to 
secondary storage component. The existing code was using vim25 and managed 
objects to query and rewrite the OVF file. I have rewritten that, using 
standard java w3c dom parser. 

   The overall flow is mostly similar and as below:
1. Decompress OVA and read the OVF file. OVF file will give information about 
various disks 
3. Create the regular cloudstack template out for the boot disk and rewrite the 
OVF file, minus the information about other disks.
4. For each additional disk create data disk templates and capture the 
relationship in db.
5. This can then be followed by creating the multi-disk cloudstack VM.

Essentially I am rewriting the original OVF file after removing the File and 
Disk information that refers to the other disks.  Given that the the VMWare is 
picky, I think it will require some more cleanup and massaging. Your inputs 
will definitely help.

Overall I think the two pieces, the tool that you have and the cloudstack multi 
disk OVA functionality can nicely complement each other. Will post my learning 
here.

Thanks and regards,
-abhi




On 02/05/17, 6:05 PM, "williamstev...@gmail.com on behalf of Will Stevens" 
 wrote:

>Hey Abhinandan,
>First, can you give us a bit more context regarding what you are doing so
>we can highlight potential areas to watch out for?  I have done some OVF
>parsing/modification and there are a bunch of gotchas to be aware of.  I
>will try to outline some of the ones I found.  I have not tried to use the
>vim25.jar, so I can't really help on that front.
>
>In my use case, I was exporting VMs via the ovftool from a source VMware
>environment, and I was migrating them to an ACS managed VMware
>environment.  In doing so, I also wanted to support VMs with multiple disks
>using a Root volume and multiple Data volumes, as well as change the nic
>type (vmxnet3), assign static IPs, etc...  I have not had the time to open
>source my migration tool, but it is on my todo list.
>
>My general flow was:
>- Export the VM with ovftool
>- Extract the resulting OVA into its parts (OVF, VMDKs, Manifest)
>- Duplicate the OVF file, once per VMDK
>- Modify a OVF file to be specific for each of the VMDKs (one OVF per VMDK)
>- Take each VMDK and the corresponding OVF and recompress them back into an
>OVA
>- Treat the first OVA as a template and the rest as data disks
>
>My initial (naive) approach was to just treat the OVF as a well behaved XML
>file and use standard XML libs (in my case in Python) to parse and
>manipulate the OVF file.  This approach had a few pitfalls which I will
>outline here.
>
>VMware is VERY picky about the format of the OVF file, if the file is not
>perfect, VMware won't import it (or at least the VM won't launch).  There
>were two main items which caused me issues.
>
>a) The  tag MUST have all of the namespace definitions even if
>they are not used in the file.  This is something that most XML parsers are
>confused by.  Most XML parsers will only include the namespaces used in the
>file when the file is saved.  I had to ensure that the resulting OVF files
>had all of the original namespace definitions for the file to import
>correctly.  If I remember correctly, they even had to be in the right
>order.  I did this by changing the resulting file after saving it with the
>XML lib.
>
>b) VMware uses namespaces which actually collide with each other.  For
>example, both the default namespace and the 'ovf' namespace share the same
>URL.  Again, XML libraries don't understand this, so I had to manage that
>manually.  Luckily, the way VMware handles these namespaces is relatively
>consistent, so I was able to find a workaround.  Basically, the default
>namespace will apply to all of the elements, and the 'ovf' namespace will
>be applied only in the attributes.  Because of this I was able to just use
>the 'ovf' namespace and then after exporting the file, I did a find replace
>from 'Those are the main gotchas which I encountered.
>
>I put the OVA Split function I wrote into a Gist [1] (for now) for your
>reference in case reviewing the code is helpful.  I was under a lot of time
>pressure when building this tool, so I have a bunch of cleanup to do before
>I release it as open source, but I can rush it out and clean it up after
>release if you are solving the same(ish) problem and my code will be useful.
>
>[1] https://gist.github.com/swill/f6b54762ffcce85772535a490a9c8cbe
>
>I hope this is helpful in your case.
>
>Cheers,
>
>*Will STEVENS*
>Lead Developer
>
>
>
>On Tue, May 2, 2017 at 3:49 AM, Abhinandan Prateek <
>abhinandan.prat...@shapeblue.com> wrote:
>
>> Hello,
>>
>> I am looking at vim25.jar to put together ovftool like functionality
>> specially around parsing and generating OVF files.