Abstracting that interaction would also make future implementations MUCH
less painful and allows third parties who want their networking products
to be CS compatible to write their own implementation of a published API
that CS defines. Let them do the legwork for their products, expand CS
compatibility, and ease future refactors of VR all in one package. I
like it.
Caveat: I am clueless about how VR configuration happens under the hood
currently or if abstracting it is a monumental task.
Matthew Smart
President
Smart Software Solutions Inc.
108 S Pierre St.
Pierre, SD 57501
Phone: (605) 280-0383
Skype: msmart13
Email: msm...@smartsoftwareinc.com
On 09/19/2016 11:50 AM, Paul Angus wrote:
Hi All,
From the looks of this thread and the different preferences that parties have,
the way forward looks to be a model that allows multiple different VRs to be
used is the answer.
This is something that I have had in mind for some time now.
By abstracting the roles and configuration of the VR, drivers can be written
which transform requests such as 'add these firewall rules' into something that
a VyOs, Fortigate, pfSense, Cisco ASAv, a Juniper vSRX or a homebrew VR can
understand and implement.
Ideally I'd like to see the features separated and service chaining introduced
such that someone might use a stock VR with a VPN appliance or a Cisco ASAv in
front of a VR doing load balancing.
This moves us forward into a much more NFV-like world.
Kind regards,
Paul Angus
paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-----Original Message-----
From: Syed Ahmed [mailto:sah...@cloudops.com]
Sent: 19 September 2016 17:07
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Replacing the VR
Hey Guys,
Will and I had a discussion in the morning on around VyOS and I have an
idea that could work, here me out.
The way Cloudstack currently communicates with VR is via the Hypervisor
(let's assume KVM and XenServer for now). The management server sends a
command as a JSON to the hypervisor which proxies them to the VR and they
get executed there. Now, what we could do is add a proxy on the hypervisor
which translates the management server JSON to VyOS commands (and
vice-versa). There is a VyOS API lib in python which we clould use. Now
this would require 0 changes on the Cloudstack core networking side. There
may be a few minor changes required to push this proxy on the hypervisor
but other than that there is nothing major. Now in the mean time, the VyOS
guys can work on their v2.0 and when they have a stable enough API we can
make it a first class citizen in Cloudstack.
This approach would not work for VMWare as in VMWare the management server
directly talks to the VR. However, this problem could be fixed by adding a
middle VM which does the proxying to-and-from the VR. This would also
improve the overall security of the system in VMWare as well.
I am not too concerned about VMWare at the moment as most of us use either
KVM or XenServer. What do you guys think of this approach?
-Syed
On Mon, Sep 19, 2016 at 11:42 AM, Stephan Seitz <
s.se...@secretresearchfacility.com> wrote:
Hi!
Just to add my 2 cents to that thread:
I'ld really like to see something like vyatta or pfsense integrated as
"standard" VR.
We'd also talked internally about replacing the VR with some more
mature "appliance"-like router distro.
pfsense e.g. comes AFAIK with no defined API but instead has a very
nice GUI.
How would this fit into the concept of configuring the VR via ACS?
Would parts of the GUI - like IP configuration and basic firewall rules
- hidden or greyed?
Where would one save the configuration, VPN certificates and so on?
- Stephan
Am Sonntag, den 18.09.2016, 15:19 +0000 schrieb Marty Godsey:
On this note I also mentioned pfsense earlier.
www.pfsense.org
Regards,
Marty Godsey
-----Original Message-----
From: ilya [mailto:ilya.mailing.li...@gmail.com]
Sent: Sunday, September 18, 2016 1:09 AM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Replacing the VR
Our options become much better if we consider BSD based routers.
Would that be on the table?
https://en.wikipedia.org/wiki/List_of_router_and_firewall_distributio
ns
On 9/16/16 12:04 PM, Will Stevens wrote:
Ya, your points are all valid Simon. The lack of standard
libraries
to handle a lot of the details is a problem. I don't think it is
an
unsolvable problem, but if we spend the time to do that, will we
have
something that will work for us for the next 5 years? This may be
the
shortest path to getting us where we need to be for the time being.
What is the best case scenario for the VR going forward which will
last us the next 5 years? Maybe we just clean up what we have to
do a
major restructuring of the pieces and how they are
implemented. We
need to keep in mind how maintainable this implementation is
because
that is going to be key going forward IMO.
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|*
tw
@CloudOps_
On Fri, Sep 16, 2016 at 2:29 PM, Simon Weller <swel...@ena.com>
wrote:
I think our other option is to take a real look at what it would
take
to fix the VR. In my opinion, a lot of the problems are related
to
the monolithic python code base and the fact nothing is actually
separated.
Secondly, the python scripts (and bash scripts) don't use any
established libraries to complete tasks and instead shell out and
run
commands that are both hard to track and hard to parse on return.
If we daemonized this, used a real api for Agent to VR
communication,
used common already existing libraries for the system service
and
network interactions and spent a bit of time separating out code
into
distinct modules, everything would behave a lot better.
The pain and suffering is due to years and years of patches and
constant shelling out to complete tasks in my opinion. If we
spend
time to rethink how we interact with the VR in general and we
abstract the systems and networking stuff and use well known and
stable libraries to do the work, the VR would be much easier to
maintain.
- Si
________________________________
From: Marty Godsey <ma...@gonsource.com>
Sent: Friday, September 16, 2016 12:24 PM
To: dev@cloudstack.apache.org
Subject: RE: [DISCUSS] Replacing the VR
So based upon this discussion would it be prudent to wait on
VyOS
2.0? The current VR is giving us issues but would the time
invested
in another "solution" be wasted especially if by the time
another
option is chose, then coded, then tested, then implemented and
right
as that time happened to be when VyOS 2.0 is released. Of course
you
said they are just in the scoping range so this could still be a
year or more out.
Thoughts?
Regards,
Marty Godsey
nSource Solutions
-----Original Message-----
From: williamstev...@gmail.com [mailto:williamstev...@gmail.com]
On
Behalf Of Will Stevens
Sent: Friday, September 16, 2016 10:31 AM
To: dev@cloudstack.apache.org
Cc: dan...@baturin.org
Subject: Re: [DISCUSS] Replacing the VR
I just had a quick chat with a couple of the guys over on the
VyOS chat.
I have CC'ed one of them in case we have more licensing
questions.
So here is the status with the license "the code inherited from
Vyatta and our modifications from it is GPLv2 (strict, not v2+).
The
config reading library is GPLv2 too, so anything that links to is
is GPLv2.
Some auxiliary components we made after the fork are more
permissive,
LGPLv2+ or MIT."
They are currently in the process of scoping a redesign (VyOS
2.0),
"we are planning a clean rewrite that will solve issues of the
current config system".
This will include the ability to configure via the API.
If we have more questions for VyOS, they are very friendly and
responsive, so we should be able to get answers.
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com
*|* tw
@CloudOps_
On Fri, Sep 16, 2016 at 9:37 AM, Syed Ahmed <sah...@cloudops.com>
wrote:
I agree with Will Ilya. There are so many problems with the VR
right now.
Most of the outages we've had recently have somehow involved
the VR.
We set custom iptables rules on the VR which can and have
easily
gone
wrong.
Openswan is broken, Strongswan replacement still needs to be
tested.
VVRP with redundant router still needs work, and not to mention
the
problems we will have when we introduce IPv6 into the whole
picture.
I think the spirit of the discussion is to rely on a 3rd party
to do
the networking for us (eg VyOS) and have us handle just the
orchestration. All the problems that I've described have
already
been solved in VyOS. We also get the advantage of a potential
wider
community to fix and maintain the VR and given our current
development velocity, it think it totally makes sense to look
for a 3rd party option.
-Syed
On Fri, Sep 16, 2016 at 9:18 AM, Will Stevens
<wstev...@cloudops.com>
wrote:
The VR has been biting us far too often recently, which is
why we
have started looking into alternative implementations.
One of the things that is nice about potentially using the
VyOS is
that
it
is based on Debian, so we should be able to run the other
services
that
we
currently have like the password server and userdata on the
VyOS.
This means we would not have to change our architecture
initially
and could focus on only replacing the networking paths.
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
cloudops.com *|*
tw @CloudOps_
On Fri, Sep 16, 2016 at 6:20 AM, Nux! <n...@li.nux.ro> wrote:
The more this is discussed the more I think we should stick
with
our
VR.
All these other options either seem unfinished or with
incompatible license.
VyOS looks the most promising so far, it's a serious,
mature project.
Adopting it though means we'll have to microservice our way
out of
it
with
extra machines for DNS/USERDATA/etc, unless we can make
VyOS serve
those
too. Imho this adds complexity we should void.
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
From: "Will Stevens" <wstev...@cloudops.com>
To: dev@cloudstack.apache.org
Sent: Thursday, 15 September, 2016 17:21:28
Subject: Re: [DISCUSS] Replacing the VR
Ya, we would need to add a daemon for VPN as well. Load
balancing is another aspect which we will need to
consider if we
went this route.
Something like https://traefik.io/ could potentially be a
good
fit
due
to
its API driven configuration, but it may be more than
what we need.
We should probably try define which pieces make sense to
be
solved
together
and which pieces would be best suited to be broken out.
I think the network connectivity, routing and firewalling
should
probably
all stay together since the majority of the tools we
would
potentially
use
would handle all of that together in a single
implementation.
The password server and userdata seems like a good option
for
being
broken
out and handled independently (and probably rewritten
completely
since
they
currently have some issues).
Load balancing is another that could warrant splitting
out, but
that depends on what direction we go and how we would be
managing
it.
DHCP
and
DNS are others which could go either way.
If we do split out services, I think we should
consolidate as
much as
we
can into each service we break out. Ideally a network
packet
would
never
hit more than one, maybe two, services. I don't think we
should
be splitting services 'just because', I think we need a
valid
case for splitting any service out because it adds
complexity.
Our project is already complex enough, we need to avoid
adding
complexity unless it
is
really needed.
Some more of my thoughts on this anyway...
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
cloudops.com
*|* tw @CloudOps_
On Thu, Sep 15, 2016 at 10:28 AM, Simon Weller <sweller@e
na.com>
wrote:
I do agree with you that this probably isn't the right
place the
password
service and user data.
Having said that, after taking a cursory look at the
dev docs,
it
doesn't
seem that difficult to add new daemons:
https://opensnaproute.github.
io/docs/developer.html#creating-new-component
<https://opensnaproute.github.io/docs/developer.html#
creating-new-component>
They've definitely build it with a microservices
architecture in
mind,
so
each individual feature is abstracted into it's own
small daemon
process.
We could just create a daemon for the password server
and the
userdata
components if we really had to.
- Si
________________________________
From: williamstev...@gmail.com <williamstev...@gmail.co
m> on
behalf
of
Will Stevens <wstev...@cloudops.com>
Sent: Thursday, September 15, 2016 9:17 AM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Replacing the VR
A big part of why I know about it is because it is
written in Go.
:P
Yes, it is definitely interesting for the routing and
traffic
handling
aspects of the VR. We will likely have to rethink some
of the
pieces
a
little bit like the password server and userdata if we
are to
adopt
a
different VR approach. This is where I think some of
JohnB and
Chiradeep's
ideas make sense. In many ways, it does not make sense
for the
device
handling routing and network traffic to also be
responsible for
passwords
and userdata.
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
cloudops.com
*|* tw @CloudOps_
On Thu, Sep 15, 2016 at 9:10 AM, Simon Weller <sweller@
ena.com>
wrote:
I hadn't heard of Flexswitch until you mentioned it.
It looks
pretty
cool!
It even supports ONIE install.
To be honest, the ipsec feature could be added, or we
could
offload
it to
separate vm if we needed to. The fact it is so
feature rich
from a
routing
perspective (and all API driven) is really nice.
Based on the roadmap, it looks like they plan to also
support
capabilities
such as BGP-MPLS based L3VPN, EVPN, VPLS in the
future. This
will
be
huge
for our carrier community that rely on these
technologies to do
private
gateway and inter-VPC interconnections today. We
handle this
stuff
on
our
ASRs right now with a vlan interconnect into the VR.
Being able
to
do
MPLS
all the way to the VR would be awesome.
It also seems to be written in GO (a language here at
ENA we
know
very
well).
- Si
________________________________
From: Will Stevens <williamstev...@gmail.com>
Sent: Thursday, September 15, 2016 7:06 AM
To: dev@cloudstack.apache.org
Subject: RE: [DISCUSS] Replacing the VR
Ya. I don't think it covers our whole use case, but
what it
does
cover is
all api driven...
On Sep 15, 2016 1:48 AM, "Marty Godsey" <marty@gonsou
rce.com>
wrote:
Though I don’t see VPN in Snaproute.. Makes sense
since it was
not
intended to do IPSec.
It seems as though VyOS is starting to look like
the best
option.
Regards,
Marty Godsey
nSource Solutions
-----Original Message-----
From: williamstev...@gmail.com
[mailto:williamstev...@gmail.com
]
On
Behalf Of Will Stevens
Sent: Wednesday, September 14, 2016 11:06 PM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Replacing the VR
Or we could go completely crazy and go with
something like
FlexSwitch
from
SnapRoute
- http://www.snaproute.com/
- https://opensnaproute.github.io/docs/apis.html
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
cloudops.com
*|*
tw
@CloudOps_
On Wed, Sep 14, 2016 at 10:55 PM, Will Stevens <
wstev...@cloudops.com>
wrote:
I tend to agree with Syed and Marty. I am not
sure what
problems
are
solved by splitting up the function of the VR
into a bunch of
separate
services. As Syed points out, the complexity
added is
non-trivial.
We now have to manage all the intercontainer
networking as
well
as
the
orchestrated ACS networking.
VyOS is interesting to me because it covers the
majority of
our
use
case with a single unified control plane. It
also has good
support
for extending features we care about, like IPv6,
VXLAN, VRRP,
transactions, etc...
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
cloudops.com
*|*
tw
@CloudOps_
On Wed, Sep 14, 2016 at 9:49 PM, Syed Ahmed <
sah...@cloudops.com>
wrote:
Agree with Marty, adding Docker containers to
the picture
although
can make the VR more flexible but the added
complexity is
just
not
worth it. Not to mention we would need to take
care of
networking
each container manually and given that our
iptable rules are
very
unstable at the moment I don't see a big value
add.
Vyos looks like a better solution to me. I know
that it does
not
provide an api but it does fit the bill quite
well
otherwise. I
specially like the fact that it has a
transaction based
model
and
you
can rollback changes if something goes wrong.
On Wed, Sep 14, 2016 at 9:06 PM Marty Godsey <
ma...@gonsource.com>
wrote:
Licensing aside, I think splitting the
various functions
into
containers is not a good route either. This
will force
users
to
have to maintain
and
use containers and adds complexity to the
networking
aspects
of
ACS.
Complexity decreases stability. Now I
understand the
argument
that
a monolithic approach also brings its own set
of issues but
it
also
simplifies it.
Regards,
Marty Godsey
nSource Solutions
-----Original Message-----
From: Chiradeep Vittal [mailto:chiradeepv@gma
il.com]
Sent: Wednesday, September 14, 2016 5:37 PM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Replacing the VR
I rather doubt that the Cloudrouter will fit
the needs
of
the
CloudStack project
- it is AGPL licensed. Many enterprises will
not
touch
anything
that
has
AGPL
- the github repo shows rather infrequent
updates.
Quite
likely
they aren't considering the use cases of the
CloudStack
community
I'd back John B's comments on disaggregating
the VR.
Split
it
into
many docker containers
- password server
- userdata server
- DHCP / DNS
- s2s VPN
- RA VPN
- intra-VPC routing and ACL
- Port forwarding + NAT
- FW
- LB (public)
- LB (internal),
- secondary storage
- agent
Glue them together with docker compose files
(one per
use
case -
basic zone, isolated, VPC, SSVM, etc).
The VR image then becomes a JeOS + docker.
You can
test
each
of
the
components independently and fixing one bug
in the
field
(say
DHCP)
is hitless to the other components. You don't
need to
build per-hypervisor VRs. You could even run
on
baremetal.
Along the way you need to figure out how to
- make the traffic traverse the containers
that are
needed
to
be
traversed (in most cases just 1)
- bootstrap the router (how does it find its
compose
file?
where
is the
registry?)
- rethink the command and control of the VR
functions. SSH
works,
but something more declarative, idempotent
should be
explored.
As you do this, it becomes clearer which of
the
functions
can
be
substituted by for example CloudRouter.
Command and
Control
of
the
docker
containers can be moved out to another
container. Etc.
On Wed, Sep 14, 2016 at 12:59 AM, Marty
Godsey
<ma...@gonsource.com>
wrote:
This one does look nice. My biggest concern
is the
lack
of
VXLANs. It seems that any of the ones we
mentioned
do not
have
an
API so we may be stuck at the SSH method.
Regards,
Marty Godsey
nSource Solutions
-----Original Message-----
From: Abhinandan Prateek
[mailto:abhinandan.prat...@shapeblue.com]
Sent: Wednesday, September 14, 2016 2:26 AM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Replacing the VR
Cloudrouter looks promising. These have
potential to
save
future
engineering effort for example on ipv6
routing, OSPF
etc.
And the best part is they come with test
automation
framework.
On 13/09/16, 4:22 PM, "Jayapal Uradi"
<jayapal.ur...@accelerite.com>
wrote:
Hi,
Instead of replacing the VR in first
place we
should add VyOS/cloudrouter
as provider. Once it is stable, network
offerings
(on
upgrade)
can be updated to use it and we can drop
the VR if
we
want
at
that release
onwards.
VR is stabilized over a period of time
and some of
them
are
running
without issues. When we replicate the ACS
VR
features in
new
solution it takes some to find the missing
pieces
(hidden
bugs).
Thanks,
Jayapal
On Sep 13, 2016, at 2:52 PM, Nux! <
n...@li.nux.ro> wrote:
Hi,
I like the idea.
Cloudrouter looks really promising, I'm
not too
keen
on
VyOS
(it
doesn't have a proper http api etc).
--
Sent from the Delta quadrant using Borg
technology!
Nux!
www.nux.ro
abhinandan.prat...@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
;
53 Chandos Place, Covent Garden,
London WC2N 4HSUK
@shapeblue
----- Original Message -----
From: "Will Stevens" <williamstevens@
gmail.com>
To: dev@cloudstack.apache.org
Sent: Monday, 12 September, 2016
21:20:11
Subject: [DISCUSS] Replacing the VR
*Disclaimer:* This is a thought
experiment and
should
be
treated as
such.
Please weigh in with the good and bad
of this
idea...
A couple of us have been discussing
the idea of
potentially
replacing the ACS VR with the VyOS
[1] (Open
Source
Vyatta
VM).
There may be a license issue because
I think it
is
licensed
under GPL, but for the sake of
discussion, let's
assume
we
can overcome any
license issues.
I have spent some time recently with
the VyOS
and I
have
to
admit, I was pretty impressed. It is
simple and
intuitive
and it gives you a lot more options
for auditing
the
configuration etc...
Items of potential interest:
- Clean up our current VR script
spaghetti to a
simpler
more
auditable configuration workflow.
- Gives a cleaner path for IPv6
support.
- Handles VPN configuration via the
same
configuration
interface.
- Support for OSPF & BGP.
- VPN support through OpenVPN &
StrongSwan.
- Easily supports HA (redundant
routers) through
VRRP.
- VXLAN support.
- Transaction based changes to the VR
with
rollback
on
error.
Items that could be difficult to
solve:
- Userdata password reset workflow
and
implementation.
- Upgrade process.
The VyOS is not the only option if we
were to
consider
this
approach.
Another option, which I don't know as
well,
would be CloudRouter (AGPL
license) [2] which is purely API
driven.
Anyway, would love to hear your
thoughts...
Will
[1] https://vyos.io/ [2]
https://cloudrouter.org/
DISCLAIMER
==========
This e-mail may contain privileged and
confidential
information
which is
the property of Accelerite, a Persistent
Systems
business.
It is
intended only for the use of the individual
or
entity to
which
it
is addressed. If you are not the intended
recipient,
you
are
not
authorized to read, retain, copy, print,
distribute
or
use
this
message. If you have received this
communication in
error,
please
notify the sender and delete all copies of
this
message.
Accelerite, a Persistent Systems business
does not
accept
any
liability for virus
infected mails.
