We have been using Vyatta (and then Vyos) after the fork for all of our
production routers for over four years now. It is rock solid and
performs better than the Juniper routers we replaced them with. It is
debian based underneath its custom shell whose syntax is very much like
Juniper. A sudo su gets you a standard bash shell just like in any other
debian distro and I have installed additional packages in the past. It
is really easy to configure in that the entire router config is held in
a single file with a dead simple file structure. You can use the
commands in their custom shell to alter the config but I prefer to just
upload a mod to that file... I know, I like to live dangerously! lol.
Seriously though, they do have versioning/rollback capability but last I
checked a rollback forced a reboot of the router.
The major downside, and one we are struggling with now is that it is
running an EOL version of debian and that is a HUGE liability for
something as security critical as an edge router. They have a lot of
work ahead of them to get out 2.0 but aside from the EOL Debian, Vyos
"just works" and out of the box would exceed the current VR in pretty
much every way. Personally, I find the API issue to be less important.
#1 priority has to be getting a VR that functions at an enterprise
level. API integration is important but pales compared to that.
Conversely, we are still not in production with Cloudstack specifically
because of the current state of the VR. IOW, the current VR
implementation is directly affecting Cloudstack adoption at least
anecdotally.
My team is very new to contributing to CS and our time is limited but,
given my many years of experience administering networks with Vyos at
the core, if you go that route I will gladly lend my knowledge in
helping you map out an implementation.
I have evaluated Cloud Router in the past and found it lacking in basic
features (and especially documentation) that made it a no go for even a
test build. It is promising and I hope it continues to mature but Vyos
is a battle hardened and proven technology. Given the option, I always
go with the known good solution over the shiny one.
I have also used PFsense. Really great project but I have not found it
to be as stable for me as Vyos, though that might be my lack of BSD
experience.
Thanks,
Matthew Smart
President
Smart Software Solutions Inc.
108 S Pierre St.
Pierre, SD 57501
Phone: (605) 280-0383
Skype: msmart13
Email: msm...@smartsoftwareinc.com
On 09/18/2016 11:09 AM, Will Stevens wrote:
At this point in the discussion, I don't think we should rule anything out.
I think it makes sense to explore all the options and then isolate some
front runners in terms of software and architecture.
On Sep 18, 2016 1:08 AM, "ilya" <ilya.mailing.li...@gmail.com> wrote:
Our options become much better if we consider BSD based routers.
Would that be on the table?
https://en.wikipedia.org/wiki/List_of_router_and_firewall_distributions
On 9/16/16 12:04 PM, Will Stevens wrote:
Ya, your points are all valid Simon. The lack of standard libraries to
handle a lot of the details is a problem. I don't think it is an
unsolvable problem, but if we spend the time to do that, will we have
something that will work for us for the next 5 years? This may be the
shortest path to getting us where we need to be for the time being.
What is the best case scenario for the VR going forward which will last
us
the next 5 years? Maybe we just clean up what we have to do a major
restructuring of the pieces and how they are implemented. We need to
keep
in mind how maintainable this implementation is because that is going to
be
key going forward IMO.
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_
On Fri, Sep 16, 2016 at 2:29 PM, Simon Weller <swel...@ena.com> wrote:
I think our other option is to take a real look at what it would take to
fix the VR. In my opinion, a lot of the problems are related to the
monolithic python code base and the fact nothing is actually separated.
Secondly, the python scripts (and bash scripts) don't use any
established
libraries to complete tasks and instead shell out and run commands that
are
both hard to track and hard to parse on return.
If we daemonized this, used a real api for Agent to VR communication,
used
common already existing libraries for the system service and network
interactions and spent a bit of time separating out code into distinct
modules, everything would behave a lot better.
The pain and suffering is due to years and years of patches and constant
shelling out to complete tasks in my opinion. If we spend time to
rethink
how we interact with the VR in general and we abstract the systems and
networking stuff and use well known and stable libraries to do the work,
the VR would be much easier to maintain.
- Si
________________________________
From: Marty Godsey <ma...@gonsource.com>
Sent: Friday, September 16, 2016 12:24 PM
To: dev@cloudstack.apache.org
Subject: RE: [DISCUSS] Replacing the VR
So based upon this discussion would it be prudent to wait on VyOS 2.0?
The
current VR is giving us issues but would the time invested in another
"solution" be wasted especially if by the time another option is chose,
then coded, then tested, then implemented and right as that time
happened
to be when VyOS 2.0 is released. Of course you said they are just in
the
scoping range so this could still be a year or more out.
Thoughts?
Regards,
Marty Godsey
nSource Solutions
-----Original Message-----
From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On
Behalf Of Will Stevens
Sent: Friday, September 16, 2016 10:31 AM
To: dev@cloudstack.apache.org
Cc: dan...@baturin.org
Subject: Re: [DISCUSS] Replacing the VR
I just had a quick chat with a couple of the guys over on the VyOS chat.
I have CC'ed one of them in case we have more licensing questions.
So here is the status with the license "the code inherited from Vyatta
and
our modifications from it is GPLv2 (strict, not v2+). The config reading
library is GPLv2 too, so anything that links to is is GPLv2.
Some auxiliary components we made after the fork are more permissive,
LGPLv2+ or MIT."
They are currently in the process of scoping a redesign (VyOS 2.0), "we
are planning a clean rewrite that will solve issues of the current
config
system".
This will include the ability to configure via the API.
If we have more questions for VyOS, they are very friendly and
responsive,
so we should be able to get answers.
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw
@CloudOps_
On Fri, Sep 16, 2016 at 9:37 AM, Syed Ahmed <sah...@cloudops.com>
wrote:
I agree with Will Ilya. There are so many problems with the VR right
now.
Most of the outages we've had recently have somehow involved the VR.
We set custom iptables rules on the VR which can and have easily gone
wrong.
Openswan is broken, Strongswan replacement still needs to be tested.
VVRP with redundant router still needs work, and not to mention the
problems we will have when we introduce IPv6 into the whole picture.
I think the spirit of the discussion is to rely on a 3rd party to do
the networking for us (eg VyOS) and have us handle just the
orchestration. All the problems that I've described have already been
solved in VyOS. We also get the advantage of a potential wider
community to fix and maintain the VR and given our current development
velocity, it think it totally makes sense to look for a 3rd party
option.
-Syed
On Fri, Sep 16, 2016 at 9:18 AM, Will Stevens <wstev...@cloudops.com>
wrote:
The VR has been biting us far too often recently, which is why we
have started looking into alternative implementations.
One of the things that is nice about potentially using the VyOS is
that
it
is based on Debian, so we should be able to run the other services
that
we
currently have like the password server and userdata on the VyOS.
This means we would not have to change our architecture initially
and could focus on only replacing the networking paths.
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|*
tw @CloudOps_
On Fri, Sep 16, 2016 at 6:20 AM, Nux! <n...@li.nux.ro> wrote:
The more this is discussed the more I think we should stick with
our
VR.
All these other options either seem unfinished or with
incompatible license.
VyOS looks the most promising so far, it's a serious, mature project.
Adopting it though means we'll have to microservice our way out of
it
with
extra machines for DNS/USERDATA/etc, unless we can make VyOS serve
those
too. Imho this adds complexity we should void.
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
From: "Will Stevens" <wstev...@cloudops.com>
To: dev@cloudstack.apache.org
Sent: Thursday, 15 September, 2016 17:21:28
Subject: Re: [DISCUSS] Replacing the VR
Ya, we would need to add a daemon for VPN as well. Load
balancing is another aspect which we will need to consider if we
went this route.
Something like https://traefik.io/ could potentially be a good
fit
due
to
its API driven configuration, but it may be more than what we need.
We should probably try define which pieces make sense to be
solved
together
and which pieces would be best suited to be broken out.
I think the network connectivity, routing and firewalling should
probably
all stay together since the majority of the tools we would
potentially
use
would handle all of that together in a single implementation.
The password server and userdata seems like a good option for
being
broken
out and handled independently (and probably rewritten completely
since
they
currently have some issues).
Load balancing is another that could warrant splitting out, but
that depends on what direction we go and how we would be managing
it.
DHCP
and
DNS are others which could go either way.
If we do split out services, I think we should consolidate as
much as
we
can into each service we break out. Ideally a network packet
would
never
hit more than one, maybe two, services. I don't think we should
be splitting services 'just because', I think we need a valid
case for splitting any service out because it adds complexity.
Our project is already complex enough, we need to avoid adding
complexity unless it
is
really needed.
Some more of my thoughts on this anyway...
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com
*|* tw @CloudOps_
On Thu, Sep 15, 2016 at 10:28 AM, Simon Weller <swel...@ena.com>
wrote:
I do agree with you that this probably isn't the right place
the
password
service and user data.
Having said that, after taking a cursory look at the dev docs,
it
doesn't
seem that difficult to add new daemons:
https://opensnaproute.github.
io/docs/developer.html#creating-new-component
<https://opensnaproute.github.io/docs/developer.html#
creating-new-component>
They've definitely build it with a microservices architecture
in
mind,
so
each individual feature is abstracted into it's own small
daemon
process.
We could just create a daemon for the password server and the
userdata
components if we really had to.
- Si
________________________________
From: williamstev...@gmail.com <williamstev...@gmail.com> on
behalf
of
Will Stevens <wstev...@cloudops.com>
Sent: Thursday, September 15, 2016 9:17 AM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Replacing the VR
A big part of why I know about it is because it is written in Go.
:P
Yes, it is definitely interesting for the routing and traffic
handling
aspects of the VR. We will likely have to rethink some of the
pieces
a
little bit like the password server and userdata if we are to
adopt
a
different VR approach. This is where I think some of JohnB and
Chiradeep's
ideas make sense. In many ways, it does not make sense for the
device
handling routing and network traffic to also be responsible for
passwords
and userdata.
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com
*|* tw @CloudOps_
On Thu, Sep 15, 2016 at 9:10 AM, Simon Weller <swel...@ena.com>
wrote:
I hadn't heard of Flexswitch until you mentioned it. It looks
pretty
cool!
It even supports ONIE install.
To be honest, the ipsec feature could be added, or we could
offload
it to
separate vm if we needed to. The fact it is so feature rich
from a
routing
perspective (and all API driven) is really nice.
Based on the roadmap, it looks like they plan to also support
capabilities
such as BGP-MPLS based L3VPN, EVPN, VPLS in the future. This
will
be
huge
for our carrier community that rely on these technologies to
do
private
gateway and inter-VPC interconnections today. We handle this
stuff
on
our
ASRs right now with a vlan interconnect into the VR. Being
able to
do
MPLS
all the way to the VR would be awesome.
It also seems to be written in GO (a language here at ENA we
know
very
well).
- Si
________________________________
From: Will Stevens <williamstev...@gmail.com>
Sent: Thursday, September 15, 2016 7:06 AM
To: dev@cloudstack.apache.org
Subject: RE: [DISCUSS] Replacing the VR
Ya. I don't think it covers our whole use case, but what it
does
cover is
all api driven...
On Sep 15, 2016 1:48 AM, "Marty Godsey" <ma...@gonsource.com>
wrote:
Though I don’t see VPN in Snaproute.. Makes sense since it
was
not
intended to do IPSec.
It seems as though VyOS is starting to look like the best
option.
Regards,
Marty Godsey
nSource Solutions
-----Original Message-----
From: williamstev...@gmail.com
[mailto:williamstev...@gmail.com
]
On
Behalf Of Will Stevens
Sent: Wednesday, September 14, 2016 11:06 PM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Replacing the VR
Or we could go completely crazy and go with something like
FlexSwitch
from
SnapRoute
- http://www.snaproute.com/
- https://opensnaproute.github.io/docs/apis.html
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
cloudops.com
*|*
tw
@CloudOps_
On Wed, Sep 14, 2016 at 10:55 PM, Will Stevens <
wstev...@cloudops.com>
wrote:
I tend to agree with Syed and Marty. I am not sure what
problems
are
solved by splitting up the function of the VR into a
bunch of
separate
services. As Syed points out, the complexity added is
non-trivial.
We now have to manage all the intercontainer networking
as
well
as
the
orchestrated ACS networking.
VyOS is interesting to me because it covers the majority
of
our
use
case with a single unified control plane. It also has
good
support
for extending features we care about, like IPv6, VXLAN,
VRRP, transactions, etc...
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
cloudops.com
*|*
tw
@CloudOps_
On Wed, Sep 14, 2016 at 9:49 PM, Syed Ahmed <
sah...@cloudops.com>
wrote:
Agree with Marty, adding Docker containers to the
picture
although
can make the VR more flexible but the added complexity
is
just
not
worth it. Not to mention we would need to take care of
networking
each container manually and given that our iptable rules
are
very
unstable at the moment I don't see a big value add.
Vyos looks like a better solution to me. I know that it
does
not
provide an api but it does fit the bill quite well
otherwise. I
specially like the fact that it has a transaction based
model
and
you
can rollback changes if something goes wrong.
On Wed, Sep 14, 2016 at 9:06 PM Marty Godsey <
ma...@gonsource.com>
wrote:
Licensing aside, I think splitting the various
functions
into
containers is not a good route either. This will force
users
to
have to maintain
and
use containers and adds complexity to the networking
aspects
of
ACS.
Complexity decreases stability. Now I understand the
argument
that
a monolithic approach also brings its own set of
issues but
it
also
simplifies it.
Regards,
Marty Godsey
nSource Solutions
-----Original Message-----
From: Chiradeep Vittal [mailto:chirade...@gmail.com]
Sent: Wednesday, September 14, 2016 5:37 PM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Replacing the VR
I rather doubt that the Cloudrouter will fit the needs
of
the
CloudStack project
- it is AGPL licensed. Many enterprises will not
touch
anything
that
has
AGPL
- the github repo shows rather infrequent updates.
Quite
likely
they aren't considering the use cases of the
CloudStack
community
I'd back John B's comments on disaggregating the VR.
Split
it
into
many docker containers
- password server
- userdata server
- DHCP / DNS
- s2s VPN
- RA VPN
- intra-VPC routing and ACL
- Port forwarding + NAT
- FW
- LB (public)
- LB (internal),
- secondary storage
- agent
Glue them together with docker compose files (one per
use
case -
basic zone, isolated, VPC, SSVM, etc).
The VR image then becomes a JeOS + docker. You can
test
each
of
the
components independently and fixing one bug in the
field
(say
DHCP)
is hitless to the other components. You don't need to
build per-hypervisor VRs. You could even run on
baremetal.
Along the way you need to figure out how to
- make the traffic traverse the containers that are
needed
to
be
traversed (in most cases just 1)
- bootstrap the router (how does it find its compose
file?
where
is the
registry?)
- rethink the command and control of the VR
functions. SSH
works,
but something more declarative, idempotent should be
explored.
As you do this, it becomes clearer which of the
functions
can
be
substituted by for example CloudRouter. Command and
Control
of
the
docker
containers can be moved out to another container. Etc.
On Wed, Sep 14, 2016 at 12:59 AM, Marty Godsey
<ma...@gonsource.com>
wrote:
This one does look nice. My biggest concern is the
lack
of
VXLANs. It seems that any of the ones we mentioned
do not
have
an
API so we may be stuck at the SSH method.
Regards,
Marty Godsey
nSource Solutions
-----Original Message-----
From: Abhinandan Prateek
[mailto:abhinandan.prat...@shapeblue.com]
Sent: Wednesday, September 14, 2016 2:26 AM
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Replacing the VR
Cloudrouter looks promising. These have potential to
save
future
engineering effort for example on ipv6 routing, OSPF
etc.
And the best part is they come with test automation
framework.
On 13/09/16, 4:22 PM, "Jayapal Uradi"
<jayapal.ur...@accelerite.com>
wrote:
Hi,
Instead of replacing the VR in first place we
should add VyOS/cloudrouter
as provider. Once it is stable, network offerings
(on
upgrade)
can be updated to use it and we can drop the VR if
we
want
at
that release
onwards.
VR is stabilized over a period of time and some of
them
are
running
without issues. When we replicate the ACS VR
features in
new
solution it takes some to find the missing pieces
(hidden
bugs).
Thanks,
Jayapal
On Sep 13, 2016, at 2:52 PM, Nux! <
n...@li.nux.ro> wrote:
Hi,
I like the idea.
Cloudrouter looks really promising, I'm not too
keen
on
VyOS
(it
doesn't have a proper http api etc).
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
abhinandan.prat...@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
----- Original Message -----
From: "Will Stevens" <williamstev...@gmail.com>
To: dev@cloudstack.apache.org
Sent: Monday, 12 September, 2016 21:20:11
Subject: [DISCUSS] Replacing the VR
*Disclaimer:* This is a thought experiment and
should
be
treated as
such.
Please weigh in with the good and bad of this
idea...
A couple of us have been discussing the idea of
potentially
replacing the ACS VR with the VyOS [1] (Open
Source
Vyatta
VM).
There may be a license issue because I think it
is
licensed
under GPL, but for the sake of discussion, let's
assume
we
can overcome any
license issues.
I have spent some time recently with the VyOS
and I
have
to
admit, I was pretty impressed. It is simple and
intuitive
and it gives you a lot more options for auditing
the
configuration etc...
Items of potential interest:
- Clean up our current VR script spaghetti to a
simpler
more
auditable configuration workflow.
- Gives a cleaner path for IPv6 support.
- Handles VPN configuration via the same
configuration
interface.
- Support for OSPF & BGP.
- VPN support through OpenVPN & StrongSwan.
- Easily supports HA (redundant routers) through
VRRP.
- VXLAN support.
- Transaction based changes to the VR with
rollback
on
error.
Items that could be difficult to solve:
- Userdata password reset workflow and
implementation.
- Upgrade process.
The VyOS is not the only option if we were to
consider
this
approach.
Another option, which I don't know as well,
would be CloudRouter (AGPL
license) [2] which is purely API driven.
Anyway, would love to hear your thoughts...
Will
[1] https://vyos.io/ [2]
https://cloudrouter.org/
DISCLAIMER
==========
This e-mail may contain privileged and confidential
information
which is
the property of Accelerite, a Persistent Systems
business.
It is
intended only for the use of the individual or
entity to
which
it
is addressed. If you are not the intended recipient,
you
are
not
authorized to read, retain, copy, print, distribute
or
use
this
message. If you have received this communication in
error,
please
notify the sender and delete all copies of this
message.
Accelerite, a Persistent Systems business does not
accept
any
liability for virus
infected mails.
