Marking as Incomplete Wishlist as per 'maas'
** Changed in: juju
Status: New => Incomplete
** Changed in: juju
Importance: Undecided => Wishlist
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1737428
Title:
VRF support to solve routing problems associated with multi-homing
Status in juju:
Incomplete
Status in MAAS:
Incomplete
Status in linux package in Ubuntu:
Incomplete
Bug description:
Problem description:
* a host is multi-homed if it has multiple network interfaces with L3
addresses configured (physical or virtual interfaces, natural to
OpenStack regardless of IPv4/IPv6 and IPv6 in general);
(see 3.3.4 Local Multihoming
https://tools.ietf.org/html/rfc1122#page-60 and 3.3.4.2 Multihoming
Requirements)
* if all hosts that need to participate in L3 communication are
located on the same L2 network there is no need for a routing device
to be present. ARP/NDP and auto-created directly connected routes are
enough;
* multi-homing with hosts located on different L2 networks requires more
intelligent routing:
- "directly connected" routes are no longer enough to talk to all relevant
hosts in the same network space;
- a default gateway in the main routing table may not be the correct
routing device that knows where to forward traffic (management network traffic
goes to a management switch and router, other traffic goes to L3 ToR switch but
may go via different bonds);
- even if a default gateway knows where to forward traffic, it may not be
the intended physical path (storage replication traffic must go through a
specific outgoing interface, not the same interface as storage access traffic
although both interfaces are connected to the same ToR);
- there is no longer a single "default gateway" as applications need either
per-logical-direction routers or to become routers themselves (if destination
== X, forward to next-hop Y). Leaf-spine architecture is a good example of how
multiple L2 networks force you to use spaces that have VLANs in different
switch fabrics => one or more hops between hosts with interfaces associated
with the same network space;
- while network spaces implicitly require L3 reachability between each host
that has a NIC associated with a network space, the current definition does not
mention routing infrastructure required for that. For a single L2 this problem
is hidden by directly connected routes, for multi-L2, no solution is provided
or discussed;
* existing solutions to multi-homing require routing table management
on a given host: complex static routing rules, dynamic routing (e.g.
running an OSPF or BGP daemon on a host);
* using static routes is rigid and requires network planning (i.e.
working with network engineers which may have varying degrees of
experience, doing VLSM planning etc.);
* using dynamic routing requires a broader integration into an
organization's L3 network infrastructure. Routing can be implemented
differently across different organizations and it is a security and
operational burden to integrate with a company's routing
infrastructure.
Summary: a mechanism is needed to associate an interface with a
forwarding table (FIB) which has its own default gateway and make an
application with a listen(2)ing socket(2) return connected sockets
associated with different FIBs. In other words, applications need to
implicitly get source/destination-based routing capabilities without
the need to use static routing schemes or dynamic routing and with
minimum or no modifications to the applications themselves.
Goals:
* avoid turning individual hosts into routers;
* avoid complex static rules;
* better support multi-fabric deployments with minimum effort (Juju, charms,
MAAS, applications, network infrastructure);
* reduce operational complexity (custom L3 infrastructure integration for
each deployment);
* reduce delivery risks (L3 infrastructure, L3 department responsiveness
varies);
* avoid any form of L2 stretching at the infrastructure level - this is
inefficient for various reasons.
NOTE: https://cumulusnetworks.com/blog/vrf-for-linux/ - I recommend to
read this post to understand suggestions below.
How to solve it?
What does it mean for Juju to support VRF devices?
* enslave certain devices on provisioning based on network space information
(physical NICs, VLAN devices, bonds AND bridges created for containers must be
considered) - VRF devices logically enslave devices similar to bridges but work
differently (on L3, not L2);
* the above is per network namespace so it will work equally well in a LXD
container;
Conceptually:
# echo 'net.ipv4.tcp_l3mdev_accept = 1' >> /etc/sysctl.conf
# echo 'net.ipv4.udp_l3mdev_accept = 1' >> /etc/sysctl.conf
# sysctl -p
# # create additional routing tables
# cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF
1 mgmt
10 pub
20 storacc
30 storrepl
EOF
# # populate per-routing table default gateways
# ip route add mgmt default via 192.168.0.1
# ip route add pub default via 172.16.0.1
# ip route add storacc default via 10.10.4.1
# ip route add storrepl default via 10.10.5.1
# # add and bring up VRF devices
# ip link add mgmt type vrf table 1 && ip link set dev mgmt up
# ip link add pub type vrf table 10 && ip link set dev pub up
# ip link add storacc type vrf table 20 && ip link set dev storacc up
# ip link add storrepl type vrf table 30 && ip link set dev storrepl up
# # enslave actual devices to VRF devices
# ip link set mgmtbr0 master mgmt
# ip link set pubbr0 master pub
# ip link set storaccbr0 master storacc
# ip link set storreplbr0 master storrepl
# make your services use INADDR_ANY for listening sockets in charms if
not done already (use 0.0.0.0)
charm-related:
* (no-op) services with listening sockets on INADDR_ANY will not need
any modifications either on the charm side or at the application level
- this is the cheapest way to solve multi-homing problems;
* (later) a more advanced functionality for applications that do not
use INADDR_ANY but bind a listening socket to a specific address -
this requires `ip vrf exec` functionality in iproute2 or application
modifications.
Notes:
* Let's follow rule number 6 (https://tools.ietf.org/html/rfc1925) and move
routing problems to L3 departments. Juju deploy "router" is a different
scenario which should reside on a model separate from IAAS;
* We are not turning hosts into routers with this - this is a way to move
routing decisions to the next hop which is available on a directly connected
route. The problem we are solving here is N next hops instead of just one.
Those hops can worry about administrative distance/different routing protocols,
route costs/metrics, routing protocol peer authentication etc.
* Linux kernel functionality was mostly upstreamed in 4.4;
* Linux kernel only while a unit agent can run on Windows too (nothing we can
do here).
Implementation description:
1. Kernel
4.4 (GA xenial)
* CONFIG_NET_VRF=m - present in xenial GA kernels
http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/tree/debian.master/config/config.common.ubuntu?id=2c5158e82d497c5eb90d6e2b8aaf07d36cb175f6#n5172
* CONFIG_NET_L3_MASTER_DEV=y - present in xenial GA kernels
http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/tree/debian.master/config/config.common.ubuntu?id=2c5158e82d497c5eb90d6e2b8aaf07d36cb175f6#n5109
backports needed from 4.5 - required for VRF-unaware applications that
use INADDR_ANY:
6dd9a14e92e54895e143f10fef4d0b9abe109aa9 (tcp_l3mdev_accept)
63a6fff353d01da5a22b72670c434bf12fa0e3b8 (udp_l3mdev_accept)
only `ip vrf exec` related - NOT required for baseline functionality:
* http://man7.org/linux/man-pages/man8/ip-vrf.8.html CGROUPS and
CGROUP_BPF enabled - xenial HWE only (not HWE-edge)
2. User space (iproute2)
iproute2 supports the vrf keyword in a version packaged with Ubuntu
16.04.
More specific functionality like `ip vrf exec <vrf-name>` is available
in later versions:
https://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git/commit/?id=1949f82cdf62c074562f04acfbce40ada0aac7e0
git tag --contains=1949f82cdf62c074562f04acfbce40ada0aac7e0
v4.10.0
v4.11.0
...
3. MAAS - already hands over per-subnet default gateways
https://github.com/maas/maas/blob/2.3.0/src/maasserver/models/node.py#L3325-L3360
https://github.com/maas/maas/blob/2.3.0/src/maasserver/api/machines.py#L363-L378
4. Juju and/or MAAS:
* create per-network-space routing tables (default gateways must be taken
from subnets in MAAS - subnets related to the same space will have different
default gateways)
* create VRF devices relevant to network spaces;
* enslave interfaces to VRF devices (this includes Linux bridges created by
Juju for containers).
5. Nothing for baseline functionality other than configuring software
to use 0.0.0.0 (INADDR_ANY or "all interfaces") for listening sockets.
(future work) configure software to use `ip vrf exec` even if it
doesn't support VRFs directly when INADDR_ANY is not used.
See https://www.kernel.org/doc/Documentation/networking/vrf.txt, note
that setsockopt requirement is worked around via `ip vrf exec` in
iproute2 (no need to rewrite every application):
"Applications that are to work within a VRF need to bind their socket
to the VRF device:
setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);
or to specify the output device using cmsg and IP_PKTINFO.
TCP & UDP services running in the default VRF context (ie., not bound
to any VRF device) can work across ***all VRF domains*** by enabling
the tcp_l3mdev_accept and udp_l3mdev_accept sysctl options:
sysctl -w net.ipv4.tcp_l3mdev_accept=1
sysctl -w net.ipv4.udp_l3mdev_accept=1"
http://man7.org/linux/man-pages/man8/ip-vrf.8.html
"This ip-vrf command is a helper to run a command against a specific VRF with
the VRF association ***inherited parent to child***."
References:
https://en.wikipedia.org/wiki/Multihoming
http://blog.ipspace.net/2016/04/host-to-network-multihoming-kludges.html
http://blog.ipspace.net/2010/09/ribs-and-fibs.html
https://cumulusnetworks.com/blog/vrf-for-linux/ <--- this is a must-
read
https://docs.cumulusnetworks.com/display/DOCS/Virtual+Routing+and+Forwarding+-+VRF
http://netdevconf.org/1.2/session.html?david-ahern-talk
https://www.kernel.org/doc/Documentation/networking/vrf.txt
https://github.com/Mellanox/mlxsw/wiki/Virtual-Routing-and-
Forwarding-%28VRF%29
http://blog.ipspace.net/2016/02/running-bgp-on-servers.html
https://tools.ietf.org/html/rfc7938
http://www.routereflector.com/2016/11/working-with-vrf-on-linux/
(usage example on 16.04)
To manage notifications about this bug go to:
https://bugs.launchpad.net/juju/+bug/1737428/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
