Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Ruzsinszky Attila]

Hi,

Why don't you make a test with OpenVSwitch?
You can setup an SDN with it.

TIA,
Ruzsi
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Raymond Ferguson]

Maybe a VPN container that peers with the other VPN containers and runs a
dynamic routing protocol?  Then just setup a route on the lxd host to the
VPN guest.  Make it config driven so you just push a new peer list to
update. Or not and just push the updated container?

On Sep 18, 2016 4:21 AM, "Tomasz Chmielewski"  wrote:

> It's easy to create a "LAN" for LXD containers on a single LXD server -
> just attach them to the same bridge, use the same subnet (i.e.
> 10.10.10.0/24) - done. Containers can communicate with each other using
> their private IP address.
>
> However, with more then one LXD server *not* in the same LAN (i.e. two LXD
> servers in different datacentres), the things get tricky.
>
>
> Is anyone using such setups, with multiple LXD servers and containers
> being able to communicate with each other?
>
>
> LXD1: IP 1.2.3.4, EuropeLXD2: IP 2.3.4.5, Asia
> container1, 10.10.10.10 container4, 10.10.10.20
> container2, 10.10.10.11 container5, 10.10.10.21
> container3, 10.10.10.12 container6, 10.10.10.22
>
>
> LXD3: IP 3.4.5.6, US
> container7, 10.10.10.30
> container8, 10.10.10.31
> container8, 10.10.10.32
>
>
> While I can imagine setting up many OpenVPN tunnels between all LXD
> servers (LXD1-LXD2, LXD1-LXD3, LXD2-LXD3) and constantly adjusting the
> routes as containers are stopped/started/migrated, it's a bit of a
> management nightmare. And even more so if the number of LXD servers grows.
>
> Hints, discussion?
>
>
> Tomasz Chmielewski
> https://lxadm.com
> ___
> lxc-users mailing list
> lxc-users@lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[brian mullan]

Serge,


>
>
>
> *> Automatically builds tunnels through firewalls and NATs without any
> further> setup (for example, port forwarding).I would not appreciate
> something which "automatically" (whatever itmeans) traverse my firewalls,
> to be honest. We should treat our dataseriously, Brian.*


First, a sysadmin person has to install/setup/configure PeerVPN on each
server so I guess like installing/configuring TINC or any other VPN
solution there is some assumption of some sort of "trust" in that person's
work.

Second, in PeerVPN's configuration file
 on each
server  you (re the sysadmin) have to
specify 2 security related items:

PORT x# the Port to be opened/used by PeerVPN

But you point out a good question -
regarding that bullet by the author on the PeerVPN web page.   Tobias Volk
may be referring to something  else as *it CLEARLY states in the short
PeerVPN tutorial  you MUST port-forward the
"port" configured for PeerVPN to use if Nodes are behind a NAT.   *

*I know PeerVPN doesn't work if you have not done that from my own use.:*


*Configuration of node A*


> Create the peervpn.conf of Node A with the following content:



port 7000
> networkname ExampleNet
> psk mysecretpassword
> enabletunneling yes
> interface peervpn0
> ifconfig4 10.8.0.1/24



This will open UDP port 7000 and create a virtual ethernet interface with
> the name peervpn0 and the IP address 10.8.0.1.



Please note that Node A needs to be directly reachable from Node B.
> *If Node A is behind a NAT device, you will have to forward port 7000. *



*Configuration of node B*


> Create the peervpn.conf of Node B with the following content:



port 7000
> networkname ExampleNet
> psk mysecretpassword
> enabletunneling yes
> interface peervpn0
> ifconfig4 10.8.0.2/24
> initpeers node-a.example.com 7000



Replace node-a.example.com with the real address of Node A.

Further there is the shared PSK crypto key generation that also limits
connections to "peers" sharing the "same" PSK "seed" in the configuration
file.

In a PeerVPN mesh different server/hosts can have multiple PSK "seed"
configured to allow any 1 host to "peer" with different specific systems in
the "mesh" who have a matching PSK "seed" configured.
I can email Tobias and ask for clarification as to what "bullet" means.

Brian
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Tomasz Chmielewski]

On 2016-09-19 05:12, Tilak Waelde wrote:

Hope this helps.  Happy to share my LXD configurations with anyone...


Please do! I'd really love to see a description of a production lxd /
lxc setup with proper networking and multiple hosts!

I haven't played around with it yet, but is it possible to include
some sort of VRF-lite[0] into such a setup for multi tenancy purposes?
Other than by using VLANs one could use the same IP ranges multiple
times from what I've come to understand?
I'm not sure how a user could put the containers interfaces into a
different network namespace..


Hi,

after some experimenting with VXLAN, I've summed up a working "LAN for 
multiple LXC servers" here:


https://lxadm.com/Unicast_VXLAN:_overlay_network_for_multiple_servers_with_dozens_of_containers


It is using in kernel VXLAN, and thus performs very well (almost wire 
speed, and much much better than any userspace programs).


On the other hand, it provides no encryption between LXD servers (or, in 
fact, any other virtualization), so may depend on your exact 
requirements.



Tomasz Chmielewski
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Sergiusz Pawlowicz]

On Mon, Sep 19, 2016 at 8:48 PM, brian mullan  wrote:

> Automatically builds tunnels through firewalls and NATs without any further
> setup (for example, port forwarding).

I would not appreciate something which "automatically" (whatever it
means) traverse my firewalls, to be honest. We should treat our data
seriously, Brian.

Serge
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[brian mullan]

Tomasz

After experimenting with TINC, OpenVPN & others I found and have used
PeerVPN to do this:

https://peervpn.net/

PeerVPN is a full-mesh VPN and is auto-learning..

   - Ethernet tunneling support using TAP devices.
   - IPv6 support.
   - Full mesh network topology.
   - Automatically builds tunnels through firewalls and NATs without any
   further setup (for example, port forwarding).
   - Shared key encryption and authentication support.

At the time *I found PeerVPN much simpler to configure that TINC !*

A PeerVPN mesh is extremely easy to configure as you only have to install
PeerVPN on each host server and edit its config file on that server.

The brief tutorial page (https://peervpn.net/tutorial/) shows that config
for a basic PeerVPN only requires about 7 lines on each of your servers.

In Apr 2015 I documented a proof-of-concept test setup

I'd done to interconnect LXC containers on AWS, DIgital Ocean and here
locally (so much like you want to do I think).

*NOTE:  * that documentation *was before* my use of LXD !

*The reason I mention this is that with the advent of LXD the entire
> container IP addressing setup using PeerVPN becomes even easier ! *   This
> is because when I did those tests I never took the time to create a single
> DHCP source for all lxc containers on all hosts.   In my 2015
> proof-of-concept test of PeerVPN I just a quick & dirty config using the
> assumption that no 2 containers (for my test purposes) would get assigned
> the same 10.0.3.x IP address by their local Server/Host.


Today, LXD *during installation* allows you much more control over each
Server/Host's LXDBR0 bridge IP and the 10.x.x.x IP address range that is
used for that Host's lxc containers.

*So if you were to use PeerVPN,  during LXD installation on each server
just reserve a portion of the 10.x.x.x IP address range for each
Server/Host to avoid any possibility of any 2 containers on any 2 Hosts
getting assigned the same IP.*

The last config step on each Server/Host is to connect the PeerVPN mesh
Tunnel End Point (TEP to the local LXDBR0 bridge to enable all the
containers attached to that bridge to pass data over the VPN tunnel to the
other servers since my documentation set up a Layer 2 VPN (note PeerVPN
supports a Layer 3 tunnel as well I believe).

Since the PeerVPN TEP "interface" ... called "peervpn0" in my tutorial
example is just like any other Linux ethernet interface you only have to
use the "ip link" command to connect "peervpn0" to the LXD "lxdbr0" bridge.

*$ sudo ip link set dev peervpn0 master lxdbr0*

After you've done that on all 3 hosts your containers on all the hosts
should be able to reach each other since they would all be on what
logically is the same "ethernet" via the VPN & the fact that you setup all
of the LXD installs with the same same 10.x.x.x network.

Also, the containers on each Server/Host still get external Internet access
(if you permit it) via their local Host.

Anyway, I thought I'd send this info in case it is helpful as this should
work with servers locally or remotely using LXD.

Brian
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Tilak Waelde]

Hope this helps.  Happy to share my LXD configurations with anyone...

-Ron


Please do! I'd really love to see a description of a production lxd / 
lxc setup with proper networking and multiple hosts!


I haven't played around with it yet, but is it possible to include some 
sort of VRF-lite[0] into such a setup for multi tenancy purposes? Other 
than by using VLANs one could use the same IP ranges multiple times from 
what I've come to understand?
I'm not sure how a user could put the containers interfaces into a 
different network namespace..


cheers,
Tilak

[0] : https://www.kernel.org/doc/Documentation/networking/vrf.txt
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Sergiusz Pawlowicz]

On Sun, Sep 18, 2016 at 7:11 PM, Tomasz Chmielewski  wrote:
> On 2016-09-18 21:05, Sergiusz Pawlowicz wrote:
>>
>> On Sun, Sep 18, 2016 at 4:16 PM, Tomasz Chmielewski 
>> wrote:
>>
>>> While I can imagine setting up many OpenVPN tunnels between all LXD
>>> servers
>>
>>
>> I cannot imagine that :-) :-)
>>
>> Use tinc, mate. Your life begins :-)
>>
>> https://www.tinc-vpn.org/
>
> I did some reading about tinc before, and according to documentation and
> mailing lists:
>
> - performance may not be so great
>
> - it gets problematic as the number of tinc instances grows (few will be OK,
> dozens will work, but beyond that, the things might get slowish)
>
> - if I'm not mistaken, you need to run a tinc instance per LXD client, not
> per LXD server, so that's extra management and performance overhead (i.e. if
> two tinc clients are running on the same server, they would still encrypt
> the traffic to each other)

I think you must read more, because everything you wrote is simply false.

Or, maybe, try it? :-) :-) :-)

cheers!
S.
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Ron Kelley]

So, just for clarity, you are saying each LXD server will have no separate 
network connection for the containers.  Thus, all containers are private to the 
LXD server, and any outbound traffic must traverse the container server 
interface.  Is this correct?  If so, sorry, I must have missed this requirement 
in your initial email.



On Sep 18, 2016, at 9:41 AM, Tomasz Chmielewski  wrote:

On 2016-09-18 22:14, Ron Kelley wrote:
> (Long reply follows…)
> Personally, I think you need to look at the big picture for such
> deployments.  From what I read below, you are asking, “how do I extend
> my layer-2 subnets between data centers such that container1 in Europe
> can talk with container6 in Asia, etc”.  If this is true, I think you
> need to look at deploying data center hardware (servers with multiple
> NICs, IPMI/DRAC/iLO interfaces) with proper L2/L3 routing (L2TP/IPSEC,
> etc).  And, you must look at how your failover services will work in
> this design.  It’s easy to get a couple of servers working with a
> simple design, but those simple designs tend to go to production very
> fast without proper testing and design.

Well, it's not only about deploying on "different continents".

It can be also in the same datacentre, where the hosting doesn't give you a LAN 
option.

For example - Amazon AWS, same region, same availability zone.

The servers will have "private" addresses like 10.x.x.x, traffic there will be 
private to your servers, but there will be no LAN. You can't assign your own 
LAN addresses (10.x.x.x).

This means, while you can launch several LXD containers on every of these 
servers - but their "LAN" will be limited per each LXD server (unless we do 
some special tricks).

Some other hostings offer a public IP, or several public IPs per servers, in 
the same datacentre, but again, no LAN.


Tomasz Chmielewski
https://lxadm.com

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Tomasz Chmielewski]

On 2016-09-18 22:14, Ron Kelley wrote:

(Long reply follows…)

Personally, I think you need to look at the big picture for such
deployments.  From what I read below, you are asking, “how do I extend
my layer-2 subnets between data centers such that container1 in Europe
can talk with container6 in Asia, etc”.  If this is true, I think you
need to look at deploying data center hardware (servers with multiple
NICs, IPMI/DRAC/iLO interfaces) with proper L2/L3 routing (L2TP/IPSEC,
etc).  And, you must look at how your failover services will work in
this design.  It’s easy to get a couple of servers working with a
simple design, but those simple designs tend to go to production very
fast without proper testing and design.


Well, it's not only about deploying on "different continents".

It can be also in the same datacentre, where the hosting doesn't give 
you a LAN option.


For example - Amazon AWS, same region, same availability zone.

The servers will have "private" addresses like 10.x.x.x, traffic there 
will be private to your servers, but there will be no LAN. You can't 
assign your own LAN addresses (10.x.x.x).


This means, while you can launch several LXD containers on every of 
these servers - but their "LAN" will be limited per each LXD server 
(unless we do some special tricks).


Some other hostings offer a public IP, or several public IPs per 
servers, in the same datacentre, but again, no LAN.



Tomasz Chmielewski
https://lxadm.com
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Ron Kelley]

(Long reply follows…)

Personally, I think you need to look at the big picture for such deployments.  
From what I read below, you are asking, “how do I extend my layer-2 subnets 
between data centers such that container1 in Europe can talk with container6 in 
Asia, etc”.  If this is true, I think you need to look at deploying data center 
hardware (servers with multiple NICs, IPMI/DRAC/iLO interfaces) with proper 
L2/L3 routing (L2TP/IPSEC, etc).  And, you must look at how your failover 
services will work in this design.  It’s easy to get a couple of servers 
working with a simple design, but those simple designs tend to go to production 
very fast without proper testing and design.


All that said, here is one way I would tackle this type of request:

* Get servers with at least 3 NICs (preferably 5)
  * One iLO/DRAC/IPMI interface for out-of-band management
  * One for Container server management (ie: LXD1 IP 1.2.3.4) - use a second 
NIC for redundancy in a bonded configuration
  * One for Container hosting network (ie container1, container2, etc) - use a 
second NIC for redundancy and VLANs to separate traffic

* Get firewalls in each location with L2TP/IPSEC support (pfSense works great)
  * Extend your L2 networks between your sites with L2TP
  * Secure the connection with IPSSEC

* On your LXD servers, create 2 bonded NICs
  * One for container management (eth0, eth1)
  * One for hosting network (eth2, eth3)
  * Use VLANs on hosting network to separate traffic
  * Configure your containers with the appropriate VLAN tag (ie: 501)

Once the above is done, your containers can talk w/each other in different 
locations.  You can use firewall rules to allow/deny IP connections from your 
container VMs.  You can extend both your container management and hosting 
networks across the L2 tunnel allowing you to move VMs at will.  


General Notes:
---
* For server bonded connections, I use linux mode type 6;  works well, provides 
great throughput, requires no special configuration on directly-connected 
switches.
* On the LXD side, create multiple profiles with VLAN configurations.  
Personally, I have 2 profiles: one for VLAN 501 and one for VLAN 502.  Local 
firewall provides security between container networks.
* Be mindful of the services you share across the tunnels.  Things like iSCSI, 
NFS, etc will kill your network performance because of the chatty type of 
traffic.


Some good references:
---
https://doc.pfsense.org/index.php/L2TP/IPsec
http://archive.openflow.org/wk/index.php/Tunneling_-_GRE/L2TP
http://www.networkworld.com/article/2163334/tech-primers/what-can-l2tp-do-for-your-network-.html

Caution: L2 networks have a lot of broadcast traffic.  If your site-to-site 
connections are slow, your entire extended L2 network will suffer.  Must find a 
way to suppress L2 broadcast/multicast between sites.


Hope this helps.  Happy to share my LXD configurations with anyone...

-Ron






On Sep 18, 2016, at 5:16 AM, Tomasz Chmielewski  wrote:

It's easy to create a "LAN" for LXD containers on a single LXD server - just 
attach them to the same bridge, use the same subnet (i.e. 10.10.10.0/24) - 
done. Containers can communicate with each other using their private IP address.

However, with more then one LXD server *not* in the same LAN (i.e. two LXD 
servers in different datacentres), the things get tricky.


Is anyone using such setups, with multiple LXD servers and containers being 
able to communicate with each other?


LXD1: IP 1.2.3.4, EuropeLXD2: IP 2.3.4.5, Asia
container1, 10.10.10.10 container4, 10.10.10.20
container2, 10.10.10.11 container5, 10.10.10.21
container3, 10.10.10.12 container6, 10.10.10.22


LXD3: IP 3.4.5.6, US
container7, 10.10.10.30
container8, 10.10.10.31
container8, 10.10.10.32


While I can imagine setting up many OpenVPN tunnels between all LXD servers 
(LXD1-LXD2, LXD1-LXD3, LXD2-LXD3) and constantly adjusting the routes as 
containers are stopped/started/migrated, it's a bit of a management nightmare. 
And even more so if the number of LXD servers grows.

Hints, discussion?


Tomasz Chmielewski
https://lxadm.com
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Tomasz Chmielewski]

On 2016-09-18 21:05, Sergiusz Pawlowicz wrote:
On Sun, Sep 18, 2016 at 4:16 PM, Tomasz Chmielewski  
wrote:


While I can imagine setting up many OpenVPN tunnels between all LXD 
servers


I cannot imagine that :-) :-)

Use tinc, mate. Your life begins :-)

https://www.tinc-vpn.org/


I did some reading about tinc before, and according to documentation and 
mailing lists:


- performance may not be so great

- it gets problematic as the number of tinc instances grows (few will be 
OK, dozens will work, but beyond that, the things might get slowish)


- if I'm not mistaken, you need to run a tinc instance per LXD client, 
not per LXD server, so that's extra management and performance overhead 
(i.e. if two tinc clients are running on the same server, they would 
still encrypt the traffic to each other)



Tomasz Chmielewski
https://lxadm.com
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Sergiusz Pawlowicz]

On Sun, Sep 18, 2016 at 4:16 PM, Tomasz Chmielewski  wrote:

> While I can imagine setting up many OpenVPN tunnels between all LXD servers

I cannot imagine that :-) :-)

Use tinc, mate. Your life begins :-)

https://www.tinc-vpn.org/

cheers,
Serge
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Micky Del Favero]

Tomasz Chmielewski  writes:

> While I can imagine setting up many OpenVPN tunnels between all LXD
> servers (LXD1-LXD2, LXD1-LXD3, LXD2-LXD3) and constantly adjusting the
> routes as containers are stopped/started/migrated, it's a bit of a
> management nightmare. And even more so if the number of LXD servers
> grows.
>
> Hints, discussion?

If you use the same subnet for the container as in your example
10.10.10.0/24 there's no routing nightmare, you have only to setup
openvpn in bridge mode and so you'll need only 2 tunnels: LXD1-LXD2 and
LXD2-LXD3, obviously if LXD2 will go offline you'll have a problem.

If you'll have many containers using the same subnet for all will be
impossible so you'll need to managed routing between hosts to be able to
reach every container.

I'm about to design and deploy similar setup: many LXD hosts running
containers that can be started on a hosts and eventually migrated to
another host in the same or in a different datacenter.

I'm thinking not to use the same subnet for all containers but to use a
different subnet for every host, so following your example it will be
like the following:

LXD1: IP 1.2.3.4, EuropeLXD2: IP 2.3.4.5, Asia
container1, 10.10.10.10 container4, 10.10.20.10
container2, 10.10.10.11 container5, 10.10.20.11
container3, 10.10.10.12 container6, 10.10.20.12

LXD3: IP 3.4.5.6, US
container7, 10.10.30.10
container8, 10.10.30.11
container9, 10.10.30.12

on every hosts a dhcp server is use to dynamically configure network on
starting container and, based on container's hostname, to dynamically
update dns servers so containers will be always reachable via hostname.

Routing problem remains, I'll probably choose to deploy some tunnels
between hosts to connect each other, then running an OSPF daemon for
dynamic routing management.

Ciao, Micky
-- 
The sysadmin has all the answers, expecially "No"
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users