(Long reply follows…)

Personally, I think you need to look at the big picture for such deployments.  
From what I read below, you are asking, “how do I extend my layer-2 subnets 
between data centers such that container1 in Europe can talk with container6 in 
Asia, etc”.  If this is true, I think you need to look at deploying data center 
hardware (servers with multiple NICs, IPMI/DRAC/iLO interfaces) with proper 
L2/L3 routing (L2TP/IPSEC, etc).  And, you must look at how your failover 
services will work in this design.  It’s easy to get a couple of servers 
working with a simple design, but those simple designs tend to go to production 
very fast without proper testing and design.

All that said, here is one way I would tackle this type of request:

* Get servers with at least 3 NICs (preferably 5)
  * One iLO/DRAC/IPMI interface for out-of-band management
  * One for Container server management (ie: LXD1 IP - use a second 
NIC for redundancy in a bonded configuration
  * One for Container hosting network (ie container1, container2, etc) - use a 
second NIC for redundancy and VLANs to separate traffic

* Get firewalls in each location with L2TP/IPSEC support (pfSense works great)
  * Extend your L2 networks between your sites with L2TP
  * Secure the connection with IPSSEC

* On your LXD servers, create 2 bonded NICs
  * One for container management (eth0, eth1)
  * One for hosting network (eth2, eth3)
  * Use VLANs on hosting network to separate traffic
  * Configure your containers with the appropriate VLAN tag (ie: 501)

Once the above is done, your containers can talk w/each other in different 
locations.  You can use firewall rules to allow/deny IP connections from your 
container VMs.  You can extend both your container management and hosting 
networks across the L2 tunnel allowing you to move VMs at will.  

General Notes:
* For server bonded connections, I use linux mode type 6;  works well, provides 
great throughput, requires no special configuration on directly-connected 
* On the LXD side, create multiple profiles with VLAN configurations.  
Personally, I have 2 profiles: one for VLAN 501 and one for VLAN 502.  Local 
firewall provides security between container networks.
* Be mindful of the services you share across the tunnels.  Things like iSCSI, 
NFS, etc will kill your network performance because of the chatty type of 

Some good references:

Caution: L2 networks have a lot of broadcast traffic.  If your site-to-site 
connections are slow, your entire extended L2 network will suffer.  Must find a 
way to suppress L2 broadcast/multicast between sites.

Hope this helps.  Happy to share my LXD configurations with anyone...


On Sep 18, 2016, at 5:16 AM, Tomasz Chmielewski <man...@wpkg.org> wrote:

It's easy to create a "LAN" for LXD containers on a single LXD server - just 
attach them to the same bridge, use the same subnet (i.e. - 
done. Containers can communicate with each other using their private IP address.

However, with more then one LXD server *not* in the same LAN (i.e. two LXD 
servers in different datacentres), the things get tricky.

Is anyone using such setups, with multiple LXD servers and containers being 
able to communicate with each other?

LXD1: IP, Europe    LXD2: IP, Asia
container1,     container4,
container2,     container5,
container3,     container6,


While I can imagine setting up many OpenVPN tunnels between all LXD servers 
(LXD1-LXD2, LXD1-LXD3, LXD2-LXD3) and constantly adjusting the routes as 
containers are stopped/started/migrated, it's a bit of a management nightmare. 
And even more so if the number of LXD servers grows.

Hints, discussion?

Tomasz Chmielewski
lxc-users mailing list

lxc-users mailing list

Reply via email to