Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Tilak Waelde]

Hope this helps.  Happy to share my LXD configurations with anyone...

-Ron


Please do! I'd really love to see a description of a production lxd / 
lxc setup with proper networking and multiple hosts!


I haven't played around with it yet, but is it possible to include some 
sort of VRF-lite[0] into such a setup for multi tenancy purposes? Other 
than by using VLANs one could use the same IP ranges multiple times from 
what I've come to understand?
I'm not sure how a user could put the containers interfaces into a 
different network namespace..


cheers,
Tilak

[0] : https://www.kernel.org/doc/Documentation/networking/vrf.txt
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Sergiusz Pawlowicz]

On Sun, Sep 18, 2016 at 7:11 PM, Tomasz Chmielewski  wrote:
> On 2016-09-18 21:05, Sergiusz Pawlowicz wrote:
>>
>> On Sun, Sep 18, 2016 at 4:16 PM, Tomasz Chmielewski 
>> wrote:
>>
>>> While I can imagine setting up many OpenVPN tunnels between all LXD
>>> servers
>>
>>
>> I cannot imagine that :-) :-)
>>
>> Use tinc, mate. Your life begins :-)
>>
>> https://www.tinc-vpn.org/
>
> I did some reading about tinc before, and according to documentation and
> mailing lists:
>
> - performance may not be so great
>
> - it gets problematic as the number of tinc instances grows (few will be OK,
> dozens will work, but beyond that, the things might get slowish)
>
> - if I'm not mistaken, you need to run a tinc instance per LXD client, not
> per LXD server, so that's extra management and performance overhead (i.e. if
> two tinc clients are running on the same server, they would still encrypt
> the traffic to each other)

I think you must read more, because everything you wrote is simply false.

Or, maybe, try it? :-) :-) :-)

cheers!
S.
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Ron Kelley]

So, just for clarity, you are saying each LXD server will have no separate 
network connection for the containers.  Thus, all containers are private to the 
LXD server, and any outbound traffic must traverse the container server 
interface.  Is this correct?  If so, sorry, I must have missed this requirement 
in your initial email.



On Sep 18, 2016, at 9:41 AM, Tomasz Chmielewski  wrote:

On 2016-09-18 22:14, Ron Kelley wrote:
> (Long reply follows…)
> Personally, I think you need to look at the big picture for such
> deployments.  From what I read below, you are asking, “how do I extend
> my layer-2 subnets between data centers such that container1 in Europe
> can talk with container6 in Asia, etc”.  If this is true, I think you
> need to look at deploying data center hardware (servers with multiple
> NICs, IPMI/DRAC/iLO interfaces) with proper L2/L3 routing (L2TP/IPSEC,
> etc).  And, you must look at how your failover services will work in
> this design.  It’s easy to get a couple of servers working with a
> simple design, but those simple designs tend to go to production very
> fast without proper testing and design.

Well, it's not only about deploying on "different continents".

It can be also in the same datacentre, where the hosting doesn't give you a LAN 
option.

For example - Amazon AWS, same region, same availability zone.

The servers will have "private" addresses like 10.x.x.x, traffic there will be 
private to your servers, but there will be no LAN. You can't assign your own 
LAN addresses (10.x.x.x).

This means, while you can launch several LXD containers on every of these 
servers - but their "LAN" will be limited per each LXD server (unless we do 
some special tricks).

Some other hostings offer a public IP, or several public IPs per servers, in 
the same datacentre, but again, no LAN.


Tomasz Chmielewski
https://lxadm.com

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Tomasz Chmielewski]

On 2016-09-18 22:14, Ron Kelley wrote:

(Long reply follows…)

Personally, I think you need to look at the big picture for such
deployments.  From what I read below, you are asking, “how do I extend
my layer-2 subnets between data centers such that container1 in Europe
can talk with container6 in Asia, etc”.  If this is true, I think you
need to look at deploying data center hardware (servers with multiple
NICs, IPMI/DRAC/iLO interfaces) with proper L2/L3 routing (L2TP/IPSEC,
etc).  And, you must look at how your failover services will work in
this design.  It’s easy to get a couple of servers working with a
simple design, but those simple designs tend to go to production very
fast without proper testing and design.


Well, it's not only about deploying on "different continents".

It can be also in the same datacentre, where the hosting doesn't give 
you a LAN option.


For example - Amazon AWS, same region, same availability zone.

The servers will have "private" addresses like 10.x.x.x, traffic there 
will be private to your servers, but there will be no LAN. You can't 
assign your own LAN addresses (10.x.x.x).


This means, while you can launch several LXD containers on every of 
these servers - but their "LAN" will be limited per each LXD server 
(unless we do some special tricks).


Some other hostings offer a public IP, or several public IPs per 
servers, in the same datacentre, but again, no LAN.



Tomasz Chmielewski
https://lxadm.com
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Ron Kelley]

(Long reply follows…)

Personally, I think you need to look at the big picture for such deployments.  
From what I read below, you are asking, “how do I extend my layer-2 subnets 
between data centers such that container1 in Europe can talk with container6 in 
Asia, etc”.  If this is true, I think you need to look at deploying data center 
hardware (servers with multiple NICs, IPMI/DRAC/iLO interfaces) with proper 
L2/L3 routing (L2TP/IPSEC, etc).  And, you must look at how your failover 
services will work in this design.  It’s easy to get a couple of servers 
working with a simple design, but those simple designs tend to go to production 
very fast without proper testing and design.


All that said, here is one way I would tackle this type of request:

* Get servers with at least 3 NICs (preferably 5)
  * One iLO/DRAC/IPMI interface for out-of-band management
  * One for Container server management (ie: LXD1 IP 1.2.3.4) - use a second 
NIC for redundancy in a bonded configuration
  * One for Container hosting network (ie container1, container2, etc) - use a 
second NIC for redundancy and VLANs to separate traffic

* Get firewalls in each location with L2TP/IPSEC support (pfSense works great)
  * Extend your L2 networks between your sites with L2TP
  * Secure the connection with IPSSEC

* On your LXD servers, create 2 bonded NICs
  * One for container management (eth0, eth1)
  * One for hosting network (eth2, eth3)
  * Use VLANs on hosting network to separate traffic
  * Configure your containers with the appropriate VLAN tag (ie: 501)

Once the above is done, your containers can talk w/each other in different 
locations.  You can use firewall rules to allow/deny IP connections from your 
container VMs.  You can extend both your container management and hosting 
networks across the L2 tunnel allowing you to move VMs at will.  


General Notes:
---
* For server bonded connections, I use linux mode type 6;  works well, provides 
great throughput, requires no special configuration on directly-connected 
switches.
* On the LXD side, create multiple profiles with VLAN configurations.  
Personally, I have 2 profiles: one for VLAN 501 and one for VLAN 502.  Local 
firewall provides security between container networks.
* Be mindful of the services you share across the tunnels.  Things like iSCSI, 
NFS, etc will kill your network performance because of the chatty type of 
traffic.


Some good references:
---
https://doc.pfsense.org/index.php/L2TP/IPsec
http://archive.openflow.org/wk/index.php/Tunneling_-_GRE/L2TP
http://www.networkworld.com/article/2163334/tech-primers/what-can-l2tp-do-for-your-network-.html

Caution: L2 networks have a lot of broadcast traffic.  If your site-to-site 
connections are slow, your entire extended L2 network will suffer.  Must find a 
way to suppress L2 broadcast/multicast between sites.


Hope this helps.  Happy to share my LXD configurations with anyone...

-Ron






On Sep 18, 2016, at 5:16 AM, Tomasz Chmielewski  wrote:

It's easy to create a "LAN" for LXD containers on a single LXD server - just 
attach them to the same bridge, use the same subnet (i.e. 10.10.10.0/24) - 
done. Containers can communicate with each other using their private IP address.

However, with more then one LXD server *not* in the same LAN (i.e. two LXD 
servers in different datacentres), the things get tricky.


Is anyone using such setups, with multiple LXD servers and containers being 
able to communicate with each other?


LXD1: IP 1.2.3.4, EuropeLXD2: IP 2.3.4.5, Asia
container1, 10.10.10.10 container4, 10.10.10.20
container2, 10.10.10.11 container5, 10.10.10.21
container3, 10.10.10.12 container6, 10.10.10.22


LXD3: IP 3.4.5.6, US
container7, 10.10.10.30
container8, 10.10.10.31
container8, 10.10.10.32


While I can imagine setting up many OpenVPN tunnels between all LXD servers 
(LXD1-LXD2, LXD1-LXD3, LXD2-LXD3) and constantly adjusting the routes as 
containers are stopped/started/migrated, it's a bit of a management nightmare. 
And even more so if the number of LXD servers grows.

Hints, discussion?


Tomasz Chmielewski
https://lxadm.com
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Tomasz Chmielewski]

On 2016-09-18 21:05, Sergiusz Pawlowicz wrote:
On Sun, Sep 18, 2016 at 4:16 PM, Tomasz Chmielewski  
wrote:


While I can imagine setting up many OpenVPN tunnels between all LXD 
servers


I cannot imagine that :-) :-)

Use tinc, mate. Your life begins :-)

https://www.tinc-vpn.org/


I did some reading about tinc before, and according to documentation and 
mailing lists:


- performance may not be so great

- it gets problematic as the number of tinc instances grows (few will be 
OK, dozens will work, but beyond that, the things might get slowish)


- if I'm not mistaken, you need to run a tinc instance per LXD client, 
not per LXD server, so that's extra management and performance overhead 
(i.e. if two tinc clients are running on the same server, they would 
still encrypt the traffic to each other)



Tomasz Chmielewski
https://lxadm.com
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Sergiusz Pawlowicz]

On Sun, Sep 18, 2016 at 4:16 PM, Tomasz Chmielewski  wrote:

> While I can imagine setting up many OpenVPN tunnels between all LXD servers

I cannot imagine that :-) :-)

Use tinc, mate. Your life begins :-)

https://www.tinc-vpn.org/

cheers,
Serge
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Micky Del Favero]

Tomasz Chmielewski  writes:

> While I can imagine setting up many OpenVPN tunnels between all LXD
> servers (LXD1-LXD2, LXD1-LXD3, LXD2-LXD3) and constantly adjusting the
> routes as containers are stopped/started/migrated, it's a bit of a
> management nightmare. And even more so if the number of LXD servers
> grows.
>
> Hints, discussion?

If you use the same subnet for the container as in your example
10.10.10.0/24 there's no routing nightmare, you have only to setup
openvpn in bridge mode and so you'll need only 2 tunnels: LXD1-LXD2 and
LXD2-LXD3, obviously if LXD2 will go offline you'll have a problem.

If you'll have many containers using the same subnet for all will be
impossible so you'll need to managed routing between hosts to be able to
reach every container.

I'm about to design and deploy similar setup: many LXD hosts running
containers that can be started on a hosts and eventually migrated to
another host in the same or in a different datacenter.

I'm thinking not to use the same subnet for all containers but to use a
different subnet for every host, so following your example it will be
like the following:

LXD1: IP 1.2.3.4, EuropeLXD2: IP 2.3.4.5, Asia
container1, 10.10.10.10 container4, 10.10.20.10
container2, 10.10.10.11 container5, 10.10.20.11
container3, 10.10.10.12 container6, 10.10.20.12

LXD3: IP 3.4.5.6, US
container7, 10.10.30.10
container8, 10.10.30.11
container9, 10.10.30.12

on every hosts a dhcp server is use to dynamically configure network on
starting container and, based on container's hostname, to dynamically
update dns servers so containers will be always reachable via hostname.

Routing problem remains, I'll probably choose to deploy some tunnels
between hosts to connect each other, then running an OSPF daemon for
dynamic routing management.

Ciao, Micky
-- 
The sysadmin has all the answers, expecially "No"
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Tomasz Chmielewski]

It's easy to create a "LAN" for LXD containers on a single LXD server - 
just attach them to the same bridge, use the same subnet (i.e. 
10.10.10.0/24) - done. Containers can communicate with each other using 
their private IP address.


However, with more then one LXD server *not* in the same LAN (i.e. two 
LXD servers in different datacentres), the things get tricky.



Is anyone using such setups, with multiple LXD servers and containers 
being able to communicate with each other?



LXD1: IP 1.2.3.4, EuropeLXD2: IP 2.3.4.5, Asia
container1, 10.10.10.10 container4, 10.10.10.20
container2, 10.10.10.11 container5, 10.10.10.21
container3, 10.10.10.12 container6, 10.10.10.22


LXD3: IP 3.4.5.6, US
container7, 10.10.10.30
container8, 10.10.10.31
container8, 10.10.10.32


While I can imagine setting up many OpenVPN tunnels between all LXD 
servers (LXD1-LXD2, LXD1-LXD3, LXD2-LXD3) and constantly adjusting the 
routes as containers are stopped/started/migrated, it's a bit of a 
management nightmare. And even more so if the number of LXD servers 
grows.


Hints, discussion?


Tomasz Chmielewski
https://lxadm.com
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users