Re: [Nix-dev] Fwd: nixos-container networking

[Danylo Hlynskyi]

Host side:

networking.bridges.${bridgeName}.interfaces = [];
networking.interfaces.${bridgeName}.ip4 = [ { address = "${subnet}.1";
prefixLength = 24; } ];

# Each container takes at least 4 inotify file handles, so you quickly
reach limit 128 when spawning many containers
boot.kernel.sysctl."fs.inotify.max_user_instances" = 2048;

# internet in containers!
networking.nat.enable = true;
networking.nat.internalInterfaces = [ "ve-+" "vb-+" bridgeName ]; #
TODO check if bridgeName is required
networking.nat.externalInterface = "eth0";

Container side:

privateNetwork = true;
hostBridge = bridgeName;
localAddress = container_ip + "/24";
config = {
networking.defaultGateway = "${subnet}.1";
networking.extraHosts = ''
  ${subnet}.1 ${hostConfig.networking.hostName}
'';
};

There is also some code to set-up hostname resolving via /etc/hosts,
because I know little about DHCP
, mDNS, nss-* and so on (https://github.com/NixOS/nixpkgs/issues/16754). It
is a wrapper around `containers`
and monitors IP clashes (which are set manually in my setup)


In general, I have 1 bridge with multiple IPs/subnets, so containers in one
subnet don't see containers in other.

2017-03-14 20:12 GMT+02:00 Tomasz Czyż :

>
> -- Forwarded message --
> From: Tomasz Czyż 
> Date: 2017-03-14 18:12 GMT+00:00
> Subject: Re: [Nix-dev] nixos-container networking
> To: Danylo Hlynskyi 
>
>
> Hey Danylo,
>
> yup, I hit 13 char limit and because I was removing - I thought it's that.
>
> I also had the issue with recreating containers, but this happened only
> sometimes and didn't happen when I restart the machine so I was not sure
> why is that. Thanks for your mail, it's very useful.
>
> Would you share your bridget networking?
> I was trying that but I'm wondering if you have one shared bridge or you
> have bridge per container and how you access containers from host (or route
> traffic to them).
>
> Cheers,
> Tom
>
>
> 2017-03-14 6:01 GMT+00:00 Danylo Hlynskyi :
>
>> Strange, I have lot's of containers with "-" and experience no problems.
>> But maybe you've exceeded by accident limit 13 symbols per container name?
>>
>> Also, last time I tried "veth" networking, I was struggling from
>> https://github.com/NixOS/nixpkgs/issues/16330. My container experience
>> was awful when I tried container renames. That's why I've already switched
>> to bridged networking
>>
>> ---
>>
>> BTW, I highly recommend patch to switch-to-configuration.pl
>> 
>> from https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b2
>> 77f78ece1bb81b79b5651897e46a2bf
>>
>> It isn't clear from commit message, but it does the following: makes
>> declarative containers truly reloadable (when you change
>> container config, it activates new configuration for container). The
>> culprit is *it should be* default behavior, because of
>>
>> 1. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/v
>> irtualisation/containers.nix#L225-L230
>> 2. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/v
>> irtualisation/containers.nix#L676
>>
>> I'd like to PR this, but got no time to test properly other parts of
>> Nixos.
>>
>> 2017-03-14 4:42 GMT+02:00 Tomasz Czyż :
>>
>>> Michael, Ian, thank you for your answers.
>>>
>>> Looks like my problem was with the container name. I tried bunch of
>>> different setups which didn't work and I discovered that when I'm using "-"
>>> in container name it doesn't work (I had impression that worked one or two
>>> times when I started machine from scratch, but most of the time didn't).
>>>
>>> After I removed "-" from the name, looks like private network is working
>>> (I can access private IP of container) so I don't need NAT actually.
>>>
>>> Tom
>>>
>>> 2017-03-13 23:54 GMT+00:00 Ian-Woo Kim :
>>>
 I've recently made nixos-container port forwarding easier (both
 imperative and declarative) and it's now merged into master.

 https://github.com/NixOS/nixpkgs/pull/20869

 Hope that this helps.

 Ian

 On Sun, Mar 12, 2017 at 7:52 PM, Michael Walker 
 wrote:
 > Tomasz,
 >
 > I have declarative container networking set up and working on a VPS,
 > but I wrote most of the configuration as I was learning things, so it
 > may not be the best way.
 >
 > Here's the configuration.nix for the VPS:
 > https://github.com/barrucadu/nixfiles/blob/master/hosts/innsmouth.nix
 > Each container has a config file here:
 > https://github.com/barrucadu/nixfiles/tree/master/containers
 >
 > Containers have ports forwarded to them via NAT; each container is
 > running a web server 

[Nix-dev] Fwd: nixos-container networking

[Tomasz Czyż]

-- Forwarded message --
From: Tomasz Czyż 
Date: 2017-03-14 18:12 GMT+00:00
Subject: Re: [Nix-dev] nixos-container networking
To: Danylo Hlynskyi 


Hey Danylo,

yup, I hit 13 char limit and because I was removing - I thought it's that.

I also had the issue with recreating containers, but this happened only
sometimes and didn't happen when I restart the machine so I was not sure
why is that. Thanks for your mail, it's very useful.

Would you share your bridget networking?
I was trying that but I'm wondering if you have one shared bridge or you
have bridge per container and how you access containers from host (or route
traffic to them).

Cheers,
Tom


2017-03-14 6:01 GMT+00:00 Danylo Hlynskyi :

> Strange, I have lot's of containers with "-" and experience no problems.
> But maybe you've exceeded by accident limit 13 symbols per container name?
>
> Also, last time I tried "veth" networking, I was struggling from
> https://github.com/NixOS/nixpkgs/issues/16330. My container experience
> was awful when I tried container renames. That's why I've already switched
> to bridged networking
>
> ---
>
> BTW, I highly recommend patch to switch-to-configuration.pl
> 
> from https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b2
> 77f78ece1bb81b79b5651897e46a2bf
>
> It isn't clear from commit message, but it does the following: makes
> declarative containers truly reloadable (when you change
> container config, it activates new configuration for container). The
> culprit is *it should be* default behavior, because of
>
> 1. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/
> virtualisation/containers.nix#L225-L230
> 2. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/
> virtualisation/containers.nix#L676
>
> I'd like to PR this, but got no time to test properly other parts of Nixos.
>
> 2017-03-14 4:42 GMT+02:00 Tomasz Czyż :
>
>> Michael, Ian, thank you for your answers.
>>
>> Looks like my problem was with the container name. I tried bunch of
>> different setups which didn't work and I discovered that when I'm using "-"
>> in container name it doesn't work (I had impression that worked one or two
>> times when I started machine from scratch, but most of the time didn't).
>>
>> After I removed "-" from the name, looks like private network is working
>> (I can access private IP of container) so I don't need NAT actually.
>>
>> Tom
>>
>> 2017-03-13 23:54 GMT+00:00 Ian-Woo Kim :
>>
>>> I've recently made nixos-container port forwarding easier (both
>>> imperative and declarative) and it's now merged into master.
>>>
>>> https://github.com/NixOS/nixpkgs/pull/20869
>>>
>>> Hope that this helps.
>>>
>>> Ian
>>>
>>> On Sun, Mar 12, 2017 at 7:52 PM, Michael Walker 
>>> wrote:
>>> > Tomasz,
>>> >
>>> > I have declarative container networking set up and working on a VPS,
>>> > but I wrote most of the configuration as I was learning things, so it
>>> > may not be the best way.
>>> >
>>> > Here's the configuration.nix for the VPS:
>>> > https://github.com/barrucadu/nixfiles/blob/master/hosts/innsmouth.nix
>>> > Each container has a config file here:
>>> > https://github.com/barrucadu/nixfiles/tree/master/containers
>>> >
>>> > Containers have ports forwarded to them via NAT; each container is
>>> > running a web server on port 80 with the host reverse-proxying via
>>> > nginx; the host also does https and letsencrypt for all the proxied
>>> > containers.
>>> >
>>> > At the top of the innsmouth.nix file, I have a "containerSpecs" record
>>> > which has all the details for each container. The relevant bits of the
>>> > config are:
>>> >
>>> > 1. Set up the networking and NAT:
>>> >
>>> > networking.nat.enable = true;
>>> > networking.nat.internalInterfaces = ["ve-+"];
>>> > networking.nat.externalInterface = "enp0s4";
>>> >
>>> > 2. Forward ports to containers:
>>> >
>>> > networking.nat.forwardPorts = concatMap
>>> > ( {num, ports, ...}:
>>> > map (p: { sourcePort = p; destination =
>>> > "192.168.255.${toString num}:${toString p}"; }) ports
>>> > ) containerSpecs';
>>> >
>>> > 3. Define all the containers:
>>> >
>>> > containers = mapAttrs
>>> > (_: {num, config, ...}:
>>> > { autoStart = true
>>> > ; privateNetwork = true
>>> > ; hostAddress = "192.168.254.${toString num}"
>>> > ; localAddress = "192.168.255.${toString num}"
>>> > ; config = config
>>> > ; }
>>> > ) containerSpecs;
>>> >
>>> > 4. Reverse-proxy HTTPS to HTTP in each container, manage letsencrypt
>>> > certificates, and forward HTTP to HTTPS.
>>> >
>>> > This is a little complex as I have a fairly custom nginx config (see
>>> > the services/nginx.nix file in the repository), but the
>>>