Host side: networking.bridges.${bridgeName}.interfaces = []; networking.interfaces.${bridgeName}.ip4 = [ { address = "${subnet}.1"; prefixLength = 24; } ];
# Each container takes at least 4 inotify file handles, so you quickly reach limit 128 when spawning many containers boot.kernel.sysctl."fs.inotify.max_user_instances" = 2048; # internet in containers! networking.nat.enable = true; networking.nat.internalInterfaces = [ "ve-+" "vb-+" bridgeName ]; # TODO check if bridgeName is required networking.nat.externalInterface = "eth0"; Container side: privateNetwork = true; hostBridge = bridgeName; localAddress = container_ip + "/24"; config = { networking.defaultGateway = "${subnet}.1"; networking.extraHosts = '' ${subnet}.1 ${hostConfig.networking.hostName} ''; }; There is also some code to set-up hostname resolving via /etc/hosts, because I know little about DHCP , mDNS, nss-* and so on (https://github.com/NixOS/nixpkgs/issues/16754). It is a wrapper around `containers` and monitors IP clashes (which are set manually in my setup) In general, I have 1 bridge with multiple IPs/subnets, so containers in one subnet don't see containers in other. 2017-03-14 20:12 GMT+02:00 Tomasz Czyż <tomasz.c...@gmail.com>: > > ---------- Forwarded message ---------- > From: Tomasz Czyż <tomasz.c...@gmail.com> > Date: 2017-03-14 18:12 GMT+00:00 > Subject: Re: [Nix-dev] nixos-container networking > To: Danylo Hlynskyi <abcz2.upr...@gmail.com> > > > Hey Danylo, > > yup, I hit 13 char limit and because I was removing - I thought it's that. > > I also had the issue with recreating containers, but this happened only > sometimes and didn't happen when I restart the machine so I was not sure > why is that. Thanks for your mail, it's very useful. > > Would you share your bridget networking? > I was trying that but I'm wondering if you have one shared bridge or you > have bridge per container and how you access containers from host (or route > traffic to them). > > Cheers, > Tom > > > 2017-03-14 6:01 GMT+00:00 Danylo Hlynskyi <abcz2.upr...@gmail.com>: > >> Strange, I have lot's of containers with "-" and experience no problems. >> But maybe you've exceeded by accident limit 13 symbols per container name? >> >> Also, last time I tried "veth" networking, I was struggling from >> https://github.com/NixOS/nixpkgs/issues/16330. My container experience >> was awful when I tried container renames. That's why I've already switched >> to bridged networking >> >> --- >> >> BTW, I highly recommend patch to switch-to-configuration.pl >> <https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b277f78ece1bb81b79b5651897e46a2bf#diff-0a057d6ff3f6f83f68b859178484f4fe> >> from https://github.com/NixOS/nixpkgs/pull/3021/commits/6e36619b2 >> 77f78ece1bb81b79b5651897e46a2bf >> >> It isn't clear from commit message, but it does the following: makes >> declarative containers truly reloadable (when you change >> container config, it activates new configuration for container). The >> culprit is *it should be* default behavior, because of >> >> 1. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/v >> irtualisation/containers.nix#L225-L230 >> 2. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/v >> irtualisation/containers.nix#L676 >> >> I'd like to PR this, but got no time to test properly other parts of >> Nixos. >> >> 2017-03-14 4:42 GMT+02:00 Tomasz Czyż <tomasz.c...@gmail.com>: >> >>> Michael, Ian, thank you for your answers. >>> >>> Looks like my problem was with the container name. I tried bunch of >>> different setups which didn't work and I discovered that when I'm using "-" >>> in container name it doesn't work (I had impression that worked one or two >>> times when I started machine from scratch, but most of the time didn't). >>> >>> After I removed "-" from the name, looks like private network is working >>> (I can access private IP of container) so I don't need NAT actually. >>> >>> Tom >>> >>> 2017-03-13 23:54 GMT+00:00 Ian-Woo Kim <ianwoo...@gmail.com>: >>> >>>> I've recently made nixos-container port forwarding easier (both >>>> imperative and declarative) and it's now merged into master. >>>> >>>> https://github.com/NixOS/nixpkgs/pull/20869 >>>> >>>> Hope that this helps. >>>> >>>> Ian >>>> >>>> On Sun, Mar 12, 2017 at 7:52 PM, Michael Walker <m...@barrucadu.co.uk> >>>> wrote: >>>> > Tomasz, >>>> > >>>> > I have declarative container networking set up and working on a VPS, >>>> > but I wrote most of the configuration as I was learning things, so it >>>> > may not be the best way. >>>> > >>>> > Here's the configuration.nix for the VPS: >>>> > https://github.com/barrucadu/nixfiles/blob/master/hosts/innsmouth.nix >>>> > Each container has a config file here: >>>> > https://github.com/barrucadu/nixfiles/tree/master/containers >>>> > >>>> > Containers have ports forwarded to them via NAT; each container is >>>> > running a web server on port 80 with the host reverse-proxying via >>>> > nginx; the host also does https and letsencrypt for all the proxied >>>> > containers. >>>> > >>>> > At the top of the innsmouth.nix file, I have a "containerSpecs" record >>>> > which has all the details for each container. The relevant bits of the >>>> > config are: >>>> > >>>> > 1. Set up the networking and NAT: >>>> > >>>> > networking.nat.enable = true; >>>> > networking.nat.internalInterfaces = ["ve-+"]; >>>> > networking.nat.externalInterface = "enp0s4"; >>>> > >>>> > 2. Forward ports to containers: >>>> > >>>> > networking.nat.forwardPorts = concatMap >>>> > ( {num, ports, ...}: >>>> > map (p: { sourcePort = p; destination = >>>> > "192.168.255.${toString num}:${toString p}"; }) ports >>>> > ) containerSpecs'; >>>> > >>>> > 3. Define all the containers: >>>> > >>>> > containers = mapAttrs >>>> > (_: {num, config, ...}: >>>> > { autoStart = true >>>> > ; privateNetwork = true >>>> > ; hostAddress = "192.168.254.${toString num}" >>>> > ; localAddress = "192.168.255.${toString num}" >>>> > ; config = config >>>> > ; } >>>> > ) containerSpecs; >>>> > >>>> > 4. Reverse-proxy HTTPS to HTTP in each container, manage letsencrypt >>>> > certificates, and forward HTTP to HTTPS. >>>> > >>>> > This is a little complex as I have a fairly custom nginx config (see >>>> > the services/nginx.nix file in the repository), but the >>>> > reverse-proxying is fairly straightfoward. Here is the generated >>>> > nginx.conf: https://misc.barrucadu.co.uk/nginx.txt >>>> > >>>> > On 13 March 2017 at 02:12, Tomasz Czyż <tomasz.c...@gmail.com> wrote: >>>> >> Hey, >>>> >> >>>> >> could anyone using nixos-container (declarative style) share how you >>>> setup >>>> >> networking? >>>> >> >>>> >> I'm trying to setup few containers with private network and http >>>> proxy at >>>> >> the front. Each container potentially could run application on port >>>> 80 and I >>>> >> would like to expose them through proxy. >>>> >> >>>> >> I tried to set this up with >>>> >> >>>> >> privateNetwork=true; >>>> >> hostAddress >>>> >> localAddress >>>> >> >>>> >> and I tried to also run nat on the host with (just to enable outbound >>>> >> traffic) >>>> >> internalInterfaces = ["ve-+"]; >>>> >> externalInterfaces = "eth0"; >>>> >> >>>> >> but no luck. >>>> >> My next try will be creating bridge on the host and add containers >>>> to that >>>> >> bridge. Is that how you do stuff or are better ways of doing >>>> container >>>> >> networking? >>>> >> >>>> >> Tom >>>> >> >>>> >> _______________________________________________ >>>> >> nix-dev mailing list >>>> >> nix-dev@lists.science.uu.nl >>>> >> http://lists.science.uu.nl/mailman/listinfo/nix-dev >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > Michael Walker (http://www.barrucadu.co.uk) >>>> > _______________________________________________ >>>> > nix-dev mailing list >>>> > nix-dev@lists.science.uu.nl >>>> > http://lists.science.uu.nl/mailman/listinfo/nix-dev >>>> >>> >>> >>> >>> -- >>> Tomasz Czyż >>> >>> _______________________________________________ >>> nix-dev mailing list >>> nix-dev@lists.science.uu.nl >>> http://lists.science.uu.nl/mailman/listinfo/nix-dev >>> >>> >> > > > -- > Tomasz Czyż > > > > -- > Tomasz Czyż > > _______________________________________________ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev > >
_______________________________________________ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev