Re: [Nix-dev] NFS performance issues after upgrade to 16.09

2016-10-17 Thread zimbatm
Great! Glad you managed to find a solution.

On Mon, 17 Oct 2016, 08:29 4levels, <4lev...@gmail.com> wrote:

> Hi all,
>
> I removed the vers=3 and some other NFS v3 related mount options since my
> host supports NFS versions up to 4.1 and the issues seem to have
> disappeared!
>
> As this is only happening (and NFS is only needed) in my local VM, this
> works for me ;-)
>
> If the issue comes back, I'll continue debugging NFS and report back here.
>
> Kind regards,
>
> Erik
>
> On Mon, 17 Oct 2016, 06:47 Wout Mertens,  wrote:
>
> Maybe it's something else?
> https://www.clearos.com/clearfoundation/social/community/solved-nfs-causing-high-load
>
> To further troubleshoot, I'd look into NFS logging to see what is being
> read/written, and snooping network traffic might be a quick way into that.
> Maybe in the new versions one of the apps started doing lots of mini writes
> with fsync?
>
> Wout.
>
> On Thu, Oct 13, 2016, 11:57 PM zimbatm  wrote:
>
> For the channels look into https://github.com/NixOS/nixpkgs-channels. The
> branches here map directly to the channels that you can fetch publicly. If
> you figure out which files are being used by nfs then it's a matter of
> running a diff between the different branches.
>
>
> https://github.com/NixOS/nixpkgs-channels/blob/nixos-16.09/nixos/modules/services/network-filesystems/nfsd.nix
> doesn't seem to have changed since 16.03.
> It seems to be using the nfs-utils which you can find in the
> pkgs/top-level/all-packages.nix and follow from there. nfs-utils seems to
> have changed:
> https://github.com/NixOS/nixpkgs-channels/blob/nixos-16.09/pkgs/os-specific/linux/nfs-utils/default.nix
> There is also the kernel module that you might want to dig into.
>
> I know it's not a direct solution but hopefully it will give you some more
> things to try out.
>
> Cheers,
> z
>
>
>
> On Thu, 13 Oct 2016 at 15:18 4levels <4lev...@gmail.com> wrote:
>
> Hi Jonas,
>
> disabling fail2ban didn't help, now the Nginx webserver keeps crashing as
> soon as a php error occurs.
> Do you know by any chance if with the upgrade to nixos-16.09 a newer
> version of NFS is included than in 16.03?  Maybe some default parameters in
> NFS have changed causing this havoc.
> As a more general question: how can I see which version of package x is
> used in nixos version y?
>
> Kind regards,
>
> Erik
>
> On Thu, Oct 13, 2016 at 11:05 AM 4levels <4lev...@gmail.com> wrote:
>
> Hi,
>
> thank you for your suggestions.  I've disabled the firewall and fail2ban
> services, let's hope that does the trick!
> I'll report back when things are better..
>
> Kind regards,
>
> Erik
>
> On Thu, Oct 13, 2016 at 9:09 AM zimbatm  wrote:
>
> Hi,
>
> What happens if you disable fail2ban? Maybe the behaviour has changed.
>
> Or try to change the kernel and NFS versions.
>
> I know it's not much help, all I can recommend is to try and replace each
> component to reduce the error.
>
> On Wed, 12 Oct 2016, 10:51 4levels, <4lev...@gmail.com> wrote:
>
> Dear Nix'ers,
>
> I've a permanent issue since the upgrade to nixos-16.09 in my local vm
> (with kvm-qemu from an SSD partition).  The load of the vm is increasing
> over time without any signs in the output of top.  Bash completion when
> traversing directories stalls and the whole system becomes unresponsive
> after about 5 to 10 minutes with top showing a load > 30.  Even rebooting
> fails with several services failing to stop (eg. fail2ban, phpfpm, ).
>
> This has everything to do with NFS: as soon as I disable the NFS mounts,
> the system maintains normal operation.  Nginx / phpfpm are using NFS
> mounted folders for local development.
>
> These are the filesystem declarations in the nixops expression:
>
>   fileSystems."/data/dev" = {
> device = "d01:/data/dev";
> fsType = "nfs";
> options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
> "actimeo=1" ];
>   };
>   fileSystems."/extra/Documents" = {
> device = "d01:/extra/Documents";
> fsType = "nfs";
> options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
> "actimeo=1" ];
>   };
>
> with d01 being declared in extrahosts
>
>   networking.extraHosts = "192.168.121.1 d01 d01.local";
>
> Has anyone an idea how this could be related to the upgrade to 16.09?  On
> 16.03 this all worked normally..
>
> Kind regards,
>
> Erik aka 4levels
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] NFS performance issues after upgrade to 16.09

2016-10-17 Thread 4levels
Hi all,

I removed the vers=3 and some other NFS v3 related mount options since my
host supports NFS versions up to 4.1 and the issues seem to have
disappeared!

As this is only happening (and NFS is only needed) in my local VM, this
works for me ;-)

If the issue comes back, I'll continue debugging NFS and report back here.

Kind regards,

Erik

On Mon, 17 Oct 2016, 06:47 Wout Mertens,  wrote:

Maybe it's something else?
https://www.clearos.com/clearfoundation/social/community/solved-nfs-causing-high-load

To further troubleshoot, I'd look into NFS logging to see what is being
read/written, and snooping network traffic might be a quick way into that.
Maybe in the new versions one of the apps started doing lots of mini writes
with fsync?

Wout.

On Thu, Oct 13, 2016, 11:57 PM zimbatm  wrote:

For the channels look into https://github.com/NixOS/nixpkgs-channels. The
branches here map directly to the channels that you can fetch publicly. If
you figure out which files are being used by nfs then it's a matter of
running a diff between the different branches.

https://github.com/NixOS/nixpkgs-channels/blob/nixos-16.09/nixos/modules/services/network-filesystems/nfsd.nix
doesn't seem to have changed since 16.03.
It seems to be using the nfs-utils which you can find in the
pkgs/top-level/all-packages.nix and follow from there. nfs-utils seems to
have changed:
https://github.com/NixOS/nixpkgs-channels/blob/nixos-16.09/pkgs/os-specific/linux/nfs-utils/default.nix
There is also the kernel module that you might want to dig into.

I know it's not a direct solution but hopefully it will give you some more
things to try out.

Cheers,
z



On Thu, 13 Oct 2016 at 15:18 4levels <4lev...@gmail.com> wrote:

Hi Jonas,

disabling fail2ban didn't help, now the Nginx webserver keeps crashing as
soon as a php error occurs.
Do you know by any chance if with the upgrade to nixos-16.09 a newer
version of NFS is included than in 16.03?  Maybe some default parameters in
NFS have changed causing this havoc.
As a more general question: how can I see which version of package x is
used in nixos version y?

Kind regards,

Erik

On Thu, Oct 13, 2016 at 11:05 AM 4levels <4lev...@gmail.com> wrote:

Hi,

thank you for your suggestions.  I've disabled the firewall and fail2ban
services, let's hope that does the trick!
I'll report back when things are better..

Kind regards,

Erik

On Thu, Oct 13, 2016 at 9:09 AM zimbatm  wrote:

Hi,

What happens if you disable fail2ban? Maybe the behaviour has changed.

Or try to change the kernel and NFS versions.

I know it's not much help, all I can recommend is to try and replace each
component to reduce the error.

On Wed, 12 Oct 2016, 10:51 4levels, <4lev...@gmail.com> wrote:

Dear Nix'ers,

I've a permanent issue since the upgrade to nixos-16.09 in my local vm
(with kvm-qemu from an SSD partition).  The load of the vm is increasing
over time without any signs in the output of top.  Bash completion when
traversing directories stalls and the whole system becomes unresponsive
after about 5 to 10 minutes with top showing a load > 30.  Even rebooting
fails with several services failing to stop (eg. fail2ban, phpfpm, ).

This has everything to do with NFS: as soon as I disable the NFS mounts,
the system maintains normal operation.  Nginx / phpfpm are using NFS
mounted folders for local development.

These are the filesystem declarations in the nixops expression:

  fileSystems."/data/dev" = {
device = "d01:/data/dev";
fsType = "nfs";
options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
"actimeo=1" ];
  };
  fileSystems."/extra/Documents" = {
device = "d01:/extra/Documents";
fsType = "nfs";
options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
"actimeo=1" ];
  };

with d01 being declared in extrahosts

  networking.extraHosts = "192.168.121.1 d01 d01.local";

Has anyone an idea how this could be related to the upgrade to 16.09?  On
16.03 this all worked normally..

Kind regards,

Erik aka 4levels

___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] NFS performance issues after upgrade to 16.09

2016-10-16 Thread Wout Mertens
Maybe it's something else?
https://www.clearos.com/clearfoundation/social/community/solved-nfs-causing-high-load

To further troubleshoot, I'd look into NFS logging to see what is being
read/written, and snooping network traffic might be a quick way into that.
Maybe in the new versions one of the apps started doing lots of mini writes
with fsync?

Wout.

On Thu, Oct 13, 2016, 11:57 PM zimbatm  wrote:

> For the channels look into https://github.com/NixOS/nixpkgs-channels. The
> branches here map directly to the channels that you can fetch publicly. If
> you figure out which files are being used by nfs then it's a matter of
> running a diff between the different branches.
>
>
> https://github.com/NixOS/nixpkgs-channels/blob/nixos-16.09/nixos/modules/services/network-filesystems/nfsd.nix
> doesn't seem to have changed since 16.03.
> It seems to be using the nfs-utils which you can find in the
> pkgs/top-level/all-packages.nix and follow from there. nfs-utils seems to
> have changed:
> https://github.com/NixOS/nixpkgs-channels/blob/nixos-16.09/pkgs/os-specific/linux/nfs-utils/default.nix
> There is also the kernel module that you might want to dig into.
>
> I know it's not a direct solution but hopefully it will give you some more
> things to try out.
>
> Cheers,
> z
>
>
>
> On Thu, 13 Oct 2016 at 15:18 4levels <4lev...@gmail.com> wrote:
>
> Hi Jonas,
>
> disabling fail2ban didn't help, now the Nginx webserver keeps crashing as
> soon as a php error occurs.
> Do you know by any chance if with the upgrade to nixos-16.09 a newer
> version of NFS is included than in 16.03?  Maybe some default parameters in
> NFS have changed causing this havoc.
> As a more general question: how can I see which version of package x is
> used in nixos version y?
>
> Kind regards,
>
> Erik
>
> On Thu, Oct 13, 2016 at 11:05 AM 4levels <4lev...@gmail.com> wrote:
>
> Hi,
>
> thank you for your suggestions.  I've disabled the firewall and fail2ban
> services, let's hope that does the trick!
> I'll report back when things are better..
>
> Kind regards,
>
> Erik
>
> On Thu, Oct 13, 2016 at 9:09 AM zimbatm  wrote:
>
> Hi,
>
> What happens if you disable fail2ban? Maybe the behaviour has changed.
>
> Or try to change the kernel and NFS versions.
>
> I know it's not much help, all I can recommend is to try and replace each
> component to reduce the error.
>
> On Wed, 12 Oct 2016, 10:51 4levels, <4lev...@gmail.com> wrote:
>
> Dear Nix'ers,
>
> I've a permanent issue since the upgrade to nixos-16.09 in my local vm
> (with kvm-qemu from an SSD partition).  The load of the vm is increasing
> over time without any signs in the output of top.  Bash completion when
> traversing directories stalls and the whole system becomes unresponsive
> after about 5 to 10 minutes with top showing a load > 30.  Even rebooting
> fails with several services failing to stop (eg. fail2ban, phpfpm, ).
>
> This has everything to do with NFS: as soon as I disable the NFS mounts,
> the system maintains normal operation.  Nginx / phpfpm are using NFS
> mounted folders for local development.
>
> These are the filesystem declarations in the nixops expression:
>
>   fileSystems."/data/dev" = {
> device = "d01:/data/dev";
> fsType = "nfs";
> options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
> "actimeo=1" ];
>   };
>   fileSystems."/extra/Documents" = {
> device = "d01:/extra/Documents";
> fsType = "nfs";
> options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
> "actimeo=1" ];
>   };
>
> with d01 being declared in extrahosts
>
>   networking.extraHosts = "192.168.121.1 d01 d01.local";
>
> Has anyone an idea how this could be related to the upgrade to 16.09?  On
> 16.03 this all worked normally..
>
> Kind regards,
>
> Erik aka 4levels
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] NFS performance issues after upgrade to 16.09

2016-10-13 Thread zimbatm
For the channels look into https://github.com/NixOS/nixpkgs-channels. The
branches here map directly to the channels that you can fetch publicly. If
you figure out which files are being used by nfs then it's a matter of
running a diff between the different branches.

https://github.com/NixOS/nixpkgs-channels/blob/nixos-16.09/nixos/modules/services/network-filesystems/nfsd.nix
doesn't seem to have changed since 16.03.
It seems to be using the nfs-utils which you can find in the
pkgs/top-level/all-packages.nix and follow from there. nfs-utils seems to
have changed:
https://github.com/NixOS/nixpkgs-channels/blob/nixos-16.09/pkgs/os-specific/linux/nfs-utils/default.nix
There is also the kernel module that you might want to dig into.

I know it's not a direct solution but hopefully it will give you some more
things to try out.

Cheers,
z



On Thu, 13 Oct 2016 at 15:18 4levels <4lev...@gmail.com> wrote:

> Hi Jonas,
>
> disabling fail2ban didn't help, now the Nginx webserver keeps crashing as
> soon as a php error occurs.
> Do you know by any chance if with the upgrade to nixos-16.09 a newer
> version of NFS is included than in 16.03?  Maybe some default parameters in
> NFS have changed causing this havoc.
> As a more general question: how can I see which version of package x is
> used in nixos version y?
>
> Kind regards,
>
> Erik
>
> On Thu, Oct 13, 2016 at 11:05 AM 4levels <4lev...@gmail.com> wrote:
>
> Hi,
>
> thank you for your suggestions.  I've disabled the firewall and fail2ban
> services, let's hope that does the trick!
> I'll report back when things are better..
>
> Kind regards,
>
> Erik
>
> On Thu, Oct 13, 2016 at 9:09 AM zimbatm  wrote:
>
> Hi,
>
> What happens if you disable fail2ban? Maybe the behaviour has changed.
>
> Or try to change the kernel and NFS versions.
>
> I know it's not much help, all I can recommend is to try and replace each
> component to reduce the error.
>
> On Wed, 12 Oct 2016, 10:51 4levels, <4lev...@gmail.com> wrote:
>
> Dear Nix'ers,
>
> I've a permanent issue since the upgrade to nixos-16.09 in my local vm
> (with kvm-qemu from an SSD partition).  The load of the vm is increasing
> over time without any signs in the output of top.  Bash completion when
> traversing directories stalls and the whole system becomes unresponsive
> after about 5 to 10 minutes with top showing a load > 30.  Even rebooting
> fails with several services failing to stop (eg. fail2ban, phpfpm, ).
>
> This has everything to do with NFS: as soon as I disable the NFS mounts,
> the system maintains normal operation.  Nginx / phpfpm are using NFS
> mounted folders for local development.
>
> These are the filesystem declarations in the nixops expression:
>
>   fileSystems."/data/dev" = {
> device = "d01:/data/dev";
> fsType = "nfs";
> options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
> "actimeo=1" ];
>   };
>   fileSystems."/extra/Documents" = {
> device = "d01:/extra/Documents";
> fsType = "nfs";
> options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
> "actimeo=1" ];
>   };
>
> with d01 being declared in extrahosts
>
>   networking.extraHosts = "192.168.121.1 d01 d01.local";
>
> Has anyone an idea how this could be related to the upgrade to 16.09?  On
> 16.03 this all worked normally..
>
> Kind regards,
>
> Erik aka 4levels
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] NFS performance issues after upgrade to 16.09

2016-10-13 Thread 4levels
Hi Jonas,

disabling fail2ban didn't help, now the Nginx webserver keeps crashing as
soon as a php error occurs.
Do you know by any chance if with the upgrade to nixos-16.09 a newer
version of NFS is included than in 16.03?  Maybe some default parameters in
NFS have changed causing this havoc.
As a more general question: how can I see which version of package x is
used in nixos version y?

Kind regards,

Erik

On Thu, Oct 13, 2016 at 11:05 AM 4levels <4lev...@gmail.com> wrote:

Hi,

thank you for your suggestions.  I've disabled the firewall and fail2ban
services, let's hope that does the trick!
I'll report back when things are better..

Kind regards,

Erik

On Thu, Oct 13, 2016 at 9:09 AM zimbatm  wrote:

Hi,

What happens if you disable fail2ban? Maybe the behaviour has changed.

Or try to change the kernel and NFS versions.

I know it's not much help, all I can recommend is to try and replace each
component to reduce the error.

On Wed, 12 Oct 2016, 10:51 4levels, <4lev...@gmail.com> wrote:

Dear Nix'ers,

I've a permanent issue since the upgrade to nixos-16.09 in my local vm
(with kvm-qemu from an SSD partition).  The load of the vm is increasing
over time without any signs in the output of top.  Bash completion when
traversing directories stalls and the whole system becomes unresponsive
after about 5 to 10 minutes with top showing a load > 30.  Even rebooting
fails with several services failing to stop (eg. fail2ban, phpfpm, ).

This has everything to do with NFS: as soon as I disable the NFS mounts,
the system maintains normal operation.  Nginx / phpfpm are using NFS
mounted folders for local development.

These are the filesystem declarations in the nixops expression:

  fileSystems."/data/dev" = {
device = "d01:/data/dev";
fsType = "nfs";
options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
"actimeo=1" ];
  };
  fileSystems."/extra/Documents" = {
device = "d01:/extra/Documents";
fsType = "nfs";
options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
"actimeo=1" ];
  };

with d01 being declared in extrahosts

  networking.extraHosts = "192.168.121.1 d01 d01.local";

Has anyone an idea how this could be related to the upgrade to 16.09?  On
16.03 this all worked normally..

Kind regards,

Erik aka 4levels

___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] NFS performance issues after upgrade to 16.09

2016-10-13 Thread 4levels
Hi,

thank you for your suggestions.  I've disabled the firewall and fail2ban
services, let's hope that does the trick!
I'll report back when things are better..

Kind regards,

Erik

On Thu, Oct 13, 2016 at 9:09 AM zimbatm  wrote:

> Hi,
>
> What happens if you disable fail2ban? Maybe the behaviour has changed.
>
> Or try to change the kernel and NFS versions.
>
> I know it's not much help, all I can recommend is to try and replace each
> component to reduce the error.
>
> On Wed, 12 Oct 2016, 10:51 4levels, <4lev...@gmail.com> wrote:
>
> Dear Nix'ers,
>
> I've a permanent issue since the upgrade to nixos-16.09 in my local vm
> (with kvm-qemu from an SSD partition).  The load of the vm is increasing
> over time without any signs in the output of top.  Bash completion when
> traversing directories stalls and the whole system becomes unresponsive
> after about 5 to 10 minutes with top showing a load > 30.  Even rebooting
> fails with several services failing to stop (eg. fail2ban, phpfpm, ).
>
> This has everything to do with NFS: as soon as I disable the NFS mounts,
> the system maintains normal operation.  Nginx / phpfpm are using NFS
> mounted folders for local development.
>
> These are the filesystem declarations in the nixops expression:
>
>   fileSystems."/data/dev" = {
> device = "d01:/data/dev";
> fsType = "nfs";
> options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
> "actimeo=1" ];
>   };
>   fileSystems."/extra/Documents" = {
> device = "d01:/extra/Documents";
> fsType = "nfs";
> options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
> "actimeo=1" ];
>   };
>
> with d01 being declared in extrahosts
>
>   networking.extraHosts = "192.168.121.1 d01 d01.local";
>
> Has anyone an idea how this could be related to the upgrade to 16.09?  On
> 16.03 this all worked normally..
>
> Kind regards,
>
> Erik aka 4levels
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] NFS performance issues after upgrade to 16.09

2016-10-13 Thread zimbatm
Hi,

What happens if you disable fail2ban? Maybe the behaviour has changed.

Or try to change the kernel and NFS versions.

I know it's not much help, all I can recommend is to try and replace each
component to reduce the error.

On Wed, 12 Oct 2016, 10:51 4levels, <4lev...@gmail.com> wrote:

> Dear Nix'ers,
>
> I've a permanent issue since the upgrade to nixos-16.09 in my local vm
> (with kvm-qemu from an SSD partition).  The load of the vm is increasing
> over time without any signs in the output of top.  Bash completion when
> traversing directories stalls and the whole system becomes unresponsive
> after about 5 to 10 minutes with top showing a load > 30.  Even rebooting
> fails with several services failing to stop (eg. fail2ban, phpfpm, ).
>
> This has everything to do with NFS: as soon as I disable the NFS mounts,
> the system maintains normal operation.  Nginx / phpfpm are using NFS
> mounted folders for local development.
>
> These are the filesystem declarations in the nixops expression:
>
>   fileSystems."/data/dev" = {
> device = "d01:/data/dev";
> fsType = "nfs";
> options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
> "actimeo=1" ];
>   };
>   fileSystems."/extra/Documents" = {
> device = "d01:/extra/Documents";
> fsType = "nfs";
> options = [ "defaults" "noatime" "nolock" "noacl" "vers=3" "udp"
> "actimeo=1" ];
>   };
>
> with d01 being declared in extrahosts
>
>   networking.extraHosts = "192.168.121.1 d01 d01.local";
>
> Has anyone an idea how this could be related to the upgrade to 16.09?  On
> 16.03 this all worked normally..
>
> Kind regards,
>
> Erik aka 4levels
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev