Re: [Nix-dev] /dev/shm inconsistency in chroot
Mark H Weaver m...@netris.org skribis: Shea Levy s...@shealevy.com writes: On 01/24/2014 05:16 AM, Sree Harsha Totakura wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/23/2014 08:56 PM, Mark H Weaver wrote: We should not inherit /dev from the host system at all, but rather create it from scratch with just the things we need. IMO, that's the only truly proper solution. We can try creating a fixed set of device nodes, for example: /dev/null, /dev/random, /dev/urandom, /dev/sda etc. Has anyone tried this before? Another option is to mount a devtmpfs there, for systems which support it. The thing is, we don't actually want most of the system's devices to be in the build environment, do we? These are all impurities. I don't think we want /dev/sda, for example. For the record, with https://github.com/NixOS/nix/commit/3fd01b171a74d28dc8e48b9ee5f2d0e9a3915fb8, the daemon creates /dev deterministically. (This change landed in guix-daemon with the latest ‘nix-upstream’ update.) Ludo’.
Re: [Nix-dev] /dev/shm inconsistency in chroot
On 01/24/2014 05:16 AM, Sree Harsha Totakura wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/23/2014 08:56 PM, Mark H Weaver wrote: We should not inherit /dev from the host system at all, but rather create it from scratch with just the things we need. IMO, that's the only truly proper solution. We can try creating a fixed set of device nodes, for example: /dev/null, /dev/random, /dev/urandom, /dev/sda etc. Has anyone tried this before? Another option is to mount a devtmpfs there, for systems which support it. Sree -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Icedove - http://www.enigmail.net/ iEYEARECAAYFAlLiPZUACgkQO2+K8UPCHzvyuwCgpPH4ndRBqFkITqbPcQ1UN4Ws JsYAniMrgj8mBvNMC7Jq1AkFv+bV/VUj =DbCU -END PGP SIGNATURE- ___ nix-dev mailing list nix-...@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] /dev/shm inconsistency in chroot
Shea Levy s...@shealevy.com skribis: On 01/24/2014 05:16 AM, Sree Harsha Totakura wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/23/2014 08:56 PM, Mark H Weaver wrote: We should not inherit /dev from the host system at all, but rather create it from scratch with just the things we need. IMO, that's the only truly proper solution. We can try creating a fixed set of device nodes, for example: /dev/null, /dev/random, /dev/urandom, /dev/sda etc. Has anyone tried this before? Another option is to mount a devtmpfs there, for systems which support it. Both options look good to me. Using devtmpfs would be easier, but I don’t see any documentation for it. Are its contents really deterministic? Thanks, Ludo’.
Re: [Nix-dev] /dev/shm inconsistency in chroot
Shea Levy s...@shealevy.com writes: On 01/24/2014 05:16 AM, Sree Harsha Totakura wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/23/2014 08:56 PM, Mark H Weaver wrote: We should not inherit /dev from the host system at all, but rather create it from scratch with just the things we need. IMO, that's the only truly proper solution. We can try creating a fixed set of device nodes, for example: /dev/null, /dev/random, /dev/urandom, /dev/sda etc. Has anyone tried this before? Another option is to mount a devtmpfs there, for systems which support it. The thing is, we don't actually want most of the system's devices to be in the build environment, do we? These are all impurities. I don't think we want /dev/sda, for example. Mark
Re: [Nix-dev] /dev/shm inconsistency in chroot
Sree Harsha Totakura sreehar...@totakura.in skribis: On 01/24/2014 06:13 PM, Mark H Weaver wrote: Shea Levy s...@shealevy.com writes: Another option is to mount a devtmpfs there, for systems which support it. devtmpfs may give different devices on each machine and they may hinder our build reproducibility. OK. The thing is, we don't actually want most of the system's devices to be in the build environment, do we? These are all impurities. I don't think we want /dev/sda, for example. Sure, I agree. I propose we start enumerating commonly needed devices and create them. Sounds good. The major/minor device numbers may not be portable across OSes, which may be a problem for Nix, so that code may need to be #ifdef’d. If in future, a package requires access to certain device while building (or during tests) we can include it in our list of created device nodes. Yes, but keep in mind that we’re not going to change that often, because it’s inconvenient. Thanks, Ludo’.
Re: [Nix-dev] /dev/shm inconsistency in chroot
On 01/24/2014 04:17 PM, Ludovic Courtès wrote: Sree Harsha Totakura sreehar...@totakura.in skribis: On 01/24/2014 06:13 PM, Mark H Weaver wrote: Shea Levy s...@shealevy.com writes: Another option is to mount a devtmpfs there, for systems which support it. devtmpfs may give different devices on each machine and they may hinder our build reproducibility. OK. The thing is, we don't actually want most of the system's devices to be in the build environment, do we? These are all impurities. I don't think we want /dev/sda, for example. Sure, I agree. I propose we start enumerating commonly needed devices and create them. Sounds good. The major/minor device numbers may not be portable across OSes, which may be a problem for Nix, so that code may need to be #ifdef’d. Generally a good idea, but note that for now chroot is not enabled on non-Linux If in future, a package requires access to certain device while building (or during tests) we can include it in our list of created device nodes. Yes, but keep in mind that we’re not going to change that often, because it’s inconvenient. Thanks, Ludo’.
/dev/shm inconsistency in chroot
Hi,
It has been known for a while that any access to /dev/shm fails inside
chroot if the host system has /dev/shm symlinked to /run/shm.
The suggested method to deal with this until now is to remove the
symlink from the host system, create /dev/shm directory and mount a
tmpfs on it.
I spent sometime investigating why this is needed and here are some
points I noted: from libstore/build.cc the daemon tries to bind mount
some paths from the host system into the chroot directory. Among these
are /dev and /dev/pts. The daemon also mounts a tmpfs file system under
chrootdir+/dev/shm should the path /dev/shm exist.
Due to bind mounting /dev, on systems where /dev/shm is a symlink, the
symlink is also present in the chrootdir+/dev. Since the symlink points
to /run/shm, and since /run is not bind mounted by default into the
chrootdir+/run, the symlink is broken inside chroot.
The above problem can be addressed by passing --chroot-directory=/run to
the guix-daemon which then includes /run into the list of mounts that
are to be bind mounted inside chroot. This will resolve the
chroot+/dev/shm symlink properly. Yet, the accesses to /dev/shm fail
inside the chroot. This is because the mount statement in build.cc for
mounting tmpfs at chroot+/dev/shm mounts the tmpfs at /run/shm the
target of the symlink and since /run tree is made private the mount does
not propagate into the bind mounted chroot+/run tree. In the chroot,
this leaves the /dev/shm symlink to point to the directory /run/shm and
obviously any shared memory accesses fail.
This problem can be fixed finally by passing
--chroot-directory=/run/shm. Although the tmpfs mount for
chroot+/dev/shm still ends up mounting tmpfs at /run/shm, since /run/shm
is now bind mounted to chroot+/run/shm, inside the chroot the symlink
/dev/shm points to /run/shm which is now a tmpfs. The shared memory
accesses work fine here. However there is caveat: since /run/shm is
bindmounted any files from the host system from that directory are also
present in the chroot.
To address this further and to get rid of having the user to pass
--chroot-directory flag, I propose the attached patch.
Regards,
Sree
From 359fda36b6dcabea79a76b56e10d4d67702f545f Mon Sep 17 00:00:00 2001
From: Sree Harsha Totakura sreehar...@totakura.in
Date: Thu, 23 Jan 2014 20:11:57 +0100
Subject: [PATCH] Create tmpfs on /dev/shm after chrooting.
src/libstore/build.cc: Create tmpfs on /dev/shm after chrooting. If /dev/shm is a link, create required directory for mounting.
---
src/libstore/build.cc | 17 +++--
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/src/libstore/build.cc b/src/libstore/build.cc
index 4329d9a..b01bf92 100644
--- a/src/libstore/build.cc
+++ b/src/libstore/build.cc
@@ -2054,12 +2054,6 @@ void DerivationGoal::initChild()
if (mount(none, (chrootRootDir + /proc).c_str(), proc, 0, 0) == -1)
throw SysError(mounting /proc);
-/* Mount a new tmpfs on /dev/shm to ensure that whatever
- the builder puts in /dev/shm is cleaned up automatically. */
-if (pathExists(/dev/shm))
-if (mount(none, (chrootRootDir + /dev/shm).c_str(), tmpfs, 0, 0) == -1)
-throw SysError(mounting /dev/shm);
-
/* Do the chroot(). Below we do a chdir() to the
temporary build directory to make sure the current
directory is in the chroot. (Actually the order
@@ -2067,6 +2061,17 @@ void DerivationGoal::initChild()
tmpRootDit/tmpDir are the same directories.) */
if (chroot(chrootRootDir.c_str()) == -1)
throw SysError(format(cannot change root directory to `%1%') % chrootRootDir);
+
+/* Mount a new tmpfs on /dev/shm to ensure that whatever
+ the builder puts in /dev/shm is cleaned up automatically. */
+if (pathExists (/dev/shm))
+{
+Path target = /dev/shm;
+if (isLink(target) !pathExists(target = readLink(target)))
+createDirs(target);
+if (mount(none, /dev/shm, tmpfs, 0, 0) == -1)
+throw SysError(mounting /dev/shm);
+}
}
#endif
--
1.7.10.4
signature.asc
Description: OpenPGP digital signature
Re: /dev/shm inconsistency in chroot
Hi, Sree Harsha Totakura sreehar...@totakura.in writes: It has been known for a while that any access to /dev/shm fails inside chroot if the host system has /dev/shm symlinked to /run/shm. The suggested method to deal with this until now is to remove the symlink from the host system, create /dev/shm directory and mount a tmpfs on it. [...] The above problem can be addressed by passing --chroot-directory=/run to the guix-daemon which then includes /run into the list of mounts that are to be bind mounted inside chroot. This proposal would take us in the wrong direction. We should not solve this problem by inheriting more directories from the host system, but rather by inheriting fewer. Everything that we inherit from the host system is a potential impurity, and this would add more of them. We should not inherit /dev from the host system at all, but rather create it from scratch with just the things we need. IMO, that's the only truly proper solution. Thoughts? Mark
