Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
Dnia 2015-01-25, o godz. 16:44:20 Thomas D. whi...@whissi.de napisał(a): Hi, Michał Górny wrote: I see two generic approaches possible here: 1. proxying distcc from within the build environment, or 2. moving distcc-spawned processes back to parent's namespace. distcc client/server solution - The most obvious solution to me is to employ a client/server model where a system-wide daemon is running, parsing /etc/distcc/hosts and doing all the network activity. [...] It is not only distcc. Please don't forget things like sys-devel/icecream. I care for what I use. If people want their icecreams, they have to write a patch themselves :). -- Best regards, Michał Górny pgpejbGCkMACU.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
Dnia 2015-01-27, o godz. 03:03:31 Andrew Savchenko birc...@gentoo.org napisał(a): On Sat, 24 Jan 2015 12:03:27 +0100 Michał Górny wrote: Dnia 2015-01-21, o godz. 11:05:34 Michał Górny mgo...@gentoo.org napisał(a): Parent namespace solution - The alternative is to wrap distcc so that parent network namespace is regained. Sadly, for a start this requires SYS_CAP_ADMIN and access to parent's /proc entry (so root privileges). We obviously don't want to lose userpriv, so we have to do with setuid wrapper. Of course, setuid wrappers need to be written in C, so we need to introduce C compiled executable in Portage. Then we need to take some special care for it not to be abused, e.g. to access restricted network namespaces of other processes. I've started to work on this, and I have all the setup.py code ready. Now it's just the C program and wrapping it all in Portage. The only remaining big issue is how to pass the correct namespace (or PID) to the distcc process, assuming it may be a deep child of the ebuild process. Possibilities: 1. Environment variables. Can't do because: a. one of the parent processes may scrub the environment, b. malicious process may override the variable, giving subprocess access to restricted system network namespace. 2. Files. Can't do because we would have to pass the path somehow (see 1.), and if we go for static path, we have collisions between multiple emerge processes. If we're not going to use PID isolation in future, PID-dependent static file name may be used, where PID belongs to emerge. This should solve the collision issue. I didn't want to make network-sandbox depend on another FEATUREs. 3. Parent process hierarchy. Seems fragile, requires a lot of scanning on every spawn and may end up in the wrong namespace if a subprocess calls setsid() and user put emerge in a separate namespace. Why not to use cgroups here? Not sure if cgroups help here. In our cgroup hierarchy, only child processes are governed. Not sure if setsid() behaves any different with that. 4. IPv4 socket. While it may seem a real dumb idea, it fits the case perfectly since it is only run if process is network-namespaced. We can surely reserve a static port early and run a daemon outputting the correct PID. Since the port is namespaced, every Portage process can have its own PID-daemon, and I don't know if there's any way of overriding the result from parent process. Looks fine, though the very idea to have extra daemons is unpleasant. Yes, that's one of the issues. And spawning anything from inside ebuild process is terribly hacky and fragile since it doesn't do any proper process management. My initial implementation had even more issues: PATH changes were permanent because of env saving (so post-distcc phases still had distcc wrappers) and ld.so complained because setuid caused libsandbox not to be preloaded. Such errors could break configure checks. -- Best regards, Michał Górny pgpxKYzNyUNS1.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
Dnia 2015-01-27, o godz. 02:46:37 Andrew Savchenko birc...@gentoo.org napisał(a): Hi, On Sun, 25 Jan 2015 14:59:01 +0100 Michał Górny wrote: Dnia 2015-01-21, o godz. 11:05:34 Michał Górny mgo...@gentoo.org napisał(a): Generic proxy solution -- The simplest solution so far seems to be setting a generic SOCKS proxy inside the build environment, and wrapping distcc so that it will use it for network access. Unless we do some extra magic which don't want to do, this means that other apps can also abuse the proxy to reach outside sandbox. However, network-sandbox is not really a security feature, so I don't think that is important. At least as long as we don't export it globally :). Of course, software is a problem. We'd need at least some SOCKS server for Portage (at least a very simple one), and as far as I'm aware distcc does not support SOCKS directly, so tsocks in addition to that. So finally went this way instead. I still don't understand why. This solution: 1) is intrusive, it requires patching distcc and upstream as good as dead (see below); Adding SOCKSv5 support is something of potentially generic benefit unlike adding Gentoo-specific hacks. 2) will require a _separate_ solution for icecream and thus a double effort; It would require a separate solution for icecream anyway. If icecream had SOCKSv5 support, the solution would be as simple as exporting extra variable for it. Unlike the other case when Portage needs to become aware of layout of another tool. 3) adds additional latency for distcc network path, which is undesirable. Unlikely to be measurable. ~20 extra bytes over UNIX socket. In fact, this provides us with ability to do some tricks to actually improve the latency -- like caching extra connections in the proxy. In fact, the other solution is more complex and therefore more likely to cause delays. Parent namespace solution looks like the most reasonable for me based on both arguments above and years of heavy distcc usage experience. [2]:https://code.google.com/p/distcc/issues/detail?id=149 Chances to have this upstream are close to zero. If it is not dead, it is very close to it. Number of bugs and patches is accumulating without any response. No releases from 2011. Probably someone should fork it... I know. Forgive me the wording but it's pretty much crap code. Involving ideas such as enabling non-blocking I/O just to read it in loops to get blocking behavior. I would rewrite it from scratch if it were simpler. But the pump mode and other extra features makes it require too much effort. At least it's not something I could do in my free time. Distcc has a problem with -march=native right now: it just falls back to local compilation if encountered it. I sent them a patch 1.5 years ago [1] and still no reply... It also requires some patches for successful cross-compilation when plain gcc is invoked by the client. (I have patches for amd64 - x86 only and they may broke pump mode (never used it anyway), thus I haven't send them upstream. [1] https://groups.google.com/forum/#!topic/distcc-patches/eeP-9pTgz7E In short, this patch expands native argument using gcc output, caches result (based on fingerprint of compiler being invoked) and sends expanded string to distcc servers. It is in my overlay (bircoph) if someone is interested. Works fine for me all these time. I'm not convinced it's worth the effort to support it in distcc. I once published a blog post how to use gcc output to replace -march=native with the expanded flags. I find that saner. That said, pump mode is the only significant change in late distcc history. And it's broken anyway, and I doubt anybody's going to fix it since the failures are pretty random and hard to reproduce. -- Best regards, Michał Górny pgpNKxnYyGjOn.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
On Sat, 24 Jan 2015 12:03:27 +0100 Michał Górny wrote: Dnia 2015-01-21, o godz. 11:05:34 Michał Górny mgo...@gentoo.org napisał(a): Parent namespace solution - The alternative is to wrap distcc so that parent network namespace is regained. Sadly, for a start this requires SYS_CAP_ADMIN and access to parent's /proc entry (so root privileges). We obviously don't want to lose userpriv, so we have to do with setuid wrapper. Of course, setuid wrappers need to be written in C, so we need to introduce C compiled executable in Portage. Then we need to take some special care for it not to be abused, e.g. to access restricted network namespaces of other processes. I've started to work on this, and I have all the setup.py code ready. Now it's just the C program and wrapping it all in Portage. The only remaining big issue is how to pass the correct namespace (or PID) to the distcc process, assuming it may be a deep child of the ebuild process. Possibilities: 1. Environment variables. Can't do because: a. one of the parent processes may scrub the environment, b. malicious process may override the variable, giving subprocess access to restricted system network namespace. 2. Files. Can't do because we would have to pass the path somehow (see 1.), and if we go for static path, we have collisions between multiple emerge processes. If we're not going to use PID isolation in future, PID-dependent static file name may be used, where PID belongs to emerge. This should solve the collision issue. 3. Parent process hierarchy. Seems fragile, requires a lot of scanning on every spawn and may end up in the wrong namespace if a subprocess calls setsid() and user put emerge in a separate namespace. Why not to use cgroups here? 4. IPv4 socket. While it may seem a real dumb idea, it fits the case perfectly since it is only run if process is network-namespaced. We can surely reserve a static port early and run a daemon outputting the correct PID. Since the port is namespaced, every Portage process can have its own PID-daemon, and I don't know if there's any way of overriding the result from parent process. Looks fine, though the very idea to have extra daemons is unpleasant. Best regards, Andrew Savchenko pgpx4R3Fllh21.pgp Description: PGP signature
Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
Hi, On Sun, 25 Jan 2015 14:59:01 +0100 Michał Górny wrote: Dnia 2015-01-21, o godz. 11:05:34 Michał Górny mgo...@gentoo.org napisał(a): Generic proxy solution -- The simplest solution so far seems to be setting a generic SOCKS proxy inside the build environment, and wrapping distcc so that it will use it for network access. Unless we do some extra magic which don't want to do, this means that other apps can also abuse the proxy to reach outside sandbox. However, network-sandbox is not really a security feature, so I don't think that is important. At least as long as we don't export it globally :). Of course, software is a problem. We'd need at least some SOCKS server for Portage (at least a very simple one), and as far as I'm aware distcc does not support SOCKS directly, so tsocks in addition to that. So finally went this way instead. I still don't understand why. This solution: 1) is intrusive, it requires patching distcc and upstream as good as dead (see below); 2) will require a _separate_ solution for icecream and thus a double effort; 3) adds additional latency for distcc network path, which is undesirable. Parent namespace solution looks like the most reasonable for me based on both arguments above and years of heavy distcc usage experience. [2]:https://code.google.com/p/distcc/issues/detail?id=149 Chances to have this upstream are close to zero. If it is not dead, it is very close to it. Number of bugs and patches is accumulating without any response. No releases from 2011. Probably someone should fork it... Distcc has a problem with -march=native right now: it just falls back to local compilation if encountered it. I sent them a patch 1.5 years ago [1] and still no reply... It also requires some patches for successful cross-compilation when plain gcc is invoked by the client. (I have patches for amd64 - x86 only and they may broke pump mode (never used it anyway), thus I haven't send them upstream. [1] https://groups.google.com/forum/#!topic/distcc-patches/eeP-9pTgz7E In short, this patch expands native argument using gcc output, caches result (based on fingerprint of compiler being invoked) and sends expanded string to distcc servers. It is in my overlay (bircoph) if someone is interested. Works fine for me all these time. Best regards, Andrew Savchenko pgp2mF3GZuotz.pgp Description: PGP signature
RE: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
Hi, Michał Górny wrote: I see two generic approaches possible here: 1. proxying distcc from within the build environment, or 2. moving distcc-spawned processes back to parent's namespace. distcc client/server solution - The most obvious solution to me is to employ a client/server model where a system-wide daemon is running, parsing /etc/distcc/hosts and doing all the network activity. [...] It is not only distcc. Please don't forget things like sys-devel/icecream. -Thomas
Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
Dnia 2015-01-21, o godz. 11:05:34 Michał Górny mgo...@gentoo.org napisał(a): Generic proxy solution -- The simplest solution so far seems to be setting a generic SOCKS proxy inside the build environment, and wrapping distcc so that it will use it for network access. Unless we do some extra magic which don't want to do, this means that other apps can also abuse the proxy to reach outside sandbox. However, network-sandbox is not really a security feature, so I don't think that is important. At least as long as we don't export it globally :). Of course, software is a problem. We'd need at least some SOCKS server for Portage (at least a very simple one), and as far as I'm aware distcc does not support SOCKS directly, so tsocks in addition to that. So finally went this way instead. I've implemented a simple SOCKSv5 server over UNIX sockets [1] and wrote a patch adding SOCKSv5 support to distcc [2,3]. With the two patches, everything works perfectly for me :). [1]:http://article.gmane.org/gmane.linux.gentoo.portage.devel/5142 [2]:https://code.google.com/p/distcc/issues/detail?id=149 [3]:https://bugs.gentoo.org/show_bug.cgi?id=537616 -- Best regards, Michał Górny pgpuy8pRvnWQr.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
Dnia 2015-01-22, o godz. 00:34:35
Luis Ressel ara...@aixah.de napisał(a):
On Wed, 21 Jan 2015 10:38:20 -0500
Rich Freeman ri...@gentoo.org wrote:
On Wed, Jan 21, 2015 at 10:00 AM, Alexis Ballier
aball...@gentoo.org wrote:
On Wed, 21 Jan 2015 11:05:34 +0100
Michał Górny mgo...@gentoo.org wrote:
Hello, developers.
As you may recall, the main blocker for wide-establishment of
FEATURES=network-sandbox prohibiting network access within the
build environment is distcc. Since all connectivity is disabled,
distcc can no longer reach other distcc servers and build
efficiently. I therefore find it important to figure out a
solution.
I see two generic approaches possible here:
1. proxying distcc from within the build environment, or
2. moving distcc-spawned processes back to parent's namespace.
I haven't followed this at all, so this might be very stupid:
Isn't it possible to whitelist distcc hosts ?
That would only work with a proxy of some kind.
A process running in a separate network namespace doesn't see any
network interfaces. It can't even get as far as iptables/etc for some
kind of filtering. Now, you could define an interface in the new
namespace, and then bridge that to the host network, and then apply
iptables rules to it.
Your last sentence mentions a nice possibility:
1) Connect the sandbox network namespace to the global namespace (using
a veth pair?)
2) Enable forwarding (if I understand it right, it's even possible to
do this for individual interfaces instead of globally, using
/proc/sys/net/ipv{4,6}/conf/veth0 )
3) Set up suitable rules in the netfiler FORWARD chain to ensure only
distcc gets through
4) Set up SNAT or MASQUERADE in netfilter's nat table
5) There you go!
This is beautiful because is doesn't require any userland proxies, but
of course, it would be difficult to set up in an automated fashion. So
Yes, and it involves changing host's configuration which I would really
like to avoid. It should be something that can be done purely
in network namespace of the build, and securely, and without having to
parse distcc configuration, and without extra kernel features...
--
Best regards,
Michał Górny
pgpHL5PzVYqfa.pgp
Description: OpenPGP digital signature
Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
Dnia 2015-01-21, o godz. 11:05:34 Michał Górny mgo...@gentoo.org napisał(a): Parent namespace solution - The alternative is to wrap distcc so that parent network namespace is regained. Sadly, for a start this requires SYS_CAP_ADMIN and access to parent's /proc entry (so root privileges). We obviously don't want to lose userpriv, so we have to do with setuid wrapper. Of course, setuid wrappers need to be written in C, so we need to introduce C compiled executable in Portage. Then we need to take some special care for it not to be abused, e.g. to access restricted network namespaces of other processes. I've started to work on this, and I have all the setup.py code ready. Now it's just the C program and wrapping it all in Portage. The only remaining big issue is how to pass the correct namespace (or PID) to the distcc process, assuming it may be a deep child of the ebuild process. Possibilities: 1. Environment variables. Can't do because: a. one of the parent processes may scrub the environment, b. malicious process may override the variable, giving subprocess access to restricted system network namespace. 2. Files. Can't do because we would have to pass the path somehow (see 1.), and if we go for static path, we have collisions between multiple emerge processes. 3. Parent process hierarchy. Seems fragile, requires a lot of scanning on every spawn and may end up in the wrong namespace if a subprocess calls setsid() and user put emerge in a separate namespace. 4. IPv4 socket. While it may seem a real dumb idea, it fits the case perfectly since it is only run if process is network-namespaced. We can surely reserve a static port early and run a daemon outputting the correct PID. Since the port is namespaced, every Portage process can have its own PID-daemon, and I don't know if there's any way of overriding the result from parent process. Any other ideas? -- Best regards, Michał Górny pgpg1IjNH_Pcv.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
Dnia 2015-01-21, o godz. 16:00:55 Alexis Ballier aball...@gentoo.org napisał(a): On Wed, 21 Jan 2015 11:05:34 +0100 Michał Górny mgo...@gentoo.org wrote: Hello, developers. As you may recall, the main blocker for wide-establishment of FEATURES=network-sandbox prohibiting network access within the build environment is distcc. Since all connectivity is disabled, distcc can no longer reach other distcc servers and build efficiently. I therefore find it important to figure out a solution. I see two generic approaches possible here: 1. proxying distcc from within the build environment, or 2. moving distcc-spawned processes back to parent's namespace. [...] Any other ideas? I haven't followed this at all, so this might be very stupid: Isn't it possible to whitelist distcc hosts ? No because the child process is completely disconnected from parent's network stack. It has only a brand new loopback that's even separate from host's loopback. -- Best regards, Michał Górny pgp2m3XoDv9ky.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
On Wed, Jan 21, 2015 at 10:00 AM, Alexis Ballier aball...@gentoo.org wrote: On Wed, 21 Jan 2015 11:05:34 +0100 Michał Górny mgo...@gentoo.org wrote: Hello, developers. As you may recall, the main blocker for wide-establishment of FEATURES=network-sandbox prohibiting network access within the build environment is distcc. Since all connectivity is disabled, distcc can no longer reach other distcc servers and build efficiently. I therefore find it important to figure out a solution. I see two generic approaches possible here: 1. proxying distcc from within the build environment, or 2. moving distcc-spawned processes back to parent's namespace. I haven't followed this at all, so this might be very stupid: Isn't it possible to whitelist distcc hosts ? That would only work with a proxy of some kind. A process running in a separate network namespace doesn't see any network interfaces. It can't even get as far as iptables/etc for some kind of filtering. Now, you could define an interface in the new namespace, and then bridge that to the host network, and then apply iptables rules to it. I will second Michael's comment that distcc support shouldn't really be required for making this feature a mainstream one. I'm all for getting distcc working with this, but there is no reason the 98% of Gentoo users who don't use distcc can't benefit from network isolation while we wait to get it working for everybody else. -- Rich
Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
Dnia 2015-01-21, o godz. 09:40:41 Michael Orlitzky m...@gentoo.org napisał(a): On 01/21/2015 05:05 AM, Michał Górny wrote: Hello, developers. As you may recall, the main blocker for wide-establishment of FEATURES=network-sandbox prohibiting network access within the build environment is distcc. Since all connectivity is disabled, distcc can no longer reach other distcc servers and build efficiently. I therefore find it important to figure out a solution. ... Any other ideas? We probably won't avoid the fact that some features are incompatible forever. Why not just let the combination be broken? Distcc isn't enabled by default, so having network-sandbox enabled won't immediately cause problems. But *I* want to use both! -- Best regards, Michał Górny pgp_4y9jYT8fr.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
On Wed, 21 Jan 2015 10:38:20 -0500
Rich Freeman ri...@gentoo.org wrote:
On Wed, Jan 21, 2015 at 10:00 AM, Alexis Ballier
aball...@gentoo.org wrote:
On Wed, 21 Jan 2015 11:05:34 +0100
Michał Górny mgo...@gentoo.org wrote:
Hello, developers.
As you may recall, the main blocker for wide-establishment of
FEATURES=network-sandbox prohibiting network access within the
build environment is distcc. Since all connectivity is disabled,
distcc can no longer reach other distcc servers and build
efficiently. I therefore find it important to figure out a
solution.
I see two generic approaches possible here:
1. proxying distcc from within the build environment, or
2. moving distcc-spawned processes back to parent's namespace.
I haven't followed this at all, so this might be very stupid:
Isn't it possible to whitelist distcc hosts ?
That would only work with a proxy of some kind.
A process running in a separate network namespace doesn't see any
network interfaces. It can't even get as far as iptables/etc for some
kind of filtering. Now, you could define an interface in the new
namespace, and then bridge that to the host network, and then apply
iptables rules to it.
Your last sentence mentions a nice possibility:
1) Connect the sandbox network namespace to the global namespace (using
a veth pair?)
2) Enable forwarding (if I understand it right, it's even possible to
do this for individual interfaces instead of globally, using
/proc/sys/net/ipv{4,6}/conf/veth0 )
3) Set up suitable rules in the netfiler FORWARD chain to ensure only
distcc gets through
4) Set up SNAT or MASQUERADE in netfilter's nat table
5) There you go!
This is beautiful because is doesn't require any userland proxies, but
of course, it would be difficult to set up in an automated fashion. So
my proposal would be just to stay with the status quo, and document the
above in the wiki for those who really want to use both network-sandbox
and distcc despite the hassle.
Regards,
Luis Ressel
Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
On Wed, 21 Jan 2015 11:05:34 +0100 Michał Górny mgo...@gentoo.org wrote: Hello, developers. As you may recall, the main blocker for wide-establishment of FEATURES=network-sandbox prohibiting network access within the build environment is distcc. Since all connectivity is disabled, distcc can no longer reach other distcc servers and build efficiently. I therefore find it important to figure out a solution. I see two generic approaches possible here: 1. proxying distcc from within the build environment, or 2. moving distcc-spawned processes back to parent's namespace. [...] Any other ideas? I haven't followed this at all, so this might be very stupid: Isn't it possible to whitelist distcc hosts ? Alexis.
Re: [gentoo-dev] Figuring out the solution to in-network-sandbox distcc
On 01/21/2015 05:05 AM, Michał Górny wrote: Hello, developers. As you may recall, the main blocker for wide-establishment of FEATURES=network-sandbox prohibiting network access within the build environment is distcc. Since all connectivity is disabled, distcc can no longer reach other distcc servers and build efficiently. I therefore find it important to figure out a solution. ... Any other ideas? We probably won't avoid the fact that some features are incompatible forever. Why not just let the combination be broken? Distcc isn't enabled by default, so having network-sandbox enabled won't immediately cause problems. We can document that, when enabling distcc, you have to set FEATURES=-network-sandbox. No one will read it, so that leaves the question of what to do when both are enabled. 1. Break normally. We can point people to the man page. 2. Disable network-sandbox and emit a warning. 3. Disable distcc and emit a warning. The third is my favorite, since it's the non-default feature and the warning will be more helpful: to enable distcc, you must add -network-sandbox to your FEATURES in make.conf... If the point of network-sandbox is to catch mistakes, having it enabled for 95% of users is just as good as 100%.
