Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Rémi Cardona]

Le mardi 20 août 2013 à 12:26 +0200, Michał Górny a écrit :
> 3. FEATURES=ipc-sandbox
> 
> Requires: CONFIG_NAMESPACES, CONFIG_IPC_NS
> 
> Applies to: src_*
> 
> This one separates the ebuild's *nix IPC stuff from host. This includes
> semaphores, shared memory etc. Similarly to network-sandbox, this could
> prevent ebuilds from communicating with some production servers.
> 
> But honestly, I have no idea if anything really does it or relies on it.
> I doubt this could break something but it's worth testing.

This could impact ebuilds using the virtualx eclass, depending on how
the launched xvfb/xorg server is launched. It'd be interesting to test
the impact.

Other than that, it looks like really sweet stuff.

Cheers,

Rémi

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Michał Górny]

Dnia 2013-08-21, o godz. 22:05:21
Albert Hopkins  napisał(a):

> This sounds like cool stuff... I wonder if this could be a step towards
> unprivileged users being able to use portage for user-installed apps.

I doubt it. Especially that those features actually require root
privileges to be enabled :). (or CAP_SYS_ADMIN)

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Sergey Popov]

22.08.2013 06:05, Albert Hopkins пишет:
> This sounds like cool stuff... I wonder if this could be a step towards
> unprivileged users being able to use portage for user-installed apps.
> 

Try Prefix[1], it works very well in some cases ;-)


[1] - http://www.gentoo.org/proj/en/gentoo-alt/prefix/

-- 
Best regards, Sergey Popov
Gentoo developer
Gentoo Desktop Effects project lead
Gentoo Qt project lead
Gentoo Proxy maintainers project lead



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Rich Freeman]

On Wed, Aug 21, 2013 at 10:05 PM, Albert Hopkins  wrote:
> This sounds like cool stuff... I wonder if this could be a step towards
> unprivileged users being able to use portage for user-installed apps.

Sounds like Prefix, lite?

Rich

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Albert Hopkins]

This sounds like cool stuff... I wonder if this could be a step towards
unprivileged users being able to use portage for user-installed apps.

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Paweł Hajdan, Jr.]

On 8/20/13 3:26 AM, Michał Górny wrote:
> I've added a few new fancy features for Gentoo developers to portage
> git. Sadly, since Zac isn't planning another release until 2.2.0 goes
> stable, you need to switch to - to use them. But I say to you, it's
> worth the hassle.
> 
> I'd really appreciate some testing and feedback. Thanks.

The new features sound great! I was thinking for some time it'd be good
to have them.

Thank you for adding them. This is really appreciated.

Paweł




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Alex Xu]

On 20/08/13 11:42 AM, Ian Stakenvicius wrote:
> It's a feature; all features are optional.  It's just not going to be
> able to be enabled along with FEATURES="distcc" is all.  I'm sure we
> have other features that collide with one-another too, so i don't see
> this being a big issue.

FEATURES="nostrip splitdebug" neither makes sense nor works.

> ... similarly, though, i wonder if this would cause issues with i.e.
> diskless systems or others, that use nfs-mounts for /var/tmp/portage ..?

I imagine that that should work fine, since the NFS client is
implemented (usually) in the kernel.

FUSE mounts *might* be an issue, but I think they should be fine too
because the handling process is outside of the sandbox.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Ian Stakenvicius]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 20/08/13 11:30 AM, Douglas Freed wrote:
> On Aug 20, 2013 11:20 AM, "Michał Górny"  > wrote:
>> 
>> Dnia 2013-08-20, o godz. 11:04:35 Alexis Ballier
>> mailto:aball...@gentoo.org>>
> napisał(a):
>> 
>>> On Tue, 20 Aug 2013 12:26:03 +0200 Michał Górny
>>> mailto:mgo...@gentoo.org>> wrote:
 
 2. FEATURES=network-sandbox
 
>>> 
>>> does distcc work with this ?
>> 
>> You could say that. It just can't connect to any other host :).
>> 
>> We may try to handle this somehow but I can't immediately think
>> of any sane way of 'escaping' the sandbox.
> 
> You could do it the same way as LXC does, with a virtual interface
> which is then NAT-ed to the real network interface, but I'm not
> sure I'd consider this sane.  The overhead required to set this up
> on every execution of gcc, let alone the modifications needed for
> NAT, pretty much makes rules this out completely.  You might be
> able to exploit iptables and ip6tables to allow only distcc to
> communicate out, but that's still painful and is a hack at best.
> 
> -Doug
> 

It's a feature; all features are optional.  It's just not going to be
able to be enabled along with FEATURES="distcc" is all.  I'm sure we
have other features that collide with one-another too, so i don't see
this being a big issue.

... similarly, though, i wonder if this would cause issues with i.e.
diskless systems or others, that use nfs-mounts for /var/tmp/portage ..?
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.20 (GNU/Linux)

iF4EAREIAAYFAlITjmgACgkQ2ugaI38ACPBf2QEAvr1vWy6hDQjvZga1ijS2Zqf9
bnaDrxjvdIo8gh+xtooA+QFxZHLiH83KCgKGLuxYMIjpjWo1pHe+Qcy20s8woKMj
=74dq
-END PGP SIGNATURE-

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Douglas Freed]

On Aug 20, 2013 11:20 AM, "Michał Górny"  wrote:
>
> Dnia 2013-08-20, o godz. 11:04:35
> Alexis Ballier  napisał(a):
>
> > On Tue, 20 Aug 2013 12:26:03 +0200
> > Michał Górny  wrote:
> > >
> > > 2. FEATURES=network-sandbox
> > >
> >
> > does distcc work with this ?
>
> You could say that. It just can't connect to any other host :).
>
> We may try to handle this somehow but I can't immediately think of any
> sane way of 'escaping' the sandbox.

You could do it the same way as LXC does, with a virtual interface which is
then NAT-ed to the real network interface, but I'm not sure I'd consider
this sane.  The overhead required to set this up on every execution of gcc,
let alone the modifications needed for NAT, pretty much makes rules this
out completely.  You might be able to exploit iptables and ip6tables to
allow only distcc to communicate out, but that's still painful and is a
hack at best.

-Doug

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Michał Górny]

Dnia 2013-08-20, o godz. 11:04:35
Alexis Ballier  napisał(a):

> On Tue, 20 Aug 2013 12:26:03 +0200
> Michał Górny  wrote:
> > 
> > 2. FEATURES=network-sandbox
> > 
> 
> does distcc work with this ?

You could say that. It just can't connect to any other host :).

We may try to handle this somehow but I can't immediately think of any
sane way of 'escaping' the sandbox.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Douglas Freed]

On Aug 20, 2013 10:58 AM, "Rick "Zero_Chaos" Farina" 
wrote:
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 08/20/2013 06:26 AM, Michał Górny wrote:
> > Hello, fellow developers.
> >
> > I've added a few new fancy features for Gentoo developers to portage
> > git. Sadly, since Zac isn't planning another release until 2.2.0 goes
> > stable, you need to switch to - to use them. But I say to you, it's
> > worth the hassle.
>
> Any idea on an eta for that?
>
> This is some fantastic looking stuff.  I'm really looking forward to
> using these features.
>
> Thanks,
> Zero

Based on the 30 day rule of thumb, and assuming no critical bugs, no sooner
than September 11th, and eix 0.29.3's stabilization (30 day rule of thumb
here is September 6th) can be considered a blocker, because it has several
fixes for behavior that has changed between 2.1 and 2.2 that causes
problems for eix users (example: bug 478320 [1]).  I participated in the
discussion about this very question a few days ago in #gentoo-portage on
IRC.

-Doug

[1]: https://bugs.gentoo.org/show_bug.cgi?id=478320

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Alexis Ballier]

On Tue, 20 Aug 2013 12:26:03 +0200
Michał Górny  wrote:
> 
> 2. FEATURES=network-sandbox
> 

does distcc work with this ?

Alexis.

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Rick "Zero_Chaos" Farina]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 08/20/2013 06:26 AM, Michał Górny wrote:
> Hello, fellow developers.
> 
> I've added a few new fancy features for Gentoo developers to portage
> git. Sadly, since Zac isn't planning another release until 2.2.0 goes
> stable, you need to switch to - to use them. But I say to you, it's
> worth the hassle.

Any idea on an eta for that?

This is some fantastic looking stuff.  I'm really looking forward to
using these features.

Thanks,
Zero
> 
> The features are off by default since they need proper testing and can
> break a lot of ebuilds. And FEATURES=network-sandbox does.
> 
> It should be noted that all of the features follow the systemd idea of
> supporting Linux only and require fancy kernel features.
> 
> 
> The following new FEATURES have been added:
> 
> 1. FEATURES=cgroup
> 
> Requires: CONFIG_CGROUPS
> 
> Applies to: all src_* phases
> 
> Enables long-awaited cgroup support in portage. Each ebuild is confined
> within a control group and all spawned processes are tracked. Once
> the phase exits, all remaining orphans are killed.
> 
> This helps especially with multiprocessing/multibuild stuff and some
> test phases that need to spawn servers. It ensures that portage does
> not leave any orphans that would otherwise need to be separately
> tracked and killed.
> 
> Control groups are applied to src_* phases only, since we expect that
> pkg_postinst() may restart external daemons, and those could end up
> being attached to the cgroup.
> 
> I doubt this could break something.
> 
> 
> 2. FEATURES=network-sandbox
> 
> Requires: CONFIG_NAMESPACES, CONFIG_NET_NS
> 
> Applies to: src_* except for src_unpack
> 
> This one uses the unshare() syscall to detach the build process from
> host's network stack. This effectively means that each of the listed
> phases will be able only to access a detached, 'local' loopback
> interface and nothing else.
> 
> This has a few implications. First of all, ebuilds that used to access
> the Internet won't be able to do that anymore. In the Python world,
> this would mean that some packages will start to fail properly instead
> of downloading missing dependencies. It will also break or skip all
> the tests that rely on the network being available.
> 
> Secondly, this will prevent any kind of communication between host
> network and ebuild, including services running on 127.0.0.1. That is,
> ebuild will no longer be able to access production services running
> on the host. This affects e.g. old mongodb frontend ebuilds which used
> to run tests on the live database server (yep, create and delete
> databases there).
> 
> Thirdly, this will prevent the daemons spawned within ebuild from being
> publicly accessible. That is, if test phase spawns some kind of TCP/IP
> server, even local users won't be able to connect to it (outside of
> the namespace). This should improve the security.
> 
> This does not apply to pkg_* phases where networking may be needed for
> some kind of IPC, and src_unpack where it is used for VCS fetching. If
> we introduce separate src_fetch in a future EAPI, the exclude will
> move there.
> 
> This one's going to trigger a lot of breakage in ebuilds. Therefore,
> I'd appreciate if developers started using it early and fixing
> the ebuilds.
> 
> 
> 3. FEATURES=ipc-sandbox
> 
> Requires: CONFIG_NAMESPACES, CONFIG_IPC_NS
> 
> Applies to: src_*
> 
> This one separates the ebuild's *nix IPC stuff from host. This includes
> semaphores, shared memory etc. Similarly to network-sandbox, this could
> prevent ebuilds from communicating with some production servers.
> 
> But honestly, I have no idea if anything really does it or relies on it.
> I doubt this could break something but it's worth testing.
> 
> 
> I'd really appreciate some testing and feedback. Thanks.
> 

-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSE4Q3AAoJEKXdFCfdEflKB+wP/3ZYGqcdOz+ojIx5+7DT+OBY
qJEz8oIDI73P/Bn6VsLAeq2bO5RaLLB1rIqB4cxbIncxLb/Vwsuk6VgryPNxaH7C
+2ctSZEaUSo+2Kg2sVrHxd+n0kC19GrZOXr9y5HvKSWofMYP2My67xAoRrg0/qe8
7A1jjMKbh29L+rt8krnSNYHt9EoUe7soeNEcCcVYkyRUkoQnkOy0qqBBvAUB8Ggy
Gt0M3zZ9aHYTCcFbVv/JW7kntMt601AMMfsz/zK2dcBu2nyPCz1oOFgsfz23qPTw
GYh2iJxMekWihxU/fJHGJIlXnNxhRHxCB7tKojgYiREqbVcBEWSr9S7QoNKL9Lts
ZZ2ubSXuwjpcyGn4ih2lWdBFb7vvG5lKVoFpQimG1/sNLTgtLCejZ05IFvd08zyo
UNpp2/lZpJuBsGi8zXVhUlDz8Fe4itNBHjXq90LqWDsQlp3h4BmYjxdrygG5LPMt
+S/6hK24muE27FDEFLGEMNl71DZykETPq6YYFgqvQpmhgTVtDBUHCQ8L1UpQwBIE
PMwEnwTJtezsfKhPR+OTNBRTAVHRL4/MF7lL3wVrLHJ3wpyFVEec2ixJ3vK5ap6Q
TgPzMm2QhYS1ncCNCt4Gwxs0DyHuZrSrkWKLlWW8o6A2qWp3TxCeaEfkfca5nn/e
9IzX5hSKEbHlyHBmm467
=4jTs
-END PGP SIGNATURE-

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Sergey Popov]

20.08.2013 14:26, Michał Górny пишет:
> Hello, fellow developers.
> 
> I've added a few new fancy features for Gentoo developers to portage
> git. Sadly, since Zac isn't planning another release until 2.2.0 goes
> stable, you need to switch to - to use them. But I say to you, it's
> worth the hassle.
> 
> The features are off by default since they need proper testing and can
> break a lot of ebuilds. And FEATURES=network-sandbox does.
> 
> It should be noted that all of the features follow the systemd idea of
> supporting Linux only and require fancy kernel features.

Great features, i will definitely try them!

-- 
Best regards, Sergey Popov
Gentoo developer
Gentoo Desktop Effects project lead
Gentoo Qt project lead
Gentoo Proxy maintainers project lead



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Markos Chandras]

On 20 August 2013 11:26, Michał Górny  wrote:
> Hello, fellow developers.
>
> I've added a few new fancy features for Gentoo developers to portage
> git. Sadly, since Zac isn't planning another release until 2.2.0 goes
> stable, you need to switch to - to use them. But I say to you, it's
> worth the hassle.
>
> The features are off by default since they need proper testing and can
> break a lot of ebuilds. And FEATURES=network-sandbox does.
>
> It should be noted that all of the features follow the systemd idea of
> supporting Linux only and require fancy kernel features.
>
>

Awesome! Thanks a lot!

-- 
Regards,
Markos Chandras - Gentoo Linux Developer
http://dev.gentoo.org/~hwoarang

[gentoo-dev] New developer features in portage: cgroup, network-sandbox, ipc-sandbox

[Michał Górny]

Hello, fellow developers.

I've added a few new fancy features for Gentoo developers to portage
git. Sadly, since Zac isn't planning another release until 2.2.0 goes
stable, you need to switch to - to use them. But I say to you, it's
worth the hassle.

The features are off by default since they need proper testing and can
break a lot of ebuilds. And FEATURES=network-sandbox does.

It should be noted that all of the features follow the systemd idea of
supporting Linux only and require fancy kernel features.


The following new FEATURES have been added:

1. FEATURES=cgroup

Requires: CONFIG_CGROUPS

Applies to: all src_* phases

Enables long-awaited cgroup support in portage. Each ebuild is confined
within a control group and all spawned processes are tracked. Once
the phase exits, all remaining orphans are killed.

This helps especially with multiprocessing/multibuild stuff and some
test phases that need to spawn servers. It ensures that portage does
not leave any orphans that would otherwise need to be separately
tracked and killed.

Control groups are applied to src_* phases only, since we expect that
pkg_postinst() may restart external daemons, and those could end up
being attached to the cgroup.

I doubt this could break something.


2. FEATURES=network-sandbox

Requires: CONFIG_NAMESPACES, CONFIG_NET_NS

Applies to: src_* except for src_unpack

This one uses the unshare() syscall to detach the build process from
host's network stack. This effectively means that each of the listed
phases will be able only to access a detached, 'local' loopback
interface and nothing else.

This has a few implications. First of all, ebuilds that used to access
the Internet won't be able to do that anymore. In the Python world,
this would mean that some packages will start to fail properly instead
of downloading missing dependencies. It will also break or skip all
the tests that rely on the network being available.

Secondly, this will prevent any kind of communication between host
network and ebuild, including services running on 127.0.0.1. That is,
ebuild will no longer be able to access production services running
on the host. This affects e.g. old mongodb frontend ebuilds which used
to run tests on the live database server (yep, create and delete
databases there).

Thirdly, this will prevent the daemons spawned within ebuild from being
publicly accessible. That is, if test phase spawns some kind of TCP/IP
server, even local users won't be able to connect to it (outside of
the namespace). This should improve the security.

This does not apply to pkg_* phases where networking may be needed for
some kind of IPC, and src_unpack where it is used for VCS fetching. If
we introduce separate src_fetch in a future EAPI, the exclude will
move there.

This one's going to trigger a lot of breakage in ebuilds. Therefore,
I'd appreciate if developers started using it early and fixing
the ebuilds.


3. FEATURES=ipc-sandbox

Requires: CONFIG_NAMESPACES, CONFIG_IPC_NS

Applies to: src_*

This one separates the ebuild's *nix IPC stuff from host. This includes
semaphores, shared memory etc. Similarly to network-sandbox, this could
prevent ebuilds from communicating with some production servers.

But honestly, I have no idea if anything really does it or relies on it.
I doubt this could break something but it's worth testing.


I'd really appreciate some testing and feedback. Thanks.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature