Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-05 Thread Philipp Marek
Hi Daniel,

  Yes, there's some semantic meaning at that level. But this level already 
  exists at the current rootwrap caller site, too - and if that one can be 
  tricked to do something against image.img  rm -rf /, then the additional 
  layer can be tricked, too.
 
 No, that is really not correct. If you are passing full command strings to
 rootwrap then the caller can trick rootwrap into running commands with those
 shell metacharacter exploits.  If you have a formal API like the one I 
 describe
 and correctly implement it, there would be no shell involved at all. ie the
 nova-compute-worker program would directly invoke the system call
 
execve(/usr/bin/qemu-img, create, image.img  rm -rf /)
 
 and this would at worse create a file called  'image.img  rm -rf /' and
 *not* invoke the rm command as you get when you use shell.
 
 This is really just another example of why rootwrap/sudo as a concept is a
 bad idea. The shell should never be involved in executing any external
 commands that Nova/Neutron/etc need to run, because it is impractical to
 correctly validate shell commands anywhere in the stack. The only safe
 thing todo is to take shell out of the picture entirely.
that was just an example.

If cinder calls rootwrap with a list of arguments, and this is then called 
via sudo, there's no shell inbetween either.


But now say there's something else, like the bash functions bug, and 
some codepath runs a command in a way that sets the environment - then any 
grandchild processes might cause a breach.


I'm not arguing that *this exact string* is a problem.
My point is that the cinder/nova code already have as much information as 
possible, and *any* library abstraction can just remove some available 
data.
So, besides increasing the LOC, it won't get more secure.


I'm arguing that the rootwrap call sites need to be fixed, ie. they need to 
proof in some way that the arguments they pass on are sane - that's more 
or less the same thing that this new library would need to do too.



Regards,

Phil

-- 
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com :

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-05 Thread Steven Dake (stdake)


On 2/4/15, 10:24 AM, Daniel P. Berrange berra...@redhat.com wrote:

On Wed, Feb 04, 2015 at 09:10:06AM -0800, James E. Blair wrote:
 Thierry Carrez thie...@openstack.org writes:
 
  You make a good point when you mention traditional distro here. I
  would argue that containers are slightly changing the rules of the
  don't-run-as-root game.
 
  Solution (2) aligns pretty well with container-powered OpenStack
  deployments -- running compute nodes as root in a container (and
  embracing abovementioned simplicity/performance gains) sounds like a
  pretty strong combo.
 
 This sounds at least a little like a suggestion that containers are a
 substitute for the security provided by running non-root.  The security
 landscape around containers is complex, and while there are a lot of
 benefits, I believe the general consensus is that uid 0 processes should
 not be seen as fully isolated.
 
 From https://docs.docker.com/articles/security/ :
 
   Docker containers are, by default, quite secure; especially if you
   take care of running your processes inside the containers as
   non-privileged users (i.e., non-root).
 
 Which is not to say that using containers is not a good idea, but
 rather, if one does, one should avoid running as root (perhaps with
 capabilities), and use selinux (or similar).

Yep, I've seen attempts by some folks to run nova-compute and libvirtd
and QEMU inside a docker container. Because of the inherantly privileged
nature of what Nova/libvirt/qemu need to do, you end up having to share
all the host namespaces with the docker container, except for the
filesystem
namespace and even that you need to bind mount a bunch of stuff over. As
a result the container isn't really offerring any security benefit over
running the things outside the container. IOW the use of containers to
confine nova is only solving a managability problem rather than a security
problem.

Regards,
Daniel

Daniel,

Agree 100% - compute in containers is all about an atomic image-based
upgrade and downgrade process, not about solving security problems.
Ideally the services that are within containers are as secure as running
natively without containers on bare metal although this might be a bit of
assumption since docker does run with all Linux capabilities enabled.

Regards
-steve

-- 
|: http://berrange.com  -o-
http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o-
http://virt-manager.org :|
|: http://autobuild.org   -o-
http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-
http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-05 Thread Tristan Cacqueray
On 02/04/2015 01:13 PM, Clint Byrum wrote:
 Excerpts from Tristan Cacqueray's message of 2015-02-04 09:02:19 -0800:
 On 02/04/2015 06:57 AM, Daniel P. Berrange wrote:
 (4) I think that ultimately we need to ditch rootwrap and provide a proper
 privilege separated, formal RPC mechanism for each project.

 eg instead of having a rootwrap command, or rootwrap server attempting
 to validate safety of

 qemu-img create -f qcow2 
 /var/lib/nova/instances/instance1/disk.qcow2

 we should have a  nova-compute-worker daemon running as root, that accepts
 an RPC command from nova-compute running unprivileged. eg

 CreateImage(instane0001, qcow2, disk.qcow)

 This immediately makes it trivial to validate that we're not trying to
 trick qemu-img into overwriting some key system file.

 This is certainly alot more work than trying to patchup rootwrap, but
 it would provide a level of security that rootwrap can never achieve IMHO.


 This 4th idea sounds interesting, though we are assuming this new service
 running as root would be exempt of bug, especially if it uses the same
 libraries as non-root services... For example a major bug in python would
 give attacker direct root access while the rootwrap solution would in
 theory keep the intruder at the sudo level...

 
 I don't believe that anyone assumes the new service would be without
 bugs. 

I meant less bug than another solution. Such RPC service daemon will
eventually requires quite some code to be robust, which could lead to
more bugs.


 But just like the OpenSSH team saw years ago, privilege separation
 means that you can absolutely know what is running as root, and what is
 not. So when you decide to commit your resources to code audits, you
 _start_ with the things that run with elevated privileges.
 

Not quite sure to follow here... OpenSSH is using privilege separation after
authentication, e.g. the root-based process is doing the authentication while
the external data processing is done through an unprivileged process.

If I understand correctly Daniel's solution (4), it's to have a semantic on the
privileged actions to avoid mis-usage (like inject command in a sudo call).

Thus if we want to emulate OpenSSH design, the rpc call would also need to
carry authentication data in order to prevent unwanted activity. And the
rpc daemon would then need to enforce some kind of acl/policy.

The amount of code to be audited could arguably be higher with such design.



 For completeness, I'd like to propose a more long-term solution:

 (5) Get ride of root! Seriously, OpenStack could support security mechanism
 like SELinux or AppArmor in order to properly isolate service and let them
 run what they need to run.

 For what it worth, the underlying issue here is having a single almighty 
 super
 user: root and thus we should, at least, consider solution that remove the
 need of such powers (e.g. kernel module loading, ptrace or raw socket).

 
 We don't need a security module to drop all of those capabilities
 entirely and run as a hobbled root user. By my measure, this process for
 nova-compute would only need CAP_NET_ADMIN, CAP_SYS_ADMIN and CAP_KILL.
 These capabilities can be audited per-agent and even verified as needed
 simply by running integration tests without each one to see what breaks.

Capabilities might be a candidate as well, but they don't cover every cases
and lack granularity...

SECCOMP filters might be good enough too.


 Beside, as long as sensitive process are not contained at the system level,
 the attack surface for a non-root user is still very wide (e.g. system calls,
 setuid binaries, ipc, ...)


 While this might sounds impossible to implement upstream because it's too
 vendor specific or just because of other technicals difficulties,
 I guess it still deserves a mention in this thread.

 
 I think OpenStack can do its part by making privilege separation a
 reality for the security sensitive parts of OpenStack, and that will
 ease pressure to implement strong controls in any security modules the
 Linux distros ship with.
 
That would be a great step forward indeed. Having some kind of privilege
separation would surely help a lot to properly configure security modules.

Regards,
Tristan



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-05 Thread Nicolas Trangez
On Thu, 2015-02-05 at 08:27 -0500, Tristan Cacqueray wrote:
 Thus if we want to emulate OpenSSH design, the rpc call would also
 need to
 carry authentication data in order to prevent unwanted activity. And
 the
 rpc daemon would then need to enforce some kind of acl/policy.

Sounds a lot like what the DBus system bus implements/provides to me...
And it took quite some time to get that done.

(Note: I'm not proposing its usage in any way)

Nicolas


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Jeremy Stanley
On 2015-02-04 13:40:29 +0200 (+0200), Duncan Thomas wrote:
 4) Write a small daemon that runs as root, accepting commands over
 a unix domain socket or similar. Easier to audit, less code
 running as root.

http://git.openstack.org/cgit/openstack/oslo.rootwrap/tree/oslo_rootwrap/daemon.py

-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Jeremy Stanley
On 2015-02-04 18:38:16 +0200 (+0200), Duncan Thomas wrote:
 If I'm reading that correctly, it does not help with the filtering issues at
 all, since it needs exactly the same kind of filter. Daniel explained the
 concept far better than I.

I didn't mean to imply that it does, merely that it fits your rather
terse description of a daemon that runs as root, accepting commands
over a unix domain socket or similar.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Tristan Cacqueray
On 02/04/2015 06:57 AM, Daniel P. Berrange wrote:
 On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
 What solutions do we have ?

 (1) we could get our act together and audit and fix those filter
 definitions. Remove superfluous usage of root rights, make use of
 advanced filters for where we actually need them. We have been preaching
 for that at many many design summits. This is a lot of work though...
 There were such efforts in the past, but they were never completed for
 some types of nodes. Worse, the bad filter definitions kept coming back,
 since developers take shortcuts, reviewers may not have sufficient
 security awareness to detect crappy filter definitions, and I don't
 think we can design a gate test that would have such awareness.

 (2) bite the bullet and accept that some types of nodes actually need
 root rights for so many different things, they should just run as root
 anyway. I know a few distributions which won't be very pleased by such a
 prospect, but that would be a more honest approach (rather than claiming
 we provide efficient isolation when we really don't). An added benefit
 is that we could replace a number of shell calls by Python code, which
 would simplify the code and increase performance.

 (3) intermediary solution where we would run as the nova user but run
 sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
 That would leave it up to distros to choose between a blanket sudoer or
 maintain their own filtering rules. I think it's a bit hypocritical
 though (pretend the distros could filter if they wanted it, when we
 dropped the towel on doing that ourselves). I'm also not convinced it's
 more secure than solution 2, and it prevents from reducing the number of
 shell-outs, which I think is a worthy idea.

 In all cases I would not drop the baby with the bath water, and keep
 rootwrap for all the cases where root rights are needed on a very
 specific set of commands (like neutron, or nova's api-metadata). The
 daemon mode should address the performance issue for the projects making
 a lot of calls.
 
 
 (4) I think that ultimately we need to ditch rootwrap and provide a proper
 privilege separated, formal RPC mechanism for each project.
 
 eg instead of having a rootwrap command, or rootwrap server attempting
 to validate safety of
 
 qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2
 
 we should have a  nova-compute-worker daemon running as root, that accepts
 an RPC command from nova-compute running unprivileged. eg
 
 CreateImage(instane0001, qcow2, disk.qcow)
 
 This immediately makes it trivial to validate that we're not trying to
 trick qemu-img into overwriting some key system file.
 
 This is certainly alot more work than trying to patchup rootwrap, but
 it would provide a level of security that rootwrap can never achieve IMHO.
 

This 4th idea sounds interesting, though we are assuming this new service
running as root would be exempt of bug, especially if it uses the same
libraries as non-root services... For example a major bug in python would
give attacker direct root access while the rootwrap solution would in
theory keep the intruder at the sudo level...


For completeness, I'd like to propose a more long-term solution:

(5) Get ride of root! Seriously, OpenStack could support security mechanism
like SELinux or AppArmor in order to properly isolate service and let them
run what they need to run.

For what it worth, the underlying issue here is having a single almighty super
user: root and thus we should, at least, consider solution that remove the
need of such powers (e.g. kernel module loading, ptrace or raw socket).

Beside, as long as sensitive process are not contained at the system level,
the attack surface for a non-root user is still very wide (e.g. system calls,
setuid binaries, ipc, ...)


While this might sounds impossible to implement upstream because it's too
vendor specific or just because of other technicals difficulties,
I guess it still deserves a mention in this thread.

Best regards,
Tristan



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Daniel P. Berrange
On Wed, Feb 04, 2015 at 05:52:12PM +0100, Philipp Marek wrote:
 Here are my 2¢.
 
(1) we could get our act together and audit and fix those filter
definitions. Remove superfluous usage of root rights, make use of
advanced filters for where we actually need them. We have been 
preaching
for that at many many design summits. This is a lot of work though...
There were such efforts in the past, but they were never completed for
some types of nodes. Worse, the bad filter definitions kept coming 
back,
since developers take shortcuts, reviewers may not have sufficient
security awareness to detect crappy filter definitions, and I don't
think we can design a gate test that would have such awareness.
 Sounds like a lot of work... ongoing, too.
 
 
(2) bite the bullet and accept that some types of nodes actually need
root rights for so many different things, they should just run as root
anyway. I know a few distributions which won't be very pleased by such 
a
prospect, but that would be a more honest approach (rather than 
claiming
we provide efficient isolation when we really don't). An added benefit
is that we could replace a number of shell calls by Python code, which
would simplify the code and increase performance.
 Practical, but unsafe.
 
 I'd very much like to have some best-effort filter against bugs in my 
 programming - even more so during development.
 
 
  (4) I think that ultimately we need to ditch rootwrap and provide a proper
  privilege separated, formal RPC mechanism for each project.
 ...
  we should have a  nova-compute-worker daemon running as root, that accepts
  an RPC command from nova-compute running unprivileged. eg
 
  CreateImage(instane0001, qcow2, disk.qcow)
 ...
  This is certainly alot more work than trying to patchup rootwrap, but
  it would provide a level of security that rootwrap can never achieve IMHO.
 A lot of work, and if input sanitation didn't work in one piece of code, 
 why should it here?
 
 I think this only leads to _more_ work, without any real benefit.
 If we can't get the filters right the first round, we won't make it here 
 either.

The difference is that the API I illustrate here has *semantic* context
about the operation. In the API example the caller is not permitted to
provide a directory path - only the name of the instance and the name
of the disk image. The privileged nova-compute-worker program can thus
enforce exactly what directory the image is created in, and ensure it
doesn't clash with a disk image from another VM.  This kind of validation
is impractical when you are just given a 'qemu-img' command line args
with a full directory path, so there is no semantic conext for the privileged
rootwrap to know whether it is reasonable to create the disk image in that
particular location.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Daniel P. Berrange
On Wed, Feb 04, 2015 at 06:38:16PM +0200, Duncan Thomas wrote:
 If I'm reading that correctly, it does not help with the filtering issues
 at all, since it needs exactly the same kind of filter. Daniel explained
 the concept far better than I.

Yep, the only thing rootwrap daemon mode does is to remove the overhead
of spawning the rootwrap command. It does nothing to improve actual
security - it is still a chocolate teapot from that POV.

 On 4 February 2015 at 18:33, Jeremy Stanley fu...@yuggoth.org wrote:
 
  On 2015-02-04 13:40:29 +0200 (+0200), Duncan Thomas wrote:
   4) Write a small daemon that runs as root, accepting commands over
   a unix domain socket or similar. Easier to audit, less code
   running as root.
 
 
  http://git.openstack.org/cgit/openstack/oslo.rootwrap/tree/oslo_rootwrap/daemon.py
 

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Daniel P. Berrange
On Wed, Feb 04, 2015 at 06:05:16PM +0100, Philipp Marek wrote:
(4) I think that ultimately we need to ditch rootwrap and provide a 
proper
privilege separated, formal RPC mechanism for each project.
   ...
we should have a  nova-compute-worker daemon running as root, that 
accepts
an RPC command from nova-compute running unprivileged. eg
   
CreateImage(instane0001, qcow2, disk.qcow)
   ...
This is certainly alot more work than trying to patchup rootwrap, but
it would provide a level of security that rootwrap can never achieve 
IMHO.
   A lot of work, and if input sanitation didn't work in one piece of code, 
   why should it here?
   
   I think this only leads to _more_ work, without any real benefit.
   If we can't get the filters right the first round, we won't make it here 
   either.
  
  The difference is that the API I illustrate here has *semantic* context
  about the operation. In the API example the caller is not permitted to
  provide a directory path - only the name of the instance and the name
  of the disk image. The privileged nova-compute-worker program can thus
  enforce exactly what directory the image is created in, and ensure it
  doesn't clash with a disk image from another VM.  This kind of validation
  is impractical when you are just given a 'qemu-img' command line args
  with a full directory path, so there is no semantic conext for the 
  privileged
  rootwrap to know whether it is reasonable to create the disk image in that
  particular location.
 Sorry about being unclear.
 
 Yes, there's some semantic meaning at that level. But this level already 
 exists at the current rootwrap caller site, too - and if that one can be 
 tricked to do something against image.img  rm -rf /, then the additional 
 layer can be tricked, too.

No, that is really not correct. If you are passing full command strings to
rootwrap then the caller can trick rootwrap into running commands with those
shell metacharacter exploits.  If you have a formal API like the one I describe
and correctly implement it, there would be no shell involved at all. ie the
nova-compute-worker program would directly invoke the system call

   execve(/usr/bin/qemu-img, create, image.img  rm -rf /)

and this would at worse create a file called  'image.img  rm -rf /' and
*not* invoke the rm command as you get when you use shell.

This is really just another example of why rootwrap/sudo as a concept is a
bad idea. The shell should never be involved in executing any external
commands that Nova/Neutron/etc need to run, because it is impractical to
correctly validate shell commands anywhere in the stack. The only safe
thing todo is to take shell out of the picture entirely.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Jeremy Stanley
On 2015-02-04 11:58:03 +0100 (+0100), Thierry Carrez wrote:
[...]
 The second problem is the quality of the filter definitions. Rootwrap is
 a framework to enable isolation. It's only as good as the filters each
 project defines. Most of them rely on CommandFilters that do not check
 any argument, instead of using more powerful filters (which are arguably
 more painful to maintain). Developers routinely add filter definitions
 that basically remove any isolation that might have been there, like
 allowing blank dd, tee, chown or chmod.
[...]

This part is my biggest concern at the moment, from a vulnerability
management standpoint. I'm worried that it's an attractive nuisance
resulting in a false sense of security in its current state because
we're not calling this shortcoming out explicitly in documentation
(as far as I'm aware), and so we're opening our operators/users up
to unexpected risks and opening ourselves up to the possibility of a
slew of vulnerability reports because this mechanism doesn't provide
the level of protection it would seem to imply.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Philipp Marek
Here are my 2¢.

   (1) we could get our act together and audit and fix those filter
   definitions. Remove superfluous usage of root rights, make use of
   advanced filters for where we actually need them. We have been preaching
   for that at many many design summits. This is a lot of work though...
   There were such efforts in the past, but they were never completed for
   some types of nodes. Worse, the bad filter definitions kept coming back,
   since developers take shortcuts, reviewers may not have sufficient
   security awareness to detect crappy filter definitions, and I don't
   think we can design a gate test that would have such awareness.
Sounds like a lot of work... ongoing, too.


   (2) bite the bullet and accept that some types of nodes actually need
   root rights for so many different things, they should just run as root
   anyway. I know a few distributions which won't be very pleased by such a
   prospect, but that would be a more honest approach (rather than claiming
   we provide efficient isolation when we really don't). An added benefit
   is that we could replace a number of shell calls by Python code, which
   would simplify the code and increase performance.
Practical, but unsafe.

I'd very much like to have some best-effort filter against bugs in my 
programming - even more so during development.


 (4) I think that ultimately we need to ditch rootwrap and provide a proper
 privilege separated, formal RPC mechanism for each project.
...
 we should have a  nova-compute-worker daemon running as root, that accepts
 an RPC command from nova-compute running unprivileged. eg

 CreateImage(instane0001, qcow2, disk.qcow)
...
 This is certainly alot more work than trying to patchup rootwrap, but
 it would provide a level of security that rootwrap can never achieve IMHO.
A lot of work, and if input sanitation didn't work in one piece of code, 
why should it here?

I think this only leads to _more_ work, without any real benefit.
If we can't get the filters right the first round, we won't make it here 
either.


Regarding the idea of using containers ... take Cinder as an example.
If the cinder container can access *all* the VM data, why should someone 
bother to get *out* of the container? Everything that she wants is already 
here...
I'm not sure what the containers would buy us, but perhaps I just don't 
understand something here.


So, IMO, solution 1 (one) would be the way to go ... it gets to security 
asymptotically (and might never reach it), but at least it provides a bit 
of help.

And if the rootwrap filter specification would be linked to in the rootwrap 
config files, it would help newcomers to see the available syntax, instead 
of simply copying a bad example ;P



-- 
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com :

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread James E. Blair
Thierry Carrez thie...@openstack.org writes:

 You make a good point when you mention traditional distro here. I
 would argue that containers are slightly changing the rules of the
 don't-run-as-root game.

 Solution (2) aligns pretty well with container-powered OpenStack
 deployments -- running compute nodes as root in a container (and
 embracing abovementioned simplicity/performance gains) sounds like a
 pretty strong combo.

This sounds at least a little like a suggestion that containers are a
substitute for the security provided by running non-root.  The security
landscape around containers is complex, and while there are a lot of
benefits, I believe the general consensus is that uid 0 processes should
not be seen as fully isolated.

From https://docs.docker.com/articles/security/ :

  Docker containers are, by default, quite secure; especially if you
  take care of running your processes inside the containers as
  non-privileged users (i.e., non-root).

Which is not to say that using containers is not a good idea, but
rather, if one does, one should avoid running as root (perhaps with
capabilities), and use selinux (or similar).

-Jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Daniel P. Berrange
On Wed, Feb 04, 2015 at 09:10:06AM -0800, James E. Blair wrote:
 Thierry Carrez thie...@openstack.org writes:
 
  You make a good point when you mention traditional distro here. I
  would argue that containers are slightly changing the rules of the
  don't-run-as-root game.
 
  Solution (2) aligns pretty well with container-powered OpenStack
  deployments -- running compute nodes as root in a container (and
  embracing abovementioned simplicity/performance gains) sounds like a
  pretty strong combo.
 
 This sounds at least a little like a suggestion that containers are a
 substitute for the security provided by running non-root.  The security
 landscape around containers is complex, and while there are a lot of
 benefits, I believe the general consensus is that uid 0 processes should
 not be seen as fully isolated.
 
 From https://docs.docker.com/articles/security/ :
 
   Docker containers are, by default, quite secure; especially if you
   take care of running your processes inside the containers as
   non-privileged users (i.e., non-root).
 
 Which is not to say that using containers is not a good idea, but
 rather, if one does, one should avoid running as root (perhaps with
 capabilities), and use selinux (or similar).

Yep, I've seen attempts by some folks to run nova-compute and libvirtd
and QEMU inside a docker container. Because of the inherantly privileged
nature of what Nova/libvirt/qemu need to do, you end up having to share
all the host namespaces with the docker container, except for the filesystem
namespace and even that you need to bind mount a bunch of stuff over. As
a result the container isn't really offerring any security benefit over
running the things outside the container. IOW the use of containers to
confine nova is only solving a managability problem rather than a security
problem.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Philipp Marek
   (4) I think that ultimately we need to ditch rootwrap and provide a proper
   privilege separated, formal RPC mechanism for each project.
  ...
   we should have a  nova-compute-worker daemon running as root, that accepts
   an RPC command from nova-compute running unprivileged. eg
  
   CreateImage(instane0001, qcow2, disk.qcow)
  ...
   This is certainly alot more work than trying to patchup rootwrap, but
   it would provide a level of security that rootwrap can never achieve IMHO.
  A lot of work, and if input sanitation didn't work in one piece of code, 
  why should it here?
  
  I think this only leads to _more_ work, without any real benefit.
  If we can't get the filters right the first round, we won't make it here 
  either.
 
 The difference is that the API I illustrate here has *semantic* context
 about the operation. In the API example the caller is not permitted to
 provide a directory path - only the name of the instance and the name
 of the disk image. The privileged nova-compute-worker program can thus
 enforce exactly what directory the image is created in, and ensure it
 doesn't clash with a disk image from another VM.  This kind of validation
 is impractical when you are just given a 'qemu-img' command line args
 with a full directory path, so there is no semantic conext for the privileged
 rootwrap to know whether it is reasonable to create the disk image in that
 particular location.
Sorry about being unclear.

Yes, there's some semantic meaning at that level. But this level already 
exists at the current rootwrap caller site, too - and if that one can be 
tricked to do something against image.img  rm -rf /, then the additional 
layer can be tricked, too.


I'm trying to get at the point
  everything that can be forgot to check at the rootwrap call site now,
  can be forgotten in the additional API too.

So let's get the current call sites tight, and we're done.  (Ha!)


-- 
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com :

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Duncan Thomas
If I'm reading that correctly, it does not help with the filtering issues
at all, since it needs exactly the same kind of filter. Daniel explained
the concept far better than I.

On 4 February 2015 at 18:33, Jeremy Stanley fu...@yuggoth.org wrote:

 On 2015-02-04 13:40:29 +0200 (+0200), Duncan Thomas wrote:
  4) Write a small daemon that runs as root, accepting commands over
  a unix domain socket or similar. Easier to audit, less code
  running as root.


 http://git.openstack.org/cgit/openstack/oslo.rootwrap/tree/oslo_rootwrap/daemon.py

 --
 Jeremy Stanley

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Clint Byrum
Excerpts from Tristan Cacqueray's message of 2015-02-04 09:02:19 -0800:
 On 02/04/2015 06:57 AM, Daniel P. Berrange wrote:
  On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
  What solutions do we have ?
 
  (1) we could get our act together and audit and fix those filter
  definitions. Remove superfluous usage of root rights, make use of
  advanced filters for where we actually need them. We have been preaching
  for that at many many design summits. This is a lot of work though...
  There were such efforts in the past, but they were never completed for
  some types of nodes. Worse, the bad filter definitions kept coming back,
  since developers take shortcuts, reviewers may not have sufficient
  security awareness to detect crappy filter definitions, and I don't
  think we can design a gate test that would have such awareness.
 
  (2) bite the bullet and accept that some types of nodes actually need
  root rights for so many different things, they should just run as root
  anyway. I know a few distributions which won't be very pleased by such a
  prospect, but that would be a more honest approach (rather than claiming
  we provide efficient isolation when we really don't). An added benefit
  is that we could replace a number of shell calls by Python code, which
  would simplify the code and increase performance.
 
  (3) intermediary solution where we would run as the nova user but run
  sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
  That would leave it up to distros to choose between a blanket sudoer or
  maintain their own filtering rules. I think it's a bit hypocritical
  though (pretend the distros could filter if they wanted it, when we
  dropped the towel on doing that ourselves). I'm also not convinced it's
  more secure than solution 2, and it prevents from reducing the number of
  shell-outs, which I think is a worthy idea.
 
  In all cases I would not drop the baby with the bath water, and keep
  rootwrap for all the cases where root rights are needed on a very
  specific set of commands (like neutron, or nova's api-metadata). The
  daemon mode should address the performance issue for the projects making
  a lot of calls.
  
  
  (4) I think that ultimately we need to ditch rootwrap and provide a proper
  privilege separated, formal RPC mechanism for each project.
  
  eg instead of having a rootwrap command, or rootwrap server attempting
  to validate safety of
  
  qemu-img create -f qcow2 
  /var/lib/nova/instances/instance1/disk.qcow2
  
  we should have a  nova-compute-worker daemon running as root, that accepts
  an RPC command from nova-compute running unprivileged. eg
  
  CreateImage(instane0001, qcow2, disk.qcow)
  
  This immediately makes it trivial to validate that we're not trying to
  trick qemu-img into overwriting some key system file.
  
  This is certainly alot more work than trying to patchup rootwrap, but
  it would provide a level of security that rootwrap can never achieve IMHO.
  
 
 This 4th idea sounds interesting, though we are assuming this new service
 running as root would be exempt of bug, especially if it uses the same
 libraries as non-root services... For example a major bug in python would
 give attacker direct root access while the rootwrap solution would in
 theory keep the intruder at the sudo level...
 

I don't believe that anyone assumes the new service would be without
bugs. But just like the OpenSSH team saw years ago, privilege separation
means that you can absolutely know what is running as root, and what is
not. So when you decide to commit your resources to code audits, you
_start_ with the things that run with elevated privileges.

 
 For completeness, I'd like to propose a more long-term solution:
 
 (5) Get ride of root! Seriously, OpenStack could support security mechanism
 like SELinux or AppArmor in order to properly isolate service and let them
 run what they need to run.
 
 For what it worth, the underlying issue here is having a single almighty super
 user: root and thus we should, at least, consider solution that remove the
 need of such powers (e.g. kernel module loading, ptrace or raw socket).
 

We don't need a security module to drop all of those capabilities
entirely and run as a hobbled root user. By my measure, this process for
nova-compute would only need CAP_NET_ADMIN, CAP_SYS_ADMIN and CAP_KILL.
These capabilities can be audited per-agent and even verified as needed
simply by running integration tests without each one to see what breaks.

 Beside, as long as sensitive process are not contained at the system level,
 the attack surface for a non-root user is still very wide (e.g. system calls,
 setuid binaries, ipc, ...)
 
 
 While this might sounds impossible to implement upstream because it's too
 vendor specific or just because of other technicals difficulties,
 I guess it still deserves a mention in this thread.
 

I think OpenStack can do its part by making privilege 

Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Clint Byrum
Excerpts from Daniel P. Berrange's message of 2015-02-04 03:57:53 -0800:
 On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
  The first one is performance -- each call would spawn a Python
  interpreter which would then call the system command. This was fine when
  there were just a few calls here and there, not so much when it's called
  a hundred times in a row. During the Juno cycle, a daemon mode was added
  to solve this issue. It is significantly faster than running sudo
  directly (the often-suggested alternative). Projects still have to start
  adopting it though. Neutron and Cinder have started work to do that in Kilo.
  
  The second problem is the quality of the filter definitions. Rootwrap is
  a framework to enable isolation. It's only as good as the filters each
  project defines. Most of them rely on CommandFilters that do not check
  any argument, instead of using more powerful filters (which are arguably
  more painful to maintain). Developers routinely add filter definitions
  that basically remove any isolation that might have been there, like
  allowing blank dd, tee, chown or chmod.
 
 I think this is really the key point which shows rootwrap as a concept
 is broken by design IMHO. Root wrap is essentially trying to provide an
 API for invoking privileged operations, but instead of actually designing
 an explicit API for the operations, we done by implicit one based on
 command args. From a security POV I think this approach is doomed to
 failure, but command arg strings are fr to expressive a concept
 to deal with.
 
  What solutions do we have ?
  
  (1) we could get our act together and audit and fix those filter
  definitions. Remove superfluous usage of root rights, make use of
  advanced filters for where we actually need them. We have been preaching
  for that at many many design summits. This is a lot of work though...
  There were such efforts in the past, but they were never completed for
  some types of nodes. Worse, the bad filter definitions kept coming back,
  since developers take shortcuts, reviewers may not have sufficient
  security awareness to detect crappy filter definitions, and I don't
  think we can design a gate test that would have such awareness.
  
  (2) bite the bullet and accept that some types of nodes actually need
  root rights for so many different things, they should just run as root
  anyway. I know a few distributions which won't be very pleased by such a
  prospect, but that would be a more honest approach (rather than claiming
  we provide efficient isolation when we really don't). An added benefit
  is that we could replace a number of shell calls by Python code, which
  would simplify the code and increase performance.
  
  (3) intermediary solution where we would run as the nova user but run
  sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
  That would leave it up to distros to choose between a blanket sudoer or
  maintain their own filtering rules. I think it's a bit hypocritical
  though (pretend the distros could filter if they wanted it, when we
  dropped the towel on doing that ourselves). I'm also not convinced it's
  more secure than solution 2, and it prevents from reducing the number of
  shell-outs, which I think is a worthy idea.
  
  In all cases I would not drop the baby with the bath water, and keep
  rootwrap for all the cases where root rights are needed on a very
  specific set of commands (like neutron, or nova's api-metadata). The
  daemon mode should address the performance issue for the projects making
  a lot of calls.
 
 
 (4) I think that ultimately we need to ditch rootwrap and provide a proper
 privilege separated, formal RPC mechanism for each project.
 
 eg instead of having a rootwrap command, or rootwrap server attempting
 to validate safety of
 
 qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2
 
 we should have a  nova-compute-worker daemon running as root, that accepts
 an RPC command from nova-compute running unprivileged. eg
 
 CreateImage(instane0001, qcow2, disk.qcow)
 
 This immediately makes it trivial to validate that we're not trying to
 trick qemu-img into overwriting some key system file.
 
 This is certainly alot more work than trying to patchup rootwrap, but
 it would provide a level of security that rootwrap can never achieve IMHO.
 

+1, I think you're right on Daniel. Count me in for future discussions
and work on this.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Robert Collins
On 5 February 2015 at 03:33, Monty Taylor mord...@inaugust.com wrote:
 On 02/04/2015 06:57 AM, Daniel P. Berrange wrote:
 to manage VMs on a laptop - you're going to use virtualbox or
 virt-manager. You're going to use nova-compute to manage compute hosts
 in a cloud - and in almost all circumstances the only thing that's going
 to be running on your compute hosts is going to be nova-compute.

Actually we know from user / operator stories at summits that many
groups run *much more* than nova-compute on their compute hosts.
Specifically the all-in-one scale-as-a-unit architecture is quite
popular. That has all the components of
cinder/swift/keystone/nova/neutron all running on all machines, and
the cluster is scaled by adding just another identical machine.

I suspect its actually really quite common outside of a) paranoid (but
not necessarily wrong :)) and b) ultra-scale environments.
...

I'm with Daniel -
 (4) I think that ultimately we need to ditch rootwrap and provide a proper
 privilege separated, formal RPC mechanism for each project.

 eg instead of having a rootwrap command, or rootwrap server attempting
 to validate safety of

 qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2

 we should have a  nova-compute-worker daemon running as root, that accepts
 an RPC command from nova-compute running unprivileged. eg

 CreateImage(instane0001, qcow2, disk.qcow)

 This immediately makes it trivial to validate that we're not trying to
 trick qemu-img into overwriting some key system file.

 This is certainly alot more work than trying to patchup rootwrap, but
 it would provide a level of security that rootwrap can never achieve IMHO.

I think a local root daemon is a solid idea, there's lots and lots of
prior art on this, from the bazillion dbus daemons we haveadays to the
Android isolation between apps.

So count me in too on design and speccing for this. I realise that for
some cases rootwrap was absolutely a performance and stability issue,
but daemon mode should address that - so I think this is a longer term
project, giving us a significant step up in security.

-Rob

-- 
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Daniel P. Berrange
On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
 The first one is performance -- each call would spawn a Python
 interpreter which would then call the system command. This was fine when
 there were just a few calls here and there, not so much when it's called
 a hundred times in a row. During the Juno cycle, a daemon mode was added
 to solve this issue. It is significantly faster than running sudo
 directly (the often-suggested alternative). Projects still have to start
 adopting it though. Neutron and Cinder have started work to do that in Kilo.
 
 The second problem is the quality of the filter definitions. Rootwrap is
 a framework to enable isolation. It's only as good as the filters each
 project defines. Most of them rely on CommandFilters that do not check
 any argument, instead of using more powerful filters (which are arguably
 more painful to maintain). Developers routinely add filter definitions
 that basically remove any isolation that might have been there, like
 allowing blank dd, tee, chown or chmod.

I think this is really the key point which shows rootwrap as a concept
is broken by design IMHO. Root wrap is essentially trying to provide an
API for invoking privileged operations, but instead of actually designing
an explicit API for the operations, we done by implicit one based on
command args. From a security POV I think this approach is doomed to
failure, but command arg strings are fr to expressive a concept
to deal with.

 What solutions do we have ?
 
 (1) we could get our act together and audit and fix those filter
 definitions. Remove superfluous usage of root rights, make use of
 advanced filters for where we actually need them. We have been preaching
 for that at many many design summits. This is a lot of work though...
 There were such efforts in the past, but they were never completed for
 some types of nodes. Worse, the bad filter definitions kept coming back,
 since developers take shortcuts, reviewers may not have sufficient
 security awareness to detect crappy filter definitions, and I don't
 think we can design a gate test that would have such awareness.
 
 (2) bite the bullet and accept that some types of nodes actually need
 root rights for so many different things, they should just run as root
 anyway. I know a few distributions which won't be very pleased by such a
 prospect, but that would be a more honest approach (rather than claiming
 we provide efficient isolation when we really don't). An added benefit
 is that we could replace a number of shell calls by Python code, which
 would simplify the code and increase performance.
 
 (3) intermediary solution where we would run as the nova user but run
 sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
 That would leave it up to distros to choose between a blanket sudoer or
 maintain their own filtering rules. I think it's a bit hypocritical
 though (pretend the distros could filter if they wanted it, when we
 dropped the towel on doing that ourselves). I'm also not convinced it's
 more secure than solution 2, and it prevents from reducing the number of
 shell-outs, which I think is a worthy idea.
 
 In all cases I would not drop the baby with the bath water, and keep
 rootwrap for all the cases where root rights are needed on a very
 specific set of commands (like neutron, or nova's api-metadata). The
 daemon mode should address the performance issue for the projects making
 a lot of calls.


(4) I think that ultimately we need to ditch rootwrap and provide a proper
privilege separated, formal RPC mechanism for each project.

eg instead of having a rootwrap command, or rootwrap server attempting
to validate safety of

qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2

we should have a  nova-compute-worker daemon running as root, that accepts
an RPC command from nova-compute running unprivileged. eg

CreateImage(instane0001, qcow2, disk.qcow)

This immediately makes it trivial to validate that we're not trying to
trick qemu-img into overwriting some key system file.

This is certainly alot more work than trying to patchup rootwrap, but
it would provide a level of security that rootwrap can never achieve IMHO.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Duncan Thomas
I suppose that the security argument against running the whole of
nova-compute as root is that a remote exploit in the service is much better
constrained when the thing isn't running as root - e.g. some input
validation fails and allows arbitrary shell in some (currently none-root)
command via an existing API call. This means that (2) is the worst possible
option - calling out to sudo at least limits the attach surface to those
code paths that do those specific exec calls.

I think Daniel's suggestion makes a great deal more sense from a security
point of view, but (3) is a significantly better architecture than (2) even
if it still has some holes.

On 4 February 2015 at 16:33, Monty Taylor mord...@inaugust.com wrote:

 On 02/04/2015 06:57 AM, Daniel P. Berrange wrote:
  On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
  The first one is performance -- each call would spawn a Python
  interpreter which would then call the system command. This was fine when
  there were just a few calls here and there, not so much when it's called
  a hundred times in a row. During the Juno cycle, a daemon mode was added
  to solve this issue. It is significantly faster than running sudo
  directly (the often-suggested alternative). Projects still have to start
  adopting it though. Neutron and Cinder have started work to do that in
 Kilo.
 
  The second problem is the quality of the filter definitions. Rootwrap is
  a framework to enable isolation. It's only as good as the filters each
  project defines. Most of them rely on CommandFilters that do not check
  any argument, instead of using more powerful filters (which are arguably
  more painful to maintain). Developers routinely add filter definitions
  that basically remove any isolation that might have been there, like
  allowing blank dd, tee, chown or chmod.
 
  I think this is really the key point which shows rootwrap as a concept
  is broken by design IMHO. Root wrap is essentially trying to provide an
  API for invoking privileged operations, but instead of actually designing
  an explicit API for the operations, we done by implicit one based on
  command args. From a security POV I think this approach is doomed to
  failure, but command arg strings are fr to expressive a concept
  to deal with.
 
  What solutions do we have ?
 
  (1) we could get our act together and audit and fix those filter
  definitions. Remove superfluous usage of root rights, make use of
  advanced filters for where we actually need them. We have been preaching
  for that at many many design summits. This is a lot of work though...
  There were such efforts in the past, but they were never completed for
  some types of nodes. Worse, the bad filter definitions kept coming back,
  since developers take shortcuts, reviewers may not have sufficient
  security awareness to detect crappy filter definitions, and I don't
  think we can design a gate test that would have such awareness.
 
  (2) bite the bullet and accept that some types of nodes actually need
  root rights for so many different things, they should just run as root
  anyway. I know a few distributions which won't be very pleased by such a
  prospect, but that would be a more honest approach (rather than claiming
  we provide efficient isolation when we really don't). An added benefit
  is that we could replace a number of shell calls by Python code, which
  would simplify the code and increase performance.

 I'm actually the biggest fan of this solution (even more than Daniel's
 suggestion below) because it's the thing that is closest to reality.

 Security isn't a useful concept in a vacuum - it's something we do to
 prevent access to or damage resources that we don't want accessed by the
 wrong people.

 On compute nodes, the main valuable thing are the VMs themselves- and
 I'd expect the most interested target of an attack to be interested in
 manipulating, stealing data from or deleting the VMs.

 No amount of rootwrap or privileges are going to prevent nova-compute
 from performing unwanted actions on the VMs in its control - for the
 reason that it's job in life is to manipulate those things.

 Is it a security hole in the traditional distro sense - that we want to
 be able to install all of these things with apt-get or yum on a single
 server and have the actions of one service not affect the state of
 another? Sure. Is it in the real world? No. You're not going to use this
 to manage VMs on a laptop - you're going to use virtualbox or
 virt-manager. You're going to use nova-compute to manage compute hosts
 in a cloud - and in almost all circumstances the only thing that's going
 to be running on your compute hosts is going to be nova-compute.


  (3) intermediary solution where we would run as the nova user but run
  sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
  That would leave it up to distros to choose between a blanket sudoer or
  maintain their own filtering rules. I think 

Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Thierry Carrez
Monty Taylor wrote:
 On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
 (2) bite the bullet and accept that some types of nodes actually need
 root rights for so many different things, they should just run as root
 anyway. I know a few distributions which won't be very pleased by such a
 prospect, but that would be a more honest approach (rather than claiming
 we provide efficient isolation when we really don't). An added benefit
 is that we could replace a number of shell calls by Python code, which
 would simplify the code and increase performance.
 
 I'm actually the biggest fan of this solution (even more than Daniel's
 suggestion below) because it's the thing that is closest to reality.
 
 Security isn't a useful concept in a vacuum - it's something we do to
 prevent access to or damage resources that we don't want accessed by the
 wrong people.
 
 On compute nodes, the main valuable thing are the VMs themselves- and
 I'd expect the most interested target of an attack to be interested in
 manipulating, stealing data from or deleting the VMs.
 
 No amount of rootwrap or privileges are going to prevent nova-compute
 from performing unwanted actions on the VMs in its control - for the
 reason that it's job in life is to manipulate those things.
 
 Is it a security hole in the traditional distro sense - that we want to
 be able to install all of these things with apt-get or yum on a single
 server and have the actions of one service not affect the state of
 another? Sure. Is it in the real world? No. You're not going to use this
 to manage VMs on a laptop - you're going to use virtualbox or
 virt-manager. You're going to use nova-compute to manage compute hosts
 in a cloud - and in almost all circumstances the only thing that's going
 to be running on your compute hosts is going to be nova-compute.

You make a good point when you mention traditional distro here. I
would argue that containers are slightly changing the rules of the
don't-run-as-root game.

Solution (2) aligns pretty well with container-powered OpenStack
deployments -- running compute nodes as root in a container (and
embracing abovementioned simplicity/performance gains) sounds like a
pretty strong combo.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Monty Taylor
On 02/04/2015 06:57 AM, Daniel P. Berrange wrote:
 On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
 The first one is performance -- each call would spawn a Python
 interpreter which would then call the system command. This was fine when
 there were just a few calls here and there, not so much when it's called
 a hundred times in a row. During the Juno cycle, a daemon mode was added
 to solve this issue. It is significantly faster than running sudo
 directly (the often-suggested alternative). Projects still have to start
 adopting it though. Neutron and Cinder have started work to do that in Kilo.

 The second problem is the quality of the filter definitions. Rootwrap is
 a framework to enable isolation. It's only as good as the filters each
 project defines. Most of them rely on CommandFilters that do not check
 any argument, instead of using more powerful filters (which are arguably
 more painful to maintain). Developers routinely add filter definitions
 that basically remove any isolation that might have been there, like
 allowing blank dd, tee, chown or chmod.
 
 I think this is really the key point which shows rootwrap as a concept
 is broken by design IMHO. Root wrap is essentially trying to provide an
 API for invoking privileged operations, but instead of actually designing
 an explicit API for the operations, we done by implicit one based on
 command args. From a security POV I think this approach is doomed to
 failure, but command arg strings are fr to expressive a concept
 to deal with.
 
 What solutions do we have ?

 (1) we could get our act together and audit and fix those filter
 definitions. Remove superfluous usage of root rights, make use of
 advanced filters for where we actually need them. We have been preaching
 for that at many many design summits. This is a lot of work though...
 There were such efforts in the past, but they were never completed for
 some types of nodes. Worse, the bad filter definitions kept coming back,
 since developers take shortcuts, reviewers may not have sufficient
 security awareness to detect crappy filter definitions, and I don't
 think we can design a gate test that would have such awareness.

 (2) bite the bullet and accept that some types of nodes actually need
 root rights for so many different things, they should just run as root
 anyway. I know a few distributions which won't be very pleased by such a
 prospect, but that would be a more honest approach (rather than claiming
 we provide efficient isolation when we really don't). An added benefit
 is that we could replace a number of shell calls by Python code, which
 would simplify the code and increase performance.

I'm actually the biggest fan of this solution (even more than Daniel's
suggestion below) because it's the thing that is closest to reality.

Security isn't a useful concept in a vacuum - it's something we do to
prevent access to or damage resources that we don't want accessed by the
wrong people.

On compute nodes, the main valuable thing are the VMs themselves- and
I'd expect the most interested target of an attack to be interested in
manipulating, stealing data from or deleting the VMs.

No amount of rootwrap or privileges are going to prevent nova-compute
from performing unwanted actions on the VMs in its control - for the
reason that it's job in life is to manipulate those things.

Is it a security hole in the traditional distro sense - that we want to
be able to install all of these things with apt-get or yum on a single
server and have the actions of one service not affect the state of
another? Sure. Is it in the real world? No. You're not going to use this
to manage VMs on a laptop - you're going to use virtualbox or
virt-manager. You're going to use nova-compute to manage compute hosts
in a cloud - and in almost all circumstances the only thing that's going
to be running on your compute hosts is going to be nova-compute.


 (3) intermediary solution where we would run as the nova user but run
 sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
 That would leave it up to distros to choose between a blanket sudoer or
 maintain their own filtering rules. I think it's a bit hypocritical
 though (pretend the distros could filter if they wanted it, when we
 dropped the towel on doing that ourselves). I'm also not convinced it's
 more secure than solution 2, and it prevents from reducing the number of
 shell-outs, which I think is a worthy idea.

 In all cases I would not drop the baby with the bath water, and keep
 rootwrap for all the cases where root rights are needed on a very
 specific set of commands (like neutron, or nova's api-metadata). The
 daemon mode should address the performance issue for the projects making
 a lot of calls.
 
 
 (4) I think that ultimately we need to ditch rootwrap and provide a proper
 privilege separated, formal RPC mechanism for each project.
 
 eg instead of having a rootwrap command, or rootwrap server attempting
 

Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes

2015-02-04 Thread Duncan Thomas
4) Write a small daemon that runs as root, accepting commands over a unix
domain socket or similar. Easier to audit, less code running as root.

On 4 February 2015 at 12:58, Thierry Carrez thie...@openstack.org wrote:

 Hi,

 This is the follow-up of the discussion we started yesterday on rootwrap
 usage at the cross-project meeting.

 A bit of history to stage this story first. OpenStack nodes sometimes
 need to run things with elevated privileges. Nova started out as calling
 sudo shell commands to execute those, which basically allowed it to run
 anything as root. Distributions didn't really like that, so packagers
 started to ship sudoers files that restricted the commands Nova could
 run with sudo. It created a maintenance nightmare, where the code and
 the packaging were routinely out of sync. Rootwrap was designed so that
 we would maintain the allowed commands in the code itself, which
 facilitated maintenance. It would also allow more complex rules than
 sudoers allowed, like the ability to only kill a certain executable
 processes.

 Rootwrap was then adopted by Cinder and Neutron, and moved to oslo to
 avoid code duplication. There were still two long-standing problems with
 it, though.

 The first one is performance -- each call would spawn a Python
 interpreter which would then call the system command. This was fine when
 there were just a few calls here and there, not so much when it's called
 a hundred times in a row. During the Juno cycle, a daemon mode was added
 to solve this issue. It is significantly faster than running sudo
 directly (the often-suggested alternative). Projects still have to start
 adopting it though. Neutron and Cinder have started work to do that in
 Kilo.

 The second problem is the quality of the filter definitions. Rootwrap is
 a framework to enable isolation. It's only as good as the filters each
 project defines. Most of them rely on CommandFilters that do not check
 any argument, instead of using more powerful filters (which are arguably
 more painful to maintain). Developers routinely add filter definitions
 that basically remove any isolation that might have been there, like
 allowing blank dd, tee, chown or chmod.

 Basically, some nodes run such a variety of commands as root that
 maintaining proper isolation is a non-trivial amount of work: that's the
 case for Nova's compute nodes and Cinder's volume nodes. I would argue
 that for all other types of nodes, rootwrap is still an appropriate way
 to provide isolation. But for nodes which need to make a very wide
 variety of root calls, we pretend to isolate, but the filter definitions
 are so full of holes that we actually don't.

 What solutions do we have ?

 (1) we could get our act together and audit and fix those filter
 definitions. Remove superfluous usage of root rights, make use of
 advanced filters for where we actually need them. We have been preaching
 for that at many many design summits. This is a lot of work though...
 There were such efforts in the past, but they were never completed for
 some types of nodes. Worse, the bad filter definitions kept coming back,
 since developers take shortcuts, reviewers may not have sufficient
 security awareness to detect crappy filter definitions, and I don't
 think we can design a gate test that would have such awareness.

 (2) bite the bullet and accept that some types of nodes actually need
 root rights for so many different things, they should just run as root
 anyway. I know a few distributions which won't be very pleased by such a
 prospect, but that would be a more honest approach (rather than claiming
 we provide efficient isolation when we really don't). An added benefit
 is that we could replace a number of shell calls by Python code, which
 would simplify the code and increase performance.

 (3) intermediary solution where we would run as the nova user but run
 sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
 That would leave it up to distros to choose between a blanket sudoer or
 maintain their own filtering rules. I think it's a bit hypocritical
 though (pretend the distros could filter if they wanted it, when we
 dropped the towel on doing that ourselves). I'm also not convinced it's
 more secure than solution 2, and it prevents from reducing the number of
 shell-outs, which I think is a worthy idea.

 In all cases I would not drop the baby with the bath water, and keep
 rootwrap for all the cases where root rights are needed on a very
 specific set of commands (like neutron, or nova's api-metadata). The
 daemon mode should address the performance issue for the projects making
 a lot of calls.

 --
 Thierry Carrez (ttx)

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Duncan Thomas