Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
Hi Daniel, Yes, there's some semantic meaning at that level. But this level already exists at the current rootwrap caller site, too - and if that one can be tricked to do something against image.img rm -rf /, then the additional layer can be tricked, too. No, that is really not correct. If you are passing full command strings to rootwrap then the caller can trick rootwrap into running commands with those shell metacharacter exploits. If you have a formal API like the one I describe and correctly implement it, there would be no shell involved at all. ie the nova-compute-worker program would directly invoke the system call execve(/usr/bin/qemu-img, create, image.img rm -rf /) and this would at worse create a file called 'image.img rm -rf /' and *not* invoke the rm command as you get when you use shell. This is really just another example of why rootwrap/sudo as a concept is a bad idea. The shell should never be involved in executing any external commands that Nova/Neutron/etc need to run, because it is impractical to correctly validate shell commands anywhere in the stack. The only safe thing todo is to take shell out of the picture entirely. that was just an example. If cinder calls rootwrap with a list of arguments, and this is then called via sudo, there's no shell inbetween either. But now say there's something else, like the bash functions bug, and some codepath runs a command in a way that sets the environment - then any grandchild processes might cause a breach. I'm not arguing that *this exact string* is a problem. My point is that the cinder/nova code already have as much information as possible, and *any* library abstraction can just remove some available data. So, besides increasing the LOC, it won't get more secure. I'm arguing that the rootwrap call sites need to be fixed, ie. they need to proof in some way that the arguments they pass on are sane - that's more or less the same thing that this new library would need to do too. Regards, Phil -- : Ing. Philipp Marek : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com : DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 2/4/15, 10:24 AM, Daniel P. Berrange berra...@redhat.com wrote: On Wed, Feb 04, 2015 at 09:10:06AM -0800, James E. Blair wrote: Thierry Carrez thie...@openstack.org writes: You make a good point when you mention traditional distro here. I would argue that containers are slightly changing the rules of the don't-run-as-root game. Solution (2) aligns pretty well with container-powered OpenStack deployments -- running compute nodes as root in a container (and embracing abovementioned simplicity/performance gains) sounds like a pretty strong combo. This sounds at least a little like a suggestion that containers are a substitute for the security provided by running non-root. The security landscape around containers is complex, and while there are a lot of benefits, I believe the general consensus is that uid 0 processes should not be seen as fully isolated. From https://docs.docker.com/articles/security/ : Docker containers are, by default, quite secure; especially if you take care of running your processes inside the containers as non-privileged users (i.e., non-root). Which is not to say that using containers is not a good idea, but rather, if one does, one should avoid running as root (perhaps with capabilities), and use selinux (or similar). Yep, I've seen attempts by some folks to run nova-compute and libvirtd and QEMU inside a docker container. Because of the inherantly privileged nature of what Nova/libvirt/qemu need to do, you end up having to share all the host namespaces with the docker container, except for the filesystem namespace and even that you need to bind mount a bunch of stuff over. As a result the container isn't really offerring any security benefit over running the things outside the container. IOW the use of containers to confine nova is only solving a managability problem rather than a security problem. Regards, Daniel Daniel, Agree 100% - compute in containers is all about an atomic image-based upgrade and downgrade process, not about solving security problems. Ideally the services that are within containers are as secure as running natively without containers on bare metal although this might be a bit of assumption since docker does run with all Linux capabilities enabled. Regards -steve -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 02/04/2015 01:13 PM, Clint Byrum wrote: Excerpts from Tristan Cacqueray's message of 2015-02-04 09:02:19 -0800: On 02/04/2015 06:57 AM, Daniel P. Berrange wrote: (4) I think that ultimately we need to ditch rootwrap and provide a proper privilege separated, formal RPC mechanism for each project. eg instead of having a rootwrap command, or rootwrap server attempting to validate safety of qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2 we should have a nova-compute-worker daemon running as root, that accepts an RPC command from nova-compute running unprivileged. eg CreateImage(instane0001, qcow2, disk.qcow) This immediately makes it trivial to validate that we're not trying to trick qemu-img into overwriting some key system file. This is certainly alot more work than trying to patchup rootwrap, but it would provide a level of security that rootwrap can never achieve IMHO. This 4th idea sounds interesting, though we are assuming this new service running as root would be exempt of bug, especially if it uses the same libraries as non-root services... For example a major bug in python would give attacker direct root access while the rootwrap solution would in theory keep the intruder at the sudo level... I don't believe that anyone assumes the new service would be without bugs. I meant less bug than another solution. Such RPC service daemon will eventually requires quite some code to be robust, which could lead to more bugs. But just like the OpenSSH team saw years ago, privilege separation means that you can absolutely know what is running as root, and what is not. So when you decide to commit your resources to code audits, you _start_ with the things that run with elevated privileges. Not quite sure to follow here... OpenSSH is using privilege separation after authentication, e.g. the root-based process is doing the authentication while the external data processing is done through an unprivileged process. If I understand correctly Daniel's solution (4), it's to have a semantic on the privileged actions to avoid mis-usage (like inject command in a sudo call). Thus if we want to emulate OpenSSH design, the rpc call would also need to carry authentication data in order to prevent unwanted activity. And the rpc daemon would then need to enforce some kind of acl/policy. The amount of code to be audited could arguably be higher with such design. For completeness, I'd like to propose a more long-term solution: (5) Get ride of root! Seriously, OpenStack could support security mechanism like SELinux or AppArmor in order to properly isolate service and let them run what they need to run. For what it worth, the underlying issue here is having a single almighty super user: root and thus we should, at least, consider solution that remove the need of such powers (e.g. kernel module loading, ptrace or raw socket). We don't need a security module to drop all of those capabilities entirely and run as a hobbled root user. By my measure, this process for nova-compute would only need CAP_NET_ADMIN, CAP_SYS_ADMIN and CAP_KILL. These capabilities can be audited per-agent and even verified as needed simply by running integration tests without each one to see what breaks. Capabilities might be a candidate as well, but they don't cover every cases and lack granularity... SECCOMP filters might be good enough too. Beside, as long as sensitive process are not contained at the system level, the attack surface for a non-root user is still very wide (e.g. system calls, setuid binaries, ipc, ...) While this might sounds impossible to implement upstream because it's too vendor specific or just because of other technicals difficulties, I guess it still deserves a mention in this thread. I think OpenStack can do its part by making privilege separation a reality for the security sensitive parts of OpenStack, and that will ease pressure to implement strong controls in any security modules the Linux distros ship with. That would be a great step forward indeed. Having some kind of privilege separation would surely help a lot to properly configure security modules. Regards, Tristan signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On Thu, 2015-02-05 at 08:27 -0500, Tristan Cacqueray wrote: Thus if we want to emulate OpenSSH design, the rpc call would also need to carry authentication data in order to prevent unwanted activity. And the rpc daemon would then need to enforce some kind of acl/policy. Sounds a lot like what the DBus system bus implements/provides to me... And it took quite some time to get that done. (Note: I'm not proposing its usage in any way) Nicolas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 2015-02-04 13:40:29 +0200 (+0200), Duncan Thomas wrote: 4) Write a small daemon that runs as root, accepting commands over a unix domain socket or similar. Easier to audit, less code running as root. http://git.openstack.org/cgit/openstack/oslo.rootwrap/tree/oslo_rootwrap/daemon.py -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 2015-02-04 18:38:16 +0200 (+0200), Duncan Thomas wrote: If I'm reading that correctly, it does not help with the filtering issues at all, since it needs exactly the same kind of filter. Daniel explained the concept far better than I. I didn't mean to imply that it does, merely that it fits your rather terse description of a daemon that runs as root, accepting commands over a unix domain socket or similar. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 02/04/2015 06:57 AM, Daniel P. Berrange wrote: On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote: What solutions do we have ? (1) we could get our act together and audit and fix those filter definitions. Remove superfluous usage of root rights, make use of advanced filters for where we actually need them. We have been preaching for that at many many design summits. This is a lot of work though... There were such efforts in the past, but they were never completed for some types of nodes. Worse, the bad filter definitions kept coming back, since developers take shortcuts, reviewers may not have sufficient security awareness to detect crappy filter definitions, and I don't think we can design a gate test that would have such awareness. (2) bite the bullet and accept that some types of nodes actually need root rights for so many different things, they should just run as root anyway. I know a few distributions which won't be very pleased by such a prospect, but that would be a more honest approach (rather than claiming we provide efficient isolation when we really don't). An added benefit is that we could replace a number of shell calls by Python code, which would simplify the code and increase performance. (3) intermediary solution where we would run as the nova user but run sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND). That would leave it up to distros to choose between a blanket sudoer or maintain their own filtering rules. I think it's a bit hypocritical though (pretend the distros could filter if they wanted it, when we dropped the towel on doing that ourselves). I'm also not convinced it's more secure than solution 2, and it prevents from reducing the number of shell-outs, which I think is a worthy idea. In all cases I would not drop the baby with the bath water, and keep rootwrap for all the cases where root rights are needed on a very specific set of commands (like neutron, or nova's api-metadata). The daemon mode should address the performance issue for the projects making a lot of calls. (4) I think that ultimately we need to ditch rootwrap and provide a proper privilege separated, formal RPC mechanism for each project. eg instead of having a rootwrap command, or rootwrap server attempting to validate safety of qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2 we should have a nova-compute-worker daemon running as root, that accepts an RPC command from nova-compute running unprivileged. eg CreateImage(instane0001, qcow2, disk.qcow) This immediately makes it trivial to validate that we're not trying to trick qemu-img into overwriting some key system file. This is certainly alot more work than trying to patchup rootwrap, but it would provide a level of security that rootwrap can never achieve IMHO. This 4th idea sounds interesting, though we are assuming this new service running as root would be exempt of bug, especially if it uses the same libraries as non-root services... For example a major bug in python would give attacker direct root access while the rootwrap solution would in theory keep the intruder at the sudo level... For completeness, I'd like to propose a more long-term solution: (5) Get ride of root! Seriously, OpenStack could support security mechanism like SELinux or AppArmor in order to properly isolate service and let them run what they need to run. For what it worth, the underlying issue here is having a single almighty super user: root and thus we should, at least, consider solution that remove the need of such powers (e.g. kernel module loading, ptrace or raw socket). Beside, as long as sensitive process are not contained at the system level, the attack surface for a non-root user is still very wide (e.g. system calls, setuid binaries, ipc, ...) While this might sounds impossible to implement upstream because it's too vendor specific or just because of other technicals difficulties, I guess it still deserves a mention in this thread. Best regards, Tristan signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On Wed, Feb 04, 2015 at 05:52:12PM +0100, Philipp Marek wrote: Here are my 2¢. (1) we could get our act together and audit and fix those filter definitions. Remove superfluous usage of root rights, make use of advanced filters for where we actually need them. We have been preaching for that at many many design summits. This is a lot of work though... There were such efforts in the past, but they were never completed for some types of nodes. Worse, the bad filter definitions kept coming back, since developers take shortcuts, reviewers may not have sufficient security awareness to detect crappy filter definitions, and I don't think we can design a gate test that would have such awareness. Sounds like a lot of work... ongoing, too. (2) bite the bullet and accept that some types of nodes actually need root rights for so many different things, they should just run as root anyway. I know a few distributions which won't be very pleased by such a prospect, but that would be a more honest approach (rather than claiming we provide efficient isolation when we really don't). An added benefit is that we could replace a number of shell calls by Python code, which would simplify the code and increase performance. Practical, but unsafe. I'd very much like to have some best-effort filter against bugs in my programming - even more so during development. (4) I think that ultimately we need to ditch rootwrap and provide a proper privilege separated, formal RPC mechanism for each project. ... we should have a nova-compute-worker daemon running as root, that accepts an RPC command from nova-compute running unprivileged. eg CreateImage(instane0001, qcow2, disk.qcow) ... This is certainly alot more work than trying to patchup rootwrap, but it would provide a level of security that rootwrap can never achieve IMHO. A lot of work, and if input sanitation didn't work in one piece of code, why should it here? I think this only leads to _more_ work, without any real benefit. If we can't get the filters right the first round, we won't make it here either. The difference is that the API I illustrate here has *semantic* context about the operation. In the API example the caller is not permitted to provide a directory path - only the name of the instance and the name of the disk image. The privileged nova-compute-worker program can thus enforce exactly what directory the image is created in, and ensure it doesn't clash with a disk image from another VM. This kind of validation is impractical when you are just given a 'qemu-img' command line args with a full directory path, so there is no semantic conext for the privileged rootwrap to know whether it is reasonable to create the disk image in that particular location. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On Wed, Feb 04, 2015 at 06:38:16PM +0200, Duncan Thomas wrote: If I'm reading that correctly, it does not help with the filtering issues at all, since it needs exactly the same kind of filter. Daniel explained the concept far better than I. Yep, the only thing rootwrap daemon mode does is to remove the overhead of spawning the rootwrap command. It does nothing to improve actual security - it is still a chocolate teapot from that POV. On 4 February 2015 at 18:33, Jeremy Stanley fu...@yuggoth.org wrote: On 2015-02-04 13:40:29 +0200 (+0200), Duncan Thomas wrote: 4) Write a small daemon that runs as root, accepting commands over a unix domain socket or similar. Easier to audit, less code running as root. http://git.openstack.org/cgit/openstack/oslo.rootwrap/tree/oslo_rootwrap/daemon.py Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On Wed, Feb 04, 2015 at 06:05:16PM +0100, Philipp Marek wrote: (4) I think that ultimately we need to ditch rootwrap and provide a proper privilege separated, formal RPC mechanism for each project. ... we should have a nova-compute-worker daemon running as root, that accepts an RPC command from nova-compute running unprivileged. eg CreateImage(instane0001, qcow2, disk.qcow) ... This is certainly alot more work than trying to patchup rootwrap, but it would provide a level of security that rootwrap can never achieve IMHO. A lot of work, and if input sanitation didn't work in one piece of code, why should it here? I think this only leads to _more_ work, without any real benefit. If we can't get the filters right the first round, we won't make it here either. The difference is that the API I illustrate here has *semantic* context about the operation. In the API example the caller is not permitted to provide a directory path - only the name of the instance and the name of the disk image. The privileged nova-compute-worker program can thus enforce exactly what directory the image is created in, and ensure it doesn't clash with a disk image from another VM. This kind of validation is impractical when you are just given a 'qemu-img' command line args with a full directory path, so there is no semantic conext for the privileged rootwrap to know whether it is reasonable to create the disk image in that particular location. Sorry about being unclear. Yes, there's some semantic meaning at that level. But this level already exists at the current rootwrap caller site, too - and if that one can be tricked to do something against image.img rm -rf /, then the additional layer can be tricked, too. No, that is really not correct. If you are passing full command strings to rootwrap then the caller can trick rootwrap into running commands with those shell metacharacter exploits. If you have a formal API like the one I describe and correctly implement it, there would be no shell involved at all. ie the nova-compute-worker program would directly invoke the system call execve(/usr/bin/qemu-img, create, image.img rm -rf /) and this would at worse create a file called 'image.img rm -rf /' and *not* invoke the rm command as you get when you use shell. This is really just another example of why rootwrap/sudo as a concept is a bad idea. The shell should never be involved in executing any external commands that Nova/Neutron/etc need to run, because it is impractical to correctly validate shell commands anywhere in the stack. The only safe thing todo is to take shell out of the picture entirely. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 2015-02-04 11:58:03 +0100 (+0100), Thierry Carrez wrote: [...] The second problem is the quality of the filter definitions. Rootwrap is a framework to enable isolation. It's only as good as the filters each project defines. Most of them rely on CommandFilters that do not check any argument, instead of using more powerful filters (which are arguably more painful to maintain). Developers routinely add filter definitions that basically remove any isolation that might have been there, like allowing blank dd, tee, chown or chmod. [...] This part is my biggest concern at the moment, from a vulnerability management standpoint. I'm worried that it's an attractive nuisance resulting in a false sense of security in its current state because we're not calling this shortcoming out explicitly in documentation (as far as I'm aware), and so we're opening our operators/users up to unexpected risks and opening ourselves up to the possibility of a slew of vulnerability reports because this mechanism doesn't provide the level of protection it would seem to imply. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
Here are my 2¢. (1) we could get our act together and audit and fix those filter definitions. Remove superfluous usage of root rights, make use of advanced filters for where we actually need them. We have been preaching for that at many many design summits. This is a lot of work though... There were such efforts in the past, but they were never completed for some types of nodes. Worse, the bad filter definitions kept coming back, since developers take shortcuts, reviewers may not have sufficient security awareness to detect crappy filter definitions, and I don't think we can design a gate test that would have such awareness. Sounds like a lot of work... ongoing, too. (2) bite the bullet and accept that some types of nodes actually need root rights for so many different things, they should just run as root anyway. I know a few distributions which won't be very pleased by such a prospect, but that would be a more honest approach (rather than claiming we provide efficient isolation when we really don't). An added benefit is that we could replace a number of shell calls by Python code, which would simplify the code and increase performance. Practical, but unsafe. I'd very much like to have some best-effort filter against bugs in my programming - even more so during development. (4) I think that ultimately we need to ditch rootwrap and provide a proper privilege separated, formal RPC mechanism for each project. ... we should have a nova-compute-worker daemon running as root, that accepts an RPC command from nova-compute running unprivileged. eg CreateImage(instane0001, qcow2, disk.qcow) ... This is certainly alot more work than trying to patchup rootwrap, but it would provide a level of security that rootwrap can never achieve IMHO. A lot of work, and if input sanitation didn't work in one piece of code, why should it here? I think this only leads to _more_ work, without any real benefit. If we can't get the filters right the first round, we won't make it here either. Regarding the idea of using containers ... take Cinder as an example. If the cinder container can access *all* the VM data, why should someone bother to get *out* of the container? Everything that she wants is already here... I'm not sure what the containers would buy us, but perhaps I just don't understand something here. So, IMO, solution 1 (one) would be the way to go ... it gets to security asymptotically (and might never reach it), but at least it provides a bit of help. And if the rootwrap filter specification would be linked to in the rootwrap config files, it would help newcomers to see the available syntax, instead of simply copying a bad example ;P -- : Ing. Philipp Marek : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com : DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
Thierry Carrez thie...@openstack.org writes: You make a good point when you mention traditional distro here. I would argue that containers are slightly changing the rules of the don't-run-as-root game. Solution (2) aligns pretty well with container-powered OpenStack deployments -- running compute nodes as root in a container (and embracing abovementioned simplicity/performance gains) sounds like a pretty strong combo. This sounds at least a little like a suggestion that containers are a substitute for the security provided by running non-root. The security landscape around containers is complex, and while there are a lot of benefits, I believe the general consensus is that uid 0 processes should not be seen as fully isolated. From https://docs.docker.com/articles/security/ : Docker containers are, by default, quite secure; especially if you take care of running your processes inside the containers as non-privileged users (i.e., non-root). Which is not to say that using containers is not a good idea, but rather, if one does, one should avoid running as root (perhaps with capabilities), and use selinux (or similar). -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On Wed, Feb 04, 2015 at 09:10:06AM -0800, James E. Blair wrote: Thierry Carrez thie...@openstack.org writes: You make a good point when you mention traditional distro here. I would argue that containers are slightly changing the rules of the don't-run-as-root game. Solution (2) aligns pretty well with container-powered OpenStack deployments -- running compute nodes as root in a container (and embracing abovementioned simplicity/performance gains) sounds like a pretty strong combo. This sounds at least a little like a suggestion that containers are a substitute for the security provided by running non-root. The security landscape around containers is complex, and while there are a lot of benefits, I believe the general consensus is that uid 0 processes should not be seen as fully isolated. From https://docs.docker.com/articles/security/ : Docker containers are, by default, quite secure; especially if you take care of running your processes inside the containers as non-privileged users (i.e., non-root). Which is not to say that using containers is not a good idea, but rather, if one does, one should avoid running as root (perhaps with capabilities), and use selinux (or similar). Yep, I've seen attempts by some folks to run nova-compute and libvirtd and QEMU inside a docker container. Because of the inherantly privileged nature of what Nova/libvirt/qemu need to do, you end up having to share all the host namespaces with the docker container, except for the filesystem namespace and even that you need to bind mount a bunch of stuff over. As a result the container isn't really offerring any security benefit over running the things outside the container. IOW the use of containers to confine nova is only solving a managability problem rather than a security problem. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
(4) I think that ultimately we need to ditch rootwrap and provide a proper privilege separated, formal RPC mechanism for each project. ... we should have a nova-compute-worker daemon running as root, that accepts an RPC command from nova-compute running unprivileged. eg CreateImage(instane0001, qcow2, disk.qcow) ... This is certainly alot more work than trying to patchup rootwrap, but it would provide a level of security that rootwrap can never achieve IMHO. A lot of work, and if input sanitation didn't work in one piece of code, why should it here? I think this only leads to _more_ work, without any real benefit. If we can't get the filters right the first round, we won't make it here either. The difference is that the API I illustrate here has *semantic* context about the operation. In the API example the caller is not permitted to provide a directory path - only the name of the instance and the name of the disk image. The privileged nova-compute-worker program can thus enforce exactly what directory the image is created in, and ensure it doesn't clash with a disk image from another VM. This kind of validation is impractical when you are just given a 'qemu-img' command line args with a full directory path, so there is no semantic conext for the privileged rootwrap to know whether it is reasonable to create the disk image in that particular location. Sorry about being unclear. Yes, there's some semantic meaning at that level. But this level already exists at the current rootwrap caller site, too - and if that one can be tricked to do something against image.img rm -rf /, then the additional layer can be tricked, too. I'm trying to get at the point everything that can be forgot to check at the rootwrap call site now, can be forgotten in the additional API too. So let's get the current call sites tight, and we're done. (Ha!) -- : Ing. Philipp Marek : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com : DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
If I'm reading that correctly, it does not help with the filtering issues at all, since it needs exactly the same kind of filter. Daniel explained the concept far better than I. On 4 February 2015 at 18:33, Jeremy Stanley fu...@yuggoth.org wrote: On 2015-02-04 13:40:29 +0200 (+0200), Duncan Thomas wrote: 4) Write a small daemon that runs as root, accepting commands over a unix domain socket or similar. Easier to audit, less code running as root. http://git.openstack.org/cgit/openstack/oslo.rootwrap/tree/oslo_rootwrap/daemon.py -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Duncan Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
Excerpts from Tristan Cacqueray's message of 2015-02-04 09:02:19 -0800: On 02/04/2015 06:57 AM, Daniel P. Berrange wrote: On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote: What solutions do we have ? (1) we could get our act together and audit and fix those filter definitions. Remove superfluous usage of root rights, make use of advanced filters for where we actually need them. We have been preaching for that at many many design summits. This is a lot of work though... There were such efforts in the past, but they were never completed for some types of nodes. Worse, the bad filter definitions kept coming back, since developers take shortcuts, reviewers may not have sufficient security awareness to detect crappy filter definitions, and I don't think we can design a gate test that would have such awareness. (2) bite the bullet and accept that some types of nodes actually need root rights for so many different things, they should just run as root anyway. I know a few distributions which won't be very pleased by such a prospect, but that would be a more honest approach (rather than claiming we provide efficient isolation when we really don't). An added benefit is that we could replace a number of shell calls by Python code, which would simplify the code and increase performance. (3) intermediary solution where we would run as the nova user but run sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND). That would leave it up to distros to choose between a blanket sudoer or maintain their own filtering rules. I think it's a bit hypocritical though (pretend the distros could filter if they wanted it, when we dropped the towel on doing that ourselves). I'm also not convinced it's more secure than solution 2, and it prevents from reducing the number of shell-outs, which I think is a worthy idea. In all cases I would not drop the baby with the bath water, and keep rootwrap for all the cases where root rights are needed on a very specific set of commands (like neutron, or nova's api-metadata). The daemon mode should address the performance issue for the projects making a lot of calls. (4) I think that ultimately we need to ditch rootwrap and provide a proper privilege separated, formal RPC mechanism for each project. eg instead of having a rootwrap command, or rootwrap server attempting to validate safety of qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2 we should have a nova-compute-worker daemon running as root, that accepts an RPC command from nova-compute running unprivileged. eg CreateImage(instane0001, qcow2, disk.qcow) This immediately makes it trivial to validate that we're not trying to trick qemu-img into overwriting some key system file. This is certainly alot more work than trying to patchup rootwrap, but it would provide a level of security that rootwrap can never achieve IMHO. This 4th idea sounds interesting, though we are assuming this new service running as root would be exempt of bug, especially if it uses the same libraries as non-root services... For example a major bug in python would give attacker direct root access while the rootwrap solution would in theory keep the intruder at the sudo level... I don't believe that anyone assumes the new service would be without bugs. But just like the OpenSSH team saw years ago, privilege separation means that you can absolutely know what is running as root, and what is not. So when you decide to commit your resources to code audits, you _start_ with the things that run with elevated privileges. For completeness, I'd like to propose a more long-term solution: (5) Get ride of root! Seriously, OpenStack could support security mechanism like SELinux or AppArmor in order to properly isolate service and let them run what they need to run. For what it worth, the underlying issue here is having a single almighty super user: root and thus we should, at least, consider solution that remove the need of such powers (e.g. kernel module loading, ptrace or raw socket). We don't need a security module to drop all of those capabilities entirely and run as a hobbled root user. By my measure, this process for nova-compute would only need CAP_NET_ADMIN, CAP_SYS_ADMIN and CAP_KILL. These capabilities can be audited per-agent and even verified as needed simply by running integration tests without each one to see what breaks. Beside, as long as sensitive process are not contained at the system level, the attack surface for a non-root user is still very wide (e.g. system calls, setuid binaries, ipc, ...) While this might sounds impossible to implement upstream because it's too vendor specific or just because of other technicals difficulties, I guess it still deserves a mention in this thread. I think OpenStack can do its part by making privilege
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
Excerpts from Daniel P. Berrange's message of 2015-02-04 03:57:53 -0800: On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote: The first one is performance -- each call would spawn a Python interpreter which would then call the system command. This was fine when there were just a few calls here and there, not so much when it's called a hundred times in a row. During the Juno cycle, a daemon mode was added to solve this issue. It is significantly faster than running sudo directly (the often-suggested alternative). Projects still have to start adopting it though. Neutron and Cinder have started work to do that in Kilo. The second problem is the quality of the filter definitions. Rootwrap is a framework to enable isolation. It's only as good as the filters each project defines. Most of them rely on CommandFilters that do not check any argument, instead of using more powerful filters (which are arguably more painful to maintain). Developers routinely add filter definitions that basically remove any isolation that might have been there, like allowing blank dd, tee, chown or chmod. I think this is really the key point which shows rootwrap as a concept is broken by design IMHO. Root wrap is essentially trying to provide an API for invoking privileged operations, but instead of actually designing an explicit API for the operations, we done by implicit one based on command args. From a security POV I think this approach is doomed to failure, but command arg strings are fr to expressive a concept to deal with. What solutions do we have ? (1) we could get our act together and audit and fix those filter definitions. Remove superfluous usage of root rights, make use of advanced filters for where we actually need them. We have been preaching for that at many many design summits. This is a lot of work though... There were such efforts in the past, but they were never completed for some types of nodes. Worse, the bad filter definitions kept coming back, since developers take shortcuts, reviewers may not have sufficient security awareness to detect crappy filter definitions, and I don't think we can design a gate test that would have such awareness. (2) bite the bullet and accept that some types of nodes actually need root rights for so many different things, they should just run as root anyway. I know a few distributions which won't be very pleased by such a prospect, but that would be a more honest approach (rather than claiming we provide efficient isolation when we really don't). An added benefit is that we could replace a number of shell calls by Python code, which would simplify the code and increase performance. (3) intermediary solution where we would run as the nova user but run sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND). That would leave it up to distros to choose between a blanket sudoer or maintain their own filtering rules. I think it's a bit hypocritical though (pretend the distros could filter if they wanted it, when we dropped the towel on doing that ourselves). I'm also not convinced it's more secure than solution 2, and it prevents from reducing the number of shell-outs, which I think is a worthy idea. In all cases I would not drop the baby with the bath water, and keep rootwrap for all the cases where root rights are needed on a very specific set of commands (like neutron, or nova's api-metadata). The daemon mode should address the performance issue for the projects making a lot of calls. (4) I think that ultimately we need to ditch rootwrap and provide a proper privilege separated, formal RPC mechanism for each project. eg instead of having a rootwrap command, or rootwrap server attempting to validate safety of qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2 we should have a nova-compute-worker daemon running as root, that accepts an RPC command from nova-compute running unprivileged. eg CreateImage(instane0001, qcow2, disk.qcow) This immediately makes it trivial to validate that we're not trying to trick qemu-img into overwriting some key system file. This is certainly alot more work than trying to patchup rootwrap, but it would provide a level of security that rootwrap can never achieve IMHO. +1, I think you're right on Daniel. Count me in for future discussions and work on this. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 5 February 2015 at 03:33, Monty Taylor mord...@inaugust.com wrote: On 02/04/2015 06:57 AM, Daniel P. Berrange wrote: to manage VMs on a laptop - you're going to use virtualbox or virt-manager. You're going to use nova-compute to manage compute hosts in a cloud - and in almost all circumstances the only thing that's going to be running on your compute hosts is going to be nova-compute. Actually we know from user / operator stories at summits that many groups run *much more* than nova-compute on their compute hosts. Specifically the all-in-one scale-as-a-unit architecture is quite popular. That has all the components of cinder/swift/keystone/nova/neutron all running on all machines, and the cluster is scaled by adding just another identical machine. I suspect its actually really quite common outside of a) paranoid (but not necessarily wrong :)) and b) ultra-scale environments. ... I'm with Daniel - (4) I think that ultimately we need to ditch rootwrap and provide a proper privilege separated, formal RPC mechanism for each project. eg instead of having a rootwrap command, or rootwrap server attempting to validate safety of qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2 we should have a nova-compute-worker daemon running as root, that accepts an RPC command from nova-compute running unprivileged. eg CreateImage(instane0001, qcow2, disk.qcow) This immediately makes it trivial to validate that we're not trying to trick qemu-img into overwriting some key system file. This is certainly alot more work than trying to patchup rootwrap, but it would provide a level of security that rootwrap can never achieve IMHO. I think a local root daemon is a solid idea, there's lots and lots of prior art on this, from the bazillion dbus daemons we haveadays to the Android isolation between apps. So count me in too on design and speccing for this. I realise that for some cases rootwrap was absolutely a performance and stability issue, but daemon mode should address that - so I think this is a longer term project, giving us a significant step up in security. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote: The first one is performance -- each call would spawn a Python interpreter which would then call the system command. This was fine when there were just a few calls here and there, not so much when it's called a hundred times in a row. During the Juno cycle, a daemon mode was added to solve this issue. It is significantly faster than running sudo directly (the often-suggested alternative). Projects still have to start adopting it though. Neutron and Cinder have started work to do that in Kilo. The second problem is the quality of the filter definitions. Rootwrap is a framework to enable isolation. It's only as good as the filters each project defines. Most of them rely on CommandFilters that do not check any argument, instead of using more powerful filters (which are arguably more painful to maintain). Developers routinely add filter definitions that basically remove any isolation that might have been there, like allowing blank dd, tee, chown or chmod. I think this is really the key point which shows rootwrap as a concept is broken by design IMHO. Root wrap is essentially trying to provide an API for invoking privileged operations, but instead of actually designing an explicit API for the operations, we done by implicit one based on command args. From a security POV I think this approach is doomed to failure, but command arg strings are fr to expressive a concept to deal with. What solutions do we have ? (1) we could get our act together and audit and fix those filter definitions. Remove superfluous usage of root rights, make use of advanced filters for where we actually need them. We have been preaching for that at many many design summits. This is a lot of work though... There were such efforts in the past, but they were never completed for some types of nodes. Worse, the bad filter definitions kept coming back, since developers take shortcuts, reviewers may not have sufficient security awareness to detect crappy filter definitions, and I don't think we can design a gate test that would have such awareness. (2) bite the bullet and accept that some types of nodes actually need root rights for so many different things, they should just run as root anyway. I know a few distributions which won't be very pleased by such a prospect, but that would be a more honest approach (rather than claiming we provide efficient isolation when we really don't). An added benefit is that we could replace a number of shell calls by Python code, which would simplify the code and increase performance. (3) intermediary solution where we would run as the nova user but run sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND). That would leave it up to distros to choose between a blanket sudoer or maintain their own filtering rules. I think it's a bit hypocritical though (pretend the distros could filter if they wanted it, when we dropped the towel on doing that ourselves). I'm also not convinced it's more secure than solution 2, and it prevents from reducing the number of shell-outs, which I think is a worthy idea. In all cases I would not drop the baby with the bath water, and keep rootwrap for all the cases where root rights are needed on a very specific set of commands (like neutron, or nova's api-metadata). The daemon mode should address the performance issue for the projects making a lot of calls. (4) I think that ultimately we need to ditch rootwrap and provide a proper privilege separated, formal RPC mechanism for each project. eg instead of having a rootwrap command, or rootwrap server attempting to validate safety of qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2 we should have a nova-compute-worker daemon running as root, that accepts an RPC command from nova-compute running unprivileged. eg CreateImage(instane0001, qcow2, disk.qcow) This immediately makes it trivial to validate that we're not trying to trick qemu-img into overwriting some key system file. This is certainly alot more work than trying to patchup rootwrap, but it would provide a level of security that rootwrap can never achieve IMHO. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
I suppose that the security argument against running the whole of nova-compute as root is that a remote exploit in the service is much better constrained when the thing isn't running as root - e.g. some input validation fails and allows arbitrary shell in some (currently none-root) command via an existing API call. This means that (2) is the worst possible option - calling out to sudo at least limits the attach surface to those code paths that do those specific exec calls. I think Daniel's suggestion makes a great deal more sense from a security point of view, but (3) is a significantly better architecture than (2) even if it still has some holes. On 4 February 2015 at 16:33, Monty Taylor mord...@inaugust.com wrote: On 02/04/2015 06:57 AM, Daniel P. Berrange wrote: On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote: The first one is performance -- each call would spawn a Python interpreter which would then call the system command. This was fine when there were just a few calls here and there, not so much when it's called a hundred times in a row. During the Juno cycle, a daemon mode was added to solve this issue. It is significantly faster than running sudo directly (the often-suggested alternative). Projects still have to start adopting it though. Neutron and Cinder have started work to do that in Kilo. The second problem is the quality of the filter definitions. Rootwrap is a framework to enable isolation. It's only as good as the filters each project defines. Most of them rely on CommandFilters that do not check any argument, instead of using more powerful filters (which are arguably more painful to maintain). Developers routinely add filter definitions that basically remove any isolation that might have been there, like allowing blank dd, tee, chown or chmod. I think this is really the key point which shows rootwrap as a concept is broken by design IMHO. Root wrap is essentially trying to provide an API for invoking privileged operations, but instead of actually designing an explicit API for the operations, we done by implicit one based on command args. From a security POV I think this approach is doomed to failure, but command arg strings are fr to expressive a concept to deal with. What solutions do we have ? (1) we could get our act together and audit and fix those filter definitions. Remove superfluous usage of root rights, make use of advanced filters for where we actually need them. We have been preaching for that at many many design summits. This is a lot of work though... There were such efforts in the past, but they were never completed for some types of nodes. Worse, the bad filter definitions kept coming back, since developers take shortcuts, reviewers may not have sufficient security awareness to detect crappy filter definitions, and I don't think we can design a gate test that would have such awareness. (2) bite the bullet and accept that some types of nodes actually need root rights for so many different things, they should just run as root anyway. I know a few distributions which won't be very pleased by such a prospect, but that would be a more honest approach (rather than claiming we provide efficient isolation when we really don't). An added benefit is that we could replace a number of shell calls by Python code, which would simplify the code and increase performance. I'm actually the biggest fan of this solution (even more than Daniel's suggestion below) because it's the thing that is closest to reality. Security isn't a useful concept in a vacuum - it's something we do to prevent access to or damage resources that we don't want accessed by the wrong people. On compute nodes, the main valuable thing are the VMs themselves- and I'd expect the most interested target of an attack to be interested in manipulating, stealing data from or deleting the VMs. No amount of rootwrap or privileges are going to prevent nova-compute from performing unwanted actions on the VMs in its control - for the reason that it's job in life is to manipulate those things. Is it a security hole in the traditional distro sense - that we want to be able to install all of these things with apt-get or yum on a single server and have the actions of one service not affect the state of another? Sure. Is it in the real world? No. You're not going to use this to manage VMs on a laptop - you're going to use virtualbox or virt-manager. You're going to use nova-compute to manage compute hosts in a cloud - and in almost all circumstances the only thing that's going to be running on your compute hosts is going to be nova-compute. (3) intermediary solution where we would run as the nova user but run sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND). That would leave it up to distros to choose between a blanket sudoer or maintain their own filtering rules. I think
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
Monty Taylor wrote: On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote: (2) bite the bullet and accept that some types of nodes actually need root rights for so many different things, they should just run as root anyway. I know a few distributions which won't be very pleased by such a prospect, but that would be a more honest approach (rather than claiming we provide efficient isolation when we really don't). An added benefit is that we could replace a number of shell calls by Python code, which would simplify the code and increase performance. I'm actually the biggest fan of this solution (even more than Daniel's suggestion below) because it's the thing that is closest to reality. Security isn't a useful concept in a vacuum - it's something we do to prevent access to or damage resources that we don't want accessed by the wrong people. On compute nodes, the main valuable thing are the VMs themselves- and I'd expect the most interested target of an attack to be interested in manipulating, stealing data from or deleting the VMs. No amount of rootwrap or privileges are going to prevent nova-compute from performing unwanted actions on the VMs in its control - for the reason that it's job in life is to manipulate those things. Is it a security hole in the traditional distro sense - that we want to be able to install all of these things with apt-get or yum on a single server and have the actions of one service not affect the state of another? Sure. Is it in the real world? No. You're not going to use this to manage VMs on a laptop - you're going to use virtualbox or virt-manager. You're going to use nova-compute to manage compute hosts in a cloud - and in almost all circumstances the only thing that's going to be running on your compute hosts is going to be nova-compute. You make a good point when you mention traditional distro here. I would argue that containers are slightly changing the rules of the don't-run-as-root game. Solution (2) aligns pretty well with container-powered OpenStack deployments -- running compute nodes as root in a container (and embracing abovementioned simplicity/performance gains) sounds like a pretty strong combo. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
Hi, This is the follow-up of the discussion we started yesterday on rootwrap usage at the cross-project meeting. A bit of history to stage this story first. OpenStack nodes sometimes need to run things with elevated privileges. Nova started out as calling sudo shell commands to execute those, which basically allowed it to run anything as root. Distributions didn't really like that, so packagers started to ship sudoers files that restricted the commands Nova could run with sudo. It created a maintenance nightmare, where the code and the packaging were routinely out of sync. Rootwrap was designed so that we would maintain the allowed commands in the code itself, which facilitated maintenance. It would also allow more complex rules than sudoers allowed, like the ability to only kill a certain executable processes. Rootwrap was then adopted by Cinder and Neutron, and moved to oslo to avoid code duplication. There were still two long-standing problems with it, though. The first one is performance -- each call would spawn a Python interpreter which would then call the system command. This was fine when there were just a few calls here and there, not so much when it's called a hundred times in a row. During the Juno cycle, a daemon mode was added to solve this issue. It is significantly faster than running sudo directly (the often-suggested alternative). Projects still have to start adopting it though. Neutron and Cinder have started work to do that in Kilo. The second problem is the quality of the filter definitions. Rootwrap is a framework to enable isolation. It's only as good as the filters each project defines. Most of them rely on CommandFilters that do not check any argument, instead of using more powerful filters (which are arguably more painful to maintain). Developers routinely add filter definitions that basically remove any isolation that might have been there, like allowing blank dd, tee, chown or chmod. Basically, some nodes run such a variety of commands as root that maintaining proper isolation is a non-trivial amount of work: that's the case for Nova's compute nodes and Cinder's volume nodes. I would argue that for all other types of nodes, rootwrap is still an appropriate way to provide isolation. But for nodes which need to make a very wide variety of root calls, we pretend to isolate, but the filter definitions are so full of holes that we actually don't. What solutions do we have ? (1) we could get our act together and audit and fix those filter definitions. Remove superfluous usage of root rights, make use of advanced filters for where we actually need them. We have been preaching for that at many many design summits. This is a lot of work though... There were such efforts in the past, but they were never completed for some types of nodes. Worse, the bad filter definitions kept coming back, since developers take shortcuts, reviewers may not have sufficient security awareness to detect crappy filter definitions, and I don't think we can design a gate test that would have such awareness. (2) bite the bullet and accept that some types of nodes actually need root rights for so many different things, they should just run as root anyway. I know a few distributions which won't be very pleased by such a prospect, but that would be a more honest approach (rather than claiming we provide efficient isolation when we really don't). An added benefit is that we could replace a number of shell calls by Python code, which would simplify the code and increase performance. (3) intermediary solution where we would run as the nova user but run sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND). That would leave it up to distros to choose between a blanket sudoer or maintain their own filtering rules. I think it's a bit hypocritical though (pretend the distros could filter if they wanted it, when we dropped the towel on doing that ourselves). I'm also not convinced it's more secure than solution 2, and it prevents from reducing the number of shell-outs, which I think is a worthy idea. In all cases I would not drop the baby with the bath water, and keep rootwrap for all the cases where root rights are needed on a very specific set of commands (like neutron, or nova's api-metadata). The daemon mode should address the performance issue for the projects making a lot of calls. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 02/04/2015 06:57 AM, Daniel P. Berrange wrote: On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote: The first one is performance -- each call would spawn a Python interpreter which would then call the system command. This was fine when there were just a few calls here and there, not so much when it's called a hundred times in a row. During the Juno cycle, a daemon mode was added to solve this issue. It is significantly faster than running sudo directly (the often-suggested alternative). Projects still have to start adopting it though. Neutron and Cinder have started work to do that in Kilo. The second problem is the quality of the filter definitions. Rootwrap is a framework to enable isolation. It's only as good as the filters each project defines. Most of them rely on CommandFilters that do not check any argument, instead of using more powerful filters (which are arguably more painful to maintain). Developers routinely add filter definitions that basically remove any isolation that might have been there, like allowing blank dd, tee, chown or chmod. I think this is really the key point which shows rootwrap as a concept is broken by design IMHO. Root wrap is essentially trying to provide an API for invoking privileged operations, but instead of actually designing an explicit API for the operations, we done by implicit one based on command args. From a security POV I think this approach is doomed to failure, but command arg strings are fr to expressive a concept to deal with. What solutions do we have ? (1) we could get our act together and audit and fix those filter definitions. Remove superfluous usage of root rights, make use of advanced filters for where we actually need them. We have been preaching for that at many many design summits. This is a lot of work though... There were such efforts in the past, but they were never completed for some types of nodes. Worse, the bad filter definitions kept coming back, since developers take shortcuts, reviewers may not have sufficient security awareness to detect crappy filter definitions, and I don't think we can design a gate test that would have such awareness. (2) bite the bullet and accept that some types of nodes actually need root rights for so many different things, they should just run as root anyway. I know a few distributions which won't be very pleased by such a prospect, but that would be a more honest approach (rather than claiming we provide efficient isolation when we really don't). An added benefit is that we could replace a number of shell calls by Python code, which would simplify the code and increase performance. I'm actually the biggest fan of this solution (even more than Daniel's suggestion below) because it's the thing that is closest to reality. Security isn't a useful concept in a vacuum - it's something we do to prevent access to or damage resources that we don't want accessed by the wrong people. On compute nodes, the main valuable thing are the VMs themselves- and I'd expect the most interested target of an attack to be interested in manipulating, stealing data from or deleting the VMs. No amount of rootwrap or privileges are going to prevent nova-compute from performing unwanted actions on the VMs in its control - for the reason that it's job in life is to manipulate those things. Is it a security hole in the traditional distro sense - that we want to be able to install all of these things with apt-get or yum on a single server and have the actions of one service not affect the state of another? Sure. Is it in the real world? No. You're not going to use this to manage VMs on a laptop - you're going to use virtualbox or virt-manager. You're going to use nova-compute to manage compute hosts in a cloud - and in almost all circumstances the only thing that's going to be running on your compute hosts is going to be nova-compute. (3) intermediary solution where we would run as the nova user but run sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND). That would leave it up to distros to choose between a blanket sudoer or maintain their own filtering rules. I think it's a bit hypocritical though (pretend the distros could filter if they wanted it, when we dropped the towel on doing that ourselves). I'm also not convinced it's more secure than solution 2, and it prevents from reducing the number of shell-outs, which I think is a worthy idea. In all cases I would not drop the baby with the bath water, and keep rootwrap for all the cases where root rights are needed on a very specific set of commands (like neutron, or nova's api-metadata). The daemon mode should address the performance issue for the projects making a lot of calls. (4) I think that ultimately we need to ditch rootwrap and provide a proper privilege separated, formal RPC mechanism for each project. eg instead of having a rootwrap command, or rootwrap server attempting
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
4) Write a small daemon that runs as root, accepting commands over a unix domain socket or similar. Easier to audit, less code running as root. On 4 February 2015 at 12:58, Thierry Carrez thie...@openstack.org wrote: Hi, This is the follow-up of the discussion we started yesterday on rootwrap usage at the cross-project meeting. A bit of history to stage this story first. OpenStack nodes sometimes need to run things with elevated privileges. Nova started out as calling sudo shell commands to execute those, which basically allowed it to run anything as root. Distributions didn't really like that, so packagers started to ship sudoers files that restricted the commands Nova could run with sudo. It created a maintenance nightmare, where the code and the packaging were routinely out of sync. Rootwrap was designed so that we would maintain the allowed commands in the code itself, which facilitated maintenance. It would also allow more complex rules than sudoers allowed, like the ability to only kill a certain executable processes. Rootwrap was then adopted by Cinder and Neutron, and moved to oslo to avoid code duplication. There were still two long-standing problems with it, though. The first one is performance -- each call would spawn a Python interpreter which would then call the system command. This was fine when there were just a few calls here and there, not so much when it's called a hundred times in a row. During the Juno cycle, a daemon mode was added to solve this issue. It is significantly faster than running sudo directly (the often-suggested alternative). Projects still have to start adopting it though. Neutron and Cinder have started work to do that in Kilo. The second problem is the quality of the filter definitions. Rootwrap is a framework to enable isolation. It's only as good as the filters each project defines. Most of them rely on CommandFilters that do not check any argument, instead of using more powerful filters (which are arguably more painful to maintain). Developers routinely add filter definitions that basically remove any isolation that might have been there, like allowing blank dd, tee, chown or chmod. Basically, some nodes run such a variety of commands as root that maintaining proper isolation is a non-trivial amount of work: that's the case for Nova's compute nodes and Cinder's volume nodes. I would argue that for all other types of nodes, rootwrap is still an appropriate way to provide isolation. But for nodes which need to make a very wide variety of root calls, we pretend to isolate, but the filter definitions are so full of holes that we actually don't. What solutions do we have ? (1) we could get our act together and audit and fix those filter definitions. Remove superfluous usage of root rights, make use of advanced filters for where we actually need them. We have been preaching for that at many many design summits. This is a lot of work though... There were such efforts in the past, but they were never completed for some types of nodes. Worse, the bad filter definitions kept coming back, since developers take shortcuts, reviewers may not have sufficient security awareness to detect crappy filter definitions, and I don't think we can design a gate test that would have such awareness. (2) bite the bullet and accept that some types of nodes actually need root rights for so many different things, they should just run as root anyway. I know a few distributions which won't be very pleased by such a prospect, but that would be a more honest approach (rather than claiming we provide efficient isolation when we really don't). An added benefit is that we could replace a number of shell calls by Python code, which would simplify the code and increase performance. (3) intermediary solution where we would run as the nova user but run sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND). That would leave it up to distros to choose between a blanket sudoer or maintain their own filtering rules. I think it's a bit hypocritical though (pretend the distros could filter if they wanted it, when we dropped the towel on doing that ourselves). I'm also not convinced it's more secure than solution 2, and it prevents from reducing the number of shell-outs, which I think is a worthy idea. In all cases I would not drop the baby with the bath water, and keep rootwrap for all the cases where root rights are needed on a very specific set of commands (like neutron, or nova's api-metadata). The daemon mode should address the performance issue for the projects making a lot of calls. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Duncan Thomas
