Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 2/4/15, 10:24 AM, "Daniel P. Berrange" wrote: >On Wed, Feb 04, 2015 at 09:10:06AM -0800, James E. Blair wrote: >> Thierry Carrez writes: >> >> > You make a good point when you mention "traditional distro" here. I >> > would argue that containers are slightly changing the rules of the >> > don't-run-as-root game. >> > >> > Solution (2) aligns pretty well with container-powered OpenStack >> > deployments -- running compute nodes as root in a container (and >> > embracing abovementioned simplicity/performance gains) sounds like a >> > pretty strong combo. >> >> This sounds at least a little like a suggestion that containers are a >> substitute for the security provided by running non-root. The security >> landscape around containers is complex, and while there are a lot of >> benefits, I believe the general consensus is that uid 0 processes should >> not be seen as fully isolated. >> >> From https://docs.docker.com/articles/security/ : >> >> Docker containers are, by default, quite secure; especially if you >> take care of running your processes inside the containers as >> non-privileged users (i.e., non-root). >> >> Which is not to say that using containers is not a good idea, but >> rather, if one does, one should avoid running as root (perhaps with >> capabilities), and use selinux (or similar). > >Yep, I've seen attempts by some folks to run nova-compute and libvirtd >and QEMU inside a docker container. Because of the inherantly privileged >nature of what Nova/libvirt/qemu need to do, you end up having to share >all the host namespaces with the docker container, except for the >filesystem >namespace and even that you need to bind mount a bunch of stuff over. As >a result the container isn't really offerring any security benefit over >running the things outside the container. IOW the use of containers to >confine nova is only solving a managability problem rather than a security >problem. > >Regards, >Daniel Daniel, Agree 100% - compute in containers is all about an atomic image-based upgrade and downgrade process, not about solving security problems. Ideally the services that are within containers are as secure as running natively without containers on bare metal although this might be a bit of assumption since docker does run with all Linux capabilities enabled. Regards -steve >-- >|: http://berrange.com -o- >http://www.flickr.com/photos/dberrange/ :| >|: http://libvirt.org -o- >http://virt-manager.org :| >|: http://autobuild.org -o- >http://search.cpan.org/~danberr/ :| >|: http://entangle-photo.org -o- >http://live.gnome.org/gtk-vnc :| > >__ >OpenStack Development Mailing List (not for usage questions) >Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On Thu, 2015-02-05 at 08:27 -0500, Tristan Cacqueray wrote: > Thus if we want to emulate OpenSSH design, the rpc call would also > need to > carry authentication data in order to prevent unwanted activity. And > the > rpc daemon would then need to enforce some kind of acl/policy. Sounds a lot like what the DBus system bus implements/provides to me... And it took quite some time to get that done. (Note: I'm not proposing its usage in any way) Nicolas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 02/04/2015 01:13 PM, Clint Byrum wrote:
> Excerpts from Tristan Cacqueray's message of 2015-02-04 09:02:19 -0800:
>> On 02/04/2015 06:57 AM, Daniel P. Berrange wrote:
>>> (4) I think that ultimately we need to ditch rootwrap and provide a proper
>>> privilege separated, formal RPC mechanism for each project.
>>>
>>> eg instead of having a rootwrap command, or rootwrap server attempting
>>> to validate safety of
>>>
>>> qemu-img create -f qcow2
>>> /var/lib/nova/instances/instance1/disk.qcow2
>>>
>>> we should have a nova-compute-worker daemon running as root, that accepts
>>> an RPC command from nova-compute running unprivileged. eg
>>>
>>> CreateImage("instane0001", "qcow2", "disk.qcow")
>>>
>>> This immediately makes it trivial to validate that we're not trying to
>>> trick qemu-img into overwriting some key system file.
>>>
>>> This is certainly alot more work than trying to patchup rootwrap, but
>>> it would provide a level of security that rootwrap can never achieve IMHO.
>>>
>>
>> This 4th idea sounds interesting, though we are assuming this new service
>> running as root would be exempt of bug, especially if it uses the same
>> libraries as non-root services... For example a major bug in python would
>> give attacker direct root access while the rootwrap solution would in
>> theory keep the intruder at the sudo level...
>>
>
> I don't believe that anyone assumes the new service would be without
> bugs.
I meant less bug than another solution. Such RPC service daemon will
eventually requires quite some code to be robust, which could lead to
more bugs.
> But just like the OpenSSH team saw years ago, privilege separation
> means that you can absolutely know what is running as root, and what is
> not. So when you decide to commit your resources to code audits, you
> _start_ with the things that run with elevated privileges.
>
Not quite sure to follow here... OpenSSH is using privilege separation after
authentication, e.g. the root-based process is doing the authentication while
the external data processing is done through an unprivileged process.
If I understand correctly Daniel's solution (4), it's to have a semantic on the
privileged actions to avoid mis-usage (like inject command in a sudo call).
Thus if we want to emulate OpenSSH design, the rpc call would also need to
carry authentication data in order to prevent unwanted activity. And the
rpc daemon would then need to enforce some kind of acl/policy.
The amount of code to be audited could arguably be higher with such design.
>>
>> For completeness, I'd like to propose a more long-term solution:
>>
>> (5) Get ride of root! Seriously, OpenStack could support security mechanism
>> like SELinux or AppArmor in order to properly isolate service and let them
>> run what they need to run.
>>
>> For what it worth, the underlying issue here is having a single almighty
>> super
>> user: root and thus we should, at least, consider solution that remove the
>> need of such powers (e.g. kernel module loading, ptrace or raw socket).
>>
>
> We don't need a security module to drop all of those capabilities
> entirely and run as a hobbled root user. By my measure, this process for
> nova-compute would only need CAP_NET_ADMIN, CAP_SYS_ADMIN and CAP_KILL.
> These capabilities can be audited per-agent and even verified as needed
> simply by running integration tests without each one to see what breaks.
>
Capabilities might be a candidate as well, but they don't cover every cases
and lack granularity...
SECCOMP filters might be good enough too.
>> Beside, as long as sensitive process are not contained at the system level,
>> the attack surface for a non-root user is still very wide (e.g. system calls,
>> setuid binaries, ipc, ...)
>>
>>
>> While this might sounds impossible to implement upstream because it's too
>> vendor specific or just because of other technicals difficulties,
>> I guess it still deserves a mention in this thread.
>>
>
> I think OpenStack can do its part by making privilege separation a
> reality for the security sensitive parts of OpenStack, and that will
> ease pressure to implement strong controls in any security modules the
> Linux distros ship with.
>
That would be a great step forward indeed. Having some kind of privilege
separation would surely help a lot to properly configure security modules.
Regards,
Tristan
signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
Hi Daniel,
> > Yes, there's some semantic meaning at that level. But this level already
> > exists at the current rootwrap caller site, too - and if that one can be
> > tricked to do something against "image.img & rm -rf /", then the additional
> > layer can be tricked, too.
>
> No, that is really not correct. If you are passing full command strings to
> rootwrap then the caller can trick rootwrap into running commands with those
> shell metacharacter exploits. If you have a formal API like the one I
> describe
> and correctly implement it, there would be no shell involved at all. ie the
> nova-compute-worker program would directly invoke the system call
>
>execve("/usr/bin/qemu-img", "create", "image.img & rm -rf /")
>
> and this would at worse create a file called 'image.img & rm -rf /' and
> *not* invoke the "rm" command as you get when you use shell.
>
> This is really just another example of why rootwrap/sudo as a concept is a
> bad idea. The shell should never be involved in executing any external
> commands that Nova/Neutron/etc need to run, because it is impractical to
> correctly validate shell commands anywhere in the stack. The only safe
> thing todo is to take shell out of the picture entirely.
that was just an example.
If cinder calls rootwrap with a list of arguments, and this is then called
via sudo, there's no shell inbetween either.
But now say there's something else, like the bash functions bug, and
some codepath runs a command in a way that sets the environment - then any
grandchild processes might cause a breach.
I'm not arguing that *this exact string* is a problem.
My point is that the cinder/nova code already have as much information as
possible, and *any* library abstraction can just remove some available
data.
So, besides increasing the LOC, it won't get more secure.
I'm arguing that the rootwrap call sites need to be fixed, ie. they need to
"proof" in some way that the arguments they pass on are sane - that's more
or less the same thing that this new library would need to do too.
Regards,
Phil
--
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com :
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 5 February 2015 at 03:33, Monty Taylor wrote:
> On 02/04/2015 06:57 AM, Daniel P. Berrange wrote:
> to manage VMs on a laptop - you're going to use virtualbox or
> virt-manager. You're going to use nova-compute to manage compute hosts
> in a cloud - and in almost all circumstances the only thing that's going
> to be running on your compute hosts is going to be nova-compute.
Actually we know from user / operator stories at summits that many
groups run *much more* than nova-compute on their compute hosts.
Specifically the all-in-one scale-as-a-unit architecture is quite
popular. That has all the components of
cinder/swift/keystone/nova/neutron all running on all machines, and
the cluster is scaled by adding just another identical machine.
I suspect its actually really quite common outside of a) paranoid (but
not necessarily wrong :)) and b) ultra-scale environments.
...
I'm with Daniel -
>> (4) I think that ultimately we need to ditch rootwrap and provide a proper
>> privilege separated, formal RPC mechanism for each project.
>>
>> eg instead of having a rootwrap command, or rootwrap server attempting
>> to validate safety of
>>
>> qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2
>>
>> we should have a nova-compute-worker daemon running as root, that accepts
>> an RPC command from nova-compute running unprivileged. eg
>>
>> CreateImage("instane0001", "qcow2", "disk.qcow")
>>
>> This immediately makes it trivial to validate that we're not trying to
>> trick qemu-img into overwriting some key system file.
>>
>> This is certainly alot more work than trying to patchup rootwrap, but
>> it would provide a level of security that rootwrap can never achieve IMHO.
I think a local root daemon is a solid idea, there's lots and lots of
prior art on this, from the bazillion dbus daemons we haveadays to the
Android isolation between apps.
So count me in too on design and speccing for this. I realise that for
some cases rootwrap was absolutely a performance and stability issue,
but daemon mode should address that - so I think this is a longer term
project, giving us a significant step up in security.
-Rob
--
Robert Collins
Distinguished Technologist
HP Converged Cloud
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
Excerpts from Daniel P. Berrange's message of 2015-02-04 03:57:53 -0800:
> On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
> > The first one is performance -- each call would spawn a Python
> > interpreter which would then call the system command. This was fine when
> > there were just a few calls here and there, not so much when it's called
> > a hundred times in a row. During the Juno cycle, a daemon mode was added
> > to solve this issue. It is significantly faster than running sudo
> > directly (the often-suggested alternative). Projects still have to start
> > adopting it though. Neutron and Cinder have started work to do that in Kilo.
> >
> > The second problem is the quality of the filter definitions. Rootwrap is
> > a framework to enable isolation. It's only as good as the filters each
> > project defines. Most of them rely on CommandFilters that do not check
> > any argument, instead of using more powerful filters (which are arguably
> > more painful to maintain). Developers routinely add filter definitions
> > that basically remove any isolation that might have been there, like
> > allowing blank dd, tee, chown or chmod.
>
> I think this is really the key point which shows rootwrap as a concept
> is broken by design IMHO. Root wrap is essentially trying to provide an
> API for invoking privileged operations, but instead of actually designing
> an explicit API for the operations, we done by implicit one based on
> command args. From a security POV I think this approach is doomed to
> failure, but command arg strings are fr to expressive a concept
> to deal with.
>
> > What solutions do we have ?
> >
> > (1) we could get our act together and audit and fix those filter
> > definitions. Remove superfluous usage of root rights, make use of
> > advanced filters for where we actually need them. We have been preaching
> > for that at many many design summits. This is a lot of work though...
> > There were such efforts in the past, but they were never completed for
> > some types of nodes. Worse, the bad filter definitions kept coming back,
> > since developers take shortcuts, reviewers may not have sufficient
> > security awareness to detect crappy filter definitions, and I don't
> > think we can design a gate test that would have such awareness.
> >
> > (2) bite the bullet and accept that some types of nodes actually need
> > root rights for so many different things, they should just run as root
> > anyway. I know a few distributions which won't be very pleased by such a
> > prospect, but that would be a more honest approach (rather than claiming
> > we provide efficient isolation when we really don't). An added benefit
> > is that we could replace a number of shell calls by Python code, which
> > would simplify the code and increase performance.
> >
> > (3) intermediary solution where we would run as the nova user but run
> > sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
> > That would leave it up to distros to choose between a blanket sudoer or
> > maintain their own filtering rules. I think it's a bit hypocritical
> > though (pretend the distros could filter if they wanted it, when we
> > dropped the towel on doing that ourselves). I'm also not convinced it's
> > more secure than solution 2, and it prevents from reducing the number of
> > shell-outs, which I think is a worthy idea.
> >
> > In all cases I would not drop the baby with the bath water, and keep
> > rootwrap for all the cases where root rights are needed on a very
> > specific set of commands (like neutron, or nova's api-metadata). The
> > daemon mode should address the performance issue for the projects making
> > a lot of calls.
>
>
> (4) I think that ultimately we need to ditch rootwrap and provide a proper
> privilege separated, formal RPC mechanism for each project.
>
> eg instead of having a rootwrap command, or rootwrap server attempting
> to validate safety of
>
> qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2
>
> we should have a nova-compute-worker daemon running as root, that accepts
> an RPC command from nova-compute running unprivileged. eg
>
> CreateImage("instane0001", "qcow2", "disk.qcow")
>
> This immediately makes it trivial to validate that we're not trying to
> trick qemu-img into overwriting some key system file.
>
> This is certainly alot more work than trying to patchup rootwrap, but
> it would provide a level of security that rootwrap can never achieve IMHO.
>
+1, I think you're right on Daniel. Count me in for future discussions
and work on this.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
Excerpts from Tristan Cacqueray's message of 2015-02-04 09:02:19 -0800:
> On 02/04/2015 06:57 AM, Daniel P. Berrange wrote:
> > On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
> >> What solutions do we have ?
> >>
> >> (1) we could get our act together and audit and fix those filter
> >> definitions. Remove superfluous usage of root rights, make use of
> >> advanced filters for where we actually need them. We have been preaching
> >> for that at many many design summits. This is a lot of work though...
> >> There were such efforts in the past, but they were never completed for
> >> some types of nodes. Worse, the bad filter definitions kept coming back,
> >> since developers take shortcuts, reviewers may not have sufficient
> >> security awareness to detect crappy filter definitions, and I don't
> >> think we can design a gate test that would have such awareness.
> >>
> >> (2) bite the bullet and accept that some types of nodes actually need
> >> root rights for so many different things, they should just run as root
> >> anyway. I know a few distributions which won't be very pleased by such a
> >> prospect, but that would be a more honest approach (rather than claiming
> >> we provide efficient isolation when we really don't). An added benefit
> >> is that we could replace a number of shell calls by Python code, which
> >> would simplify the code and increase performance.
> >>
> >> (3) intermediary solution where we would run as the nova user but run
> >> sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
> >> That would leave it up to distros to choose between a blanket sudoer or
> >> maintain their own filtering rules. I think it's a bit hypocritical
> >> though (pretend the distros could filter if they wanted it, when we
> >> dropped the towel on doing that ourselves). I'm also not convinced it's
> >> more secure than solution 2, and it prevents from reducing the number of
> >> shell-outs, which I think is a worthy idea.
> >>
> >> In all cases I would not drop the baby with the bath water, and keep
> >> rootwrap for all the cases where root rights are needed on a very
> >> specific set of commands (like neutron, or nova's api-metadata). The
> >> daemon mode should address the performance issue for the projects making
> >> a lot of calls.
> >
> >
> > (4) I think that ultimately we need to ditch rootwrap and provide a proper
> > privilege separated, formal RPC mechanism for each project.
> >
> > eg instead of having a rootwrap command, or rootwrap server attempting
> > to validate safety of
> >
> > qemu-img create -f qcow2
> > /var/lib/nova/instances/instance1/disk.qcow2
> >
> > we should have a nova-compute-worker daemon running as root, that accepts
> > an RPC command from nova-compute running unprivileged. eg
> >
> > CreateImage("instane0001", "qcow2", "disk.qcow")
> >
> > This immediately makes it trivial to validate that we're not trying to
> > trick qemu-img into overwriting some key system file.
> >
> > This is certainly alot more work than trying to patchup rootwrap, but
> > it would provide a level of security that rootwrap can never achieve IMHO.
> >
>
> This 4th idea sounds interesting, though we are assuming this new service
> running as root would be exempt of bug, especially if it uses the same
> libraries as non-root services... For example a major bug in python would
> give attacker direct root access while the rootwrap solution would in
> theory keep the intruder at the sudo level...
>
I don't believe that anyone assumes the new service would be without
bugs. But just like the OpenSSH team saw years ago, privilege separation
means that you can absolutely know what is running as root, and what is
not. So when you decide to commit your resources to code audits, you
_start_ with the things that run with elevated privileges.
>
> For completeness, I'd like to propose a more long-term solution:
>
> (5) Get ride of root! Seriously, OpenStack could support security mechanism
> like SELinux or AppArmor in order to properly isolate service and let them
> run what they need to run.
>
> For what it worth, the underlying issue here is having a single almighty super
> user: root and thus we should, at least, consider solution that remove the
> need of such powers (e.g. kernel module loading, ptrace or raw socket).
>
We don't need a security module to drop all of those capabilities
entirely and run as a hobbled root user. By my measure, this process for
nova-compute would only need CAP_NET_ADMIN, CAP_SYS_ADMIN and CAP_KILL.
These capabilities can be audited per-agent and even verified as needed
simply by running integration tests without each one to see what breaks.
> Beside, as long as sensitive process are not contained at the system level,
> the attack surface for a non-root user is still very wide (e.g. system calls,
> setuid binaries, ipc, ...)
>
>
> While this might sounds impossible to implement upstream because it's too
> vend
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On Wed, Feb 04, 2015 at 09:10:06AM -0800, James E. Blair wrote: > Thierry Carrez writes: > > > You make a good point when you mention "traditional distro" here. I > > would argue that containers are slightly changing the rules of the > > don't-run-as-root game. > > > > Solution (2) aligns pretty well with container-powered OpenStack > > deployments -- running compute nodes as root in a container (and > > embracing abovementioned simplicity/performance gains) sounds like a > > pretty strong combo. > > This sounds at least a little like a suggestion that containers are a > substitute for the security provided by running non-root. The security > landscape around containers is complex, and while there are a lot of > benefits, I believe the general consensus is that uid 0 processes should > not be seen as fully isolated. > > From https://docs.docker.com/articles/security/ : > > Docker containers are, by default, quite secure; especially if you > take care of running your processes inside the containers as > non-privileged users (i.e., non-root). > > Which is not to say that using containers is not a good idea, but > rather, if one does, one should avoid running as root (perhaps with > capabilities), and use selinux (or similar). Yep, I've seen attempts by some folks to run nova-compute and libvirtd and QEMU inside a docker container. Because of the inherantly privileged nature of what Nova/libvirt/qemu need to do, you end up having to share all the host namespaces with the docker container, except for the filesystem namespace and even that you need to bind mount a bunch of stuff over. As a result the container isn't really offerring any security benefit over running the things outside the container. IOW the use of containers to confine nova is only solving a managability problem rather than a security problem. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On Wed, Feb 04, 2015 at 06:05:16PM +0100, Philipp Marek wrote:
> > > > (4) I think that ultimately we need to ditch rootwrap and provide a
> > > > proper
> > > > privilege separated, formal RPC mechanism for each project.
> > > ...
> > > > we should have a nova-compute-worker daemon running as root, that
> > > > accepts
> > > > an RPC command from nova-compute running unprivileged. eg
> > > >
> > > > CreateImage("instane0001", "qcow2", "disk.qcow")
> > > ...
> > > > This is certainly alot more work than trying to patchup rootwrap, but
> > > > it would provide a level of security that rootwrap can never achieve
> > > > IMHO.
> > > A lot of work, and if input sanitation didn't work in one piece of code,
> > > why should it here?
> > >
> > > I think this only leads to _more_ work, without any real benefit.
> > > If we can't get the filters right the first round, we won't make it here
> > > either.
> >
> > The difference is that the API I illustrate here has *semantic* context
> > about the operation. In the API example the caller is not permitted to
> > provide a directory path - only the name of the instance and the name
> > of the disk image. The privileged nova-compute-worker program can thus
> > enforce exactly what directory the image is created in, and ensure it
> > doesn't clash with a disk image from another VM. This kind of validation
> > is impractical when you are just given a 'qemu-img' command line args
> > with a full directory path, so there is no semantic conext for the
> > privileged
> > rootwrap to know whether it is reasonable to create the disk image in that
> > particular location.
> Sorry about being unclear.
>
> Yes, there's some semantic meaning at that level. But this level already
> exists at the current rootwrap caller site, too - and if that one can be
> tricked to do something against "image.img & rm -rf /", then the additional
> layer can be tricked, too.
No, that is really not correct. If you are passing full command strings to
rootwrap then the caller can trick rootwrap into running commands with those
shell metacharacter exploits. If you have a formal API like the one I describe
and correctly implement it, there would be no shell involved at all. ie the
nova-compute-worker program would directly invoke the system call
execve("/usr/bin/qemu-img", "create", "image.img & rm -rf /")
and this would at worse create a file called 'image.img & rm -rf /' and
*not* invoke the "rm" command as you get when you use shell.
This is really just another example of why rootwrap/sudo as a concept is a
bad idea. The shell should never be involved in executing any external
commands that Nova/Neutron/etc need to run, because it is impractical to
correctly validate shell commands anywhere in the stack. The only safe
thing todo is to take shell out of the picture entirely.
Regards,
Daniel
--
|: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
> > > (4) I think that ultimately we need to ditch rootwrap and provide a proper
> > > privilege separated, formal RPC mechanism for each project.
> > ...
> > > we should have a nova-compute-worker daemon running as root, that accepts
> > > an RPC command from nova-compute running unprivileged. eg
> > >
> > > CreateImage("instane0001", "qcow2", "disk.qcow")
> > ...
> > > This is certainly alot more work than trying to patchup rootwrap, but
> > > it would provide a level of security that rootwrap can never achieve IMHO.
> > A lot of work, and if input sanitation didn't work in one piece of code,
> > why should it here?
> >
> > I think this only leads to _more_ work, without any real benefit.
> > If we can't get the filters right the first round, we won't make it here
> > either.
>
> The difference is that the API I illustrate here has *semantic* context
> about the operation. In the API example the caller is not permitted to
> provide a directory path - only the name of the instance and the name
> of the disk image. The privileged nova-compute-worker program can thus
> enforce exactly what directory the image is created in, and ensure it
> doesn't clash with a disk image from another VM. This kind of validation
> is impractical when you are just given a 'qemu-img' command line args
> with a full directory path, so there is no semantic conext for the privileged
> rootwrap to know whether it is reasonable to create the disk image in that
> particular location.
Sorry about being unclear.
Yes, there's some semantic meaning at that level. But this level already
exists at the current rootwrap caller site, too - and if that one can be
tricked to do something against "image.img & rm -rf /", then the additional
layer can be tricked, too.
I'm trying to get at the point
everything that can be forgot to check at the rootwrap call site now,
can be forgotten in the additional API too.
So let's get the current call sites tight, and we're done. (Ha!)
--
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com :
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
Thierry Carrez writes: > You make a good point when you mention "traditional distro" here. I > would argue that containers are slightly changing the rules of the > don't-run-as-root game. > > Solution (2) aligns pretty well with container-powered OpenStack > deployments -- running compute nodes as root in a container (and > embracing abovementioned simplicity/performance gains) sounds like a > pretty strong combo. This sounds at least a little like a suggestion that containers are a substitute for the security provided by running non-root. The security landscape around containers is complex, and while there are a lot of benefits, I believe the general consensus is that uid 0 processes should not be seen as fully isolated. >From https://docs.docker.com/articles/security/ : Docker containers are, by default, quite secure; especially if you take care of running your processes inside the containers as non-privileged users (i.e., non-root). Which is not to say that using containers is not a good idea, but rather, if one does, one should avoid running as root (perhaps with capabilities), and use selinux (or similar). -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 02/04/2015 06:57 AM, Daniel P. Berrange wrote:
> On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
>> What solutions do we have ?
>>
>> (1) we could get our act together and audit and fix those filter
>> definitions. Remove superfluous usage of root rights, make use of
>> advanced filters for where we actually need them. We have been preaching
>> for that at many many design summits. This is a lot of work though...
>> There were such efforts in the past, but they were never completed for
>> some types of nodes. Worse, the bad filter definitions kept coming back,
>> since developers take shortcuts, reviewers may not have sufficient
>> security awareness to detect crappy filter definitions, and I don't
>> think we can design a gate test that would have such awareness.
>>
>> (2) bite the bullet and accept that some types of nodes actually need
>> root rights for so many different things, they should just run as root
>> anyway. I know a few distributions which won't be very pleased by such a
>> prospect, but that would be a more honest approach (rather than claiming
>> we provide efficient isolation when we really don't). An added benefit
>> is that we could replace a number of shell calls by Python code, which
>> would simplify the code and increase performance.
>>
>> (3) intermediary solution where we would run as the nova user but run
>> sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
>> That would leave it up to distros to choose between a blanket sudoer or
>> maintain their own filtering rules. I think it's a bit hypocritical
>> though (pretend the distros could filter if they wanted it, when we
>> dropped the towel on doing that ourselves). I'm also not convinced it's
>> more secure than solution 2, and it prevents from reducing the number of
>> shell-outs, which I think is a worthy idea.
>>
>> In all cases I would not drop the baby with the bath water, and keep
>> rootwrap for all the cases where root rights are needed on a very
>> specific set of commands (like neutron, or nova's api-metadata). The
>> daemon mode should address the performance issue for the projects making
>> a lot of calls.
>
>
> (4) I think that ultimately we need to ditch rootwrap and provide a proper
> privilege separated, formal RPC mechanism for each project.
>
> eg instead of having a rootwrap command, or rootwrap server attempting
> to validate safety of
>
> qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2
>
> we should have a nova-compute-worker daemon running as root, that accepts
> an RPC command from nova-compute running unprivileged. eg
>
> CreateImage("instane0001", "qcow2", "disk.qcow")
>
> This immediately makes it trivial to validate that we're not trying to
> trick qemu-img into overwriting some key system file.
>
> This is certainly alot more work than trying to patchup rootwrap, but
> it would provide a level of security that rootwrap can never achieve IMHO.
>
This 4th idea sounds interesting, though we are assuming this new service
running as root would be exempt of bug, especially if it uses the same
libraries as non-root services... For example a major bug in python would
give attacker direct root access while the rootwrap solution would in
theory keep the intruder at the sudo level...
For completeness, I'd like to propose a more long-term solution:
(5) Get ride of root! Seriously, OpenStack could support security mechanism
like SELinux or AppArmor in order to properly isolate service and let them
run what they need to run.
For what it worth, the underlying issue here is having a single almighty super
user: root and thus we should, at least, consider solution that remove the
need of such powers (e.g. kernel module loading, ptrace or raw socket).
Beside, as long as sensitive process are not contained at the system level,
the attack surface for a non-root user is still very wide (e.g. system calls,
setuid binaries, ipc, ...)
While this might sounds impossible to implement upstream because it's too
vendor specific or just because of other technicals difficulties,
I guess it still deserves a mention in this thread.
Best regards,
Tristan
signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On Wed, Feb 04, 2015 at 05:52:12PM +0100, Philipp Marek wrote:
> Here are my 2¢.
>
> > > >> (1) we could get our act together and audit and fix those filter
> > > >> definitions. Remove superfluous usage of root rights, make use of
> > > >> advanced filters for where we actually need them. We have been
> > > >> preaching
> > > >> for that at many many design summits. This is a lot of work though...
> > > >> There were such efforts in the past, but they were never completed for
> > > >> some types of nodes. Worse, the bad filter definitions kept coming
> > > >> back,
> > > >> since developers take shortcuts, reviewers may not have sufficient
> > > >> security awareness to detect crappy filter definitions, and I don't
> > > >> think we can design a gate test that would have such awareness.
> Sounds like a lot of work... ongoing, too.
>
>
> > > >> (2) bite the bullet and accept that some types of nodes actually need
> > > >> root rights for so many different things, they should just run as root
> > > >> anyway. I know a few distributions which won't be very pleased by such
> > > >> a
> > > >> prospect, but that would be a more honest approach (rather than
> > > >> claiming
> > > >> we provide efficient isolation when we really don't). An added benefit
> > > >> is that we could replace a number of shell calls by Python code, which
> > > >> would simplify the code and increase performance.
> Practical, but unsafe.
>
> I'd very much like to have some best-effort filter against bugs in my
> programming - even more so during development.
>
>
> > (4) I think that ultimately we need to ditch rootwrap and provide a proper
> > privilege separated, formal RPC mechanism for each project.
> ...
> > we should have a nova-compute-worker daemon running as root, that accepts
> > an RPC command from nova-compute running unprivileged. eg
> >
> > CreateImage("instane0001", "qcow2", "disk.qcow")
> ...
> > This is certainly alot more work than trying to patchup rootwrap, but
> > it would provide a level of security that rootwrap can never achieve IMHO.
> A lot of work, and if input sanitation didn't work in one piece of code,
> why should it here?
>
> I think this only leads to _more_ work, without any real benefit.
> If we can't get the filters right the first round, we won't make it here
> either.
The difference is that the API I illustrate here has *semantic* context
about the operation. In the API example the caller is not permitted to
provide a directory path - only the name of the instance and the name
of the disk image. The privileged nova-compute-worker program can thus
enforce exactly what directory the image is created in, and ensure it
doesn't clash with a disk image from another VM. This kind of validation
is impractical when you are just given a 'qemu-img' command line args
with a full directory path, so there is no semantic conext for the privileged
rootwrap to know whether it is reasonable to create the disk image in that
particular location.
Regards,
Daniel
--
|: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
Here are my 2¢.
> > >> (1) we could get our act together and audit and fix those filter
> > >> definitions. Remove superfluous usage of root rights, make use of
> > >> advanced filters for where we actually need them. We have been preaching
> > >> for that at many many design summits. This is a lot of work though...
> > >> There were such efforts in the past, but they were never completed for
> > >> some types of nodes. Worse, the bad filter definitions kept coming back,
> > >> since developers take shortcuts, reviewers may not have sufficient
> > >> security awareness to detect crappy filter definitions, and I don't
> > >> think we can design a gate test that would have such awareness.
Sounds like a lot of work... ongoing, too.
> > >> (2) bite the bullet and accept that some types of nodes actually need
> > >> root rights for so many different things, they should just run as root
> > >> anyway. I know a few distributions which won't be very pleased by such a
> > >> prospect, but that would be a more honest approach (rather than claiming
> > >> we provide efficient isolation when we really don't). An added benefit
> > >> is that we could replace a number of shell calls by Python code, which
> > >> would simplify the code and increase performance.
Practical, but unsafe.
I'd very much like to have some best-effort filter against bugs in my
programming - even more so during development.
> (4) I think that ultimately we need to ditch rootwrap and provide a proper
> privilege separated, formal RPC mechanism for each project.
...
> we should have a nova-compute-worker daemon running as root, that accepts
> an RPC command from nova-compute running unprivileged. eg
>
> CreateImage("instane0001", "qcow2", "disk.qcow")
...
> This is certainly alot more work than trying to patchup rootwrap, but
> it would provide a level of security that rootwrap can never achieve IMHO.
A lot of work, and if input sanitation didn't work in one piece of code,
why should it here?
I think this only leads to _more_ work, without any real benefit.
If we can't get the filters right the first round, we won't make it here
either.
Regarding the idea of using containers ... take Cinder as an example.
If the cinder container can access *all* the VM data, why should someone
bother to get *out* of the container? Everything that she wants is already
here...
I'm not sure what the containers would buy us, but perhaps I just don't
understand something here.
So, IMO, solution 1 (one) would be the way to go ... it gets to security
asymptotically (and might never reach it), but at least it provides a bit
of help.
And if the rootwrap filter specification would be linked to in the rootwrap
config files, it would help newcomers to see the available syntax, instead
of simply copying a bad example ;P
--
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com :
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 2015-02-04 18:38:16 +0200 (+0200), Duncan Thomas wrote: > If I'm reading that correctly, it does not help with the filtering issues at > all, since it needs exactly the same kind of filter. Daniel explained the > concept far better than I. I didn't mean to imply that it does, merely that it fits your rather terse description of a "daemon that runs as root, accepting commands over a unix domain socket or similar." -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On Wed, Feb 04, 2015 at 06:38:16PM +0200, Duncan Thomas wrote: > If I'm reading that correctly, it does not help with the filtering issues > at all, since it needs exactly the same kind of filter. Daniel explained > the concept far better than I. Yep, the only thing rootwrap daemon mode does is to remove the overhead of spawning the rootwrap command. It does nothing to improve actual security - it is still a chocolate teapot from that POV. > On 4 February 2015 at 18:33, Jeremy Stanley wrote: > > > On 2015-02-04 13:40:29 +0200 (+0200), Duncan Thomas wrote: > > > 4) Write a small daemon that runs as root, accepting commands over > > > a unix domain socket or similar. Easier to audit, less code > > > running as root. > > > > > > http://git.openstack.org/cgit/openstack/oslo.rootwrap/tree/oslo_rootwrap/daemon.py > > Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 2015-02-04 11:58:03 +0100 (+0100), Thierry Carrez wrote: [...] > The second problem is the quality of the filter definitions. Rootwrap is > a framework to enable isolation. It's only as good as the filters each > project defines. Most of them rely on CommandFilters that do not check > any argument, instead of using more powerful filters (which are arguably > more painful to maintain). Developers routinely add filter definitions > that basically remove any isolation that might have been there, like > allowing blank dd, tee, chown or chmod. [...] This part is my biggest concern at the moment, from a vulnerability management standpoint. I'm worried that it's an attractive nuisance resulting in a false sense of security in its current state because we're not calling this shortcoming out explicitly in documentation (as far as I'm aware), and so we're opening our operators/users up to unexpected risks and opening ourselves up to the possibility of a slew of vulnerability reports because this mechanism doesn't provide the level of protection it would seem to imply. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
If I'm reading that correctly, it does not help with the filtering issues at all, since it needs exactly the same kind of filter. Daniel explained the concept far better than I. On 4 February 2015 at 18:33, Jeremy Stanley wrote: > On 2015-02-04 13:40:29 +0200 (+0200), Duncan Thomas wrote: > > 4) Write a small daemon that runs as root, accepting commands over > > a unix domain socket or similar. Easier to audit, less code > > running as root. > > > http://git.openstack.org/cgit/openstack/oslo.rootwrap/tree/oslo_rootwrap/daemon.py > > -- > Jeremy Stanley > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Duncan Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 2015-02-04 13:40:29 +0200 (+0200), Duncan Thomas wrote: > 4) Write a small daemon that runs as root, accepting commands over > a unix domain socket or similar. Easier to audit, less code > running as root. http://git.openstack.org/cgit/openstack/oslo.rootwrap/tree/oslo_rootwrap/daemon.py -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
Monty Taylor wrote: >> On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote: >>> (2) bite the bullet and accept that some types of nodes actually need >>> root rights for so many different things, they should just run as root >>> anyway. I know a few distributions which won't be very pleased by such a >>> prospect, but that would be a more honest approach (rather than claiming >>> we provide efficient isolation when we really don't). An added benefit >>> is that we could replace a number of shell calls by Python code, which >>> would simplify the code and increase performance. > > I'm actually the biggest fan of this solution (even more than Daniel's > suggestion below) because it's the thing that is closest to reality. > > Security isn't a useful concept in a vacuum - it's something we do to > prevent access to or damage resources that we don't want accessed by the > wrong people. > > On compute nodes, the main valuable thing are the VMs themselves- and > I'd expect the most interested target of an attack to be interested in > manipulating, stealing data from or deleting the VMs. > > No amount of rootwrap or privileges are going to prevent nova-compute > from performing unwanted actions on the VMs in its control - for the > reason that it's job in life is to manipulate those things. > > Is it a security hole in the traditional distro sense - that we want to > be able to install all of these things with apt-get or yum on a single > server and have the actions of one service not affect the state of > another? Sure. Is it in the real world? No. You're not going to use this > to manage VMs on a laptop - you're going to use virtualbox or > virt-manager. You're going to use nova-compute to manage compute hosts > in a cloud - and in almost all circumstances the only thing that's going > to be running on your compute hosts is going to be nova-compute. You make a good point when you mention "traditional distro" here. I would argue that containers are slightly changing the rules of the don't-run-as-root game. Solution (2) aligns pretty well with container-powered OpenStack deployments -- running compute nodes as root in a container (and embracing abovementioned simplicity/performance gains) sounds like a pretty strong combo. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
I suppose that the security argument against running the whole of nova-compute as root is that a remote exploit in the service is much better constrained when the thing isn't running as root - e.g. some input validation fails and allows arbitrary shell in some (currently none-root) command via an existing API call. This means that (2) is the worst possible option - calling out to sudo at least limits the attach surface to those code paths that do those specific exec calls. I think Daniel's suggestion makes a great deal more sense from a security point of view, but (3) is a significantly better architecture than (2) even if it still has some holes. On 4 February 2015 at 16:33, Monty Taylor wrote: > On 02/04/2015 06:57 AM, Daniel P. Berrange wrote: > > On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote: > >> The first one is performance -- each call would spawn a Python > >> interpreter which would then call the system command. This was fine when > >> there were just a few calls here and there, not so much when it's called > >> a hundred times in a row. During the Juno cycle, a daemon mode was added > >> to solve this issue. It is significantly faster than running sudo > >> directly (the often-suggested alternative). Projects still have to start > >> adopting it though. Neutron and Cinder have started work to do that in > Kilo. > >> > >> The second problem is the quality of the filter definitions. Rootwrap is > >> a framework to enable isolation. It's only as good as the filters each > >> project defines. Most of them rely on CommandFilters that do not check > >> any argument, instead of using more powerful filters (which are arguably > >> more painful to maintain). Developers routinely add filter definitions > >> that basically remove any isolation that might have been there, like > >> allowing blank dd, tee, chown or chmod. > > > > I think this is really the key point which shows rootwrap as a concept > > is broken by design IMHO. Root wrap is essentially trying to provide an > > API for invoking privileged operations, but instead of actually designing > > an explicit API for the operations, we done by implicit one based on > > command args. From a security POV I think this approach is doomed to > > failure, but command arg strings are fr to expressive a concept > > to deal with. > > > >> What solutions do we have ? > >> > >> (1) we could get our act together and audit and fix those filter > >> definitions. Remove superfluous usage of root rights, make use of > >> advanced filters for where we actually need them. We have been preaching > >> for that at many many design summits. This is a lot of work though... > >> There were such efforts in the past, but they were never completed for > >> some types of nodes. Worse, the bad filter definitions kept coming back, > >> since developers take shortcuts, reviewers may not have sufficient > >> security awareness to detect crappy filter definitions, and I don't > >> think we can design a gate test that would have such awareness. > >> > >> (2) bite the bullet and accept that some types of nodes actually need > >> root rights for so many different things, they should just run as root > >> anyway. I know a few distributions which won't be very pleased by such a > >> prospect, but that would be a more honest approach (rather than claiming > >> we provide efficient isolation when we really don't). An added benefit > >> is that we could replace a number of shell calls by Python code, which > >> would simplify the code and increase performance. > > I'm actually the biggest fan of this solution (even more than Daniel's > suggestion below) because it's the thing that is closest to reality. > > Security isn't a useful concept in a vacuum - it's something we do to > prevent access to or damage resources that we don't want accessed by the > wrong people. > > On compute nodes, the main valuable thing are the VMs themselves- and > I'd expect the most interested target of an attack to be interested in > manipulating, stealing data from or deleting the VMs. > > No amount of rootwrap or privileges are going to prevent nova-compute > from performing unwanted actions on the VMs in its control - for the > reason that it's job in life is to manipulate those things. > > Is it a security hole in the traditional distro sense - that we want to > be able to install all of these things with apt-get or yum on a single > server and have the actions of one service not affect the state of > another? Sure. Is it in the real world? No. You're not going to use this > to manage VMs on a laptop - you're going to use virtualbox or > virt-manager. You're going to use nova-compute to manage compute hosts > in a cloud - and in almost all circumstances the only thing that's going > to be running on your compute hosts is going to be nova-compute. > > > >> (3) intermediary solution where we would run as the nova user but run > >> sudo COMMAND directly (instead of sudo nova-rootwrap CONFI
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On 02/04/2015 06:57 AM, Daniel P. Berrange wrote: > On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote: >> The first one is performance -- each call would spawn a Python >> interpreter which would then call the system command. This was fine when >> there were just a few calls here and there, not so much when it's called >> a hundred times in a row. During the Juno cycle, a daemon mode was added >> to solve this issue. It is significantly faster than running sudo >> directly (the often-suggested alternative). Projects still have to start >> adopting it though. Neutron and Cinder have started work to do that in Kilo. >> >> The second problem is the quality of the filter definitions. Rootwrap is >> a framework to enable isolation. It's only as good as the filters each >> project defines. Most of them rely on CommandFilters that do not check >> any argument, instead of using more powerful filters (which are arguably >> more painful to maintain). Developers routinely add filter definitions >> that basically remove any isolation that might have been there, like >> allowing blank dd, tee, chown or chmod. > > I think this is really the key point which shows rootwrap as a concept > is broken by design IMHO. Root wrap is essentially trying to provide an > API for invoking privileged operations, but instead of actually designing > an explicit API for the operations, we done by implicit one based on > command args. From a security POV I think this approach is doomed to > failure, but command arg strings are fr to expressive a concept > to deal with. > >> What solutions do we have ? >> >> (1) we could get our act together and audit and fix those filter >> definitions. Remove superfluous usage of root rights, make use of >> advanced filters for where we actually need them. We have been preaching >> for that at many many design summits. This is a lot of work though... >> There were such efforts in the past, but they were never completed for >> some types of nodes. Worse, the bad filter definitions kept coming back, >> since developers take shortcuts, reviewers may not have sufficient >> security awareness to detect crappy filter definitions, and I don't >> think we can design a gate test that would have such awareness. >> >> (2) bite the bullet and accept that some types of nodes actually need >> root rights for so many different things, they should just run as root >> anyway. I know a few distributions which won't be very pleased by such a >> prospect, but that would be a more honest approach (rather than claiming >> we provide efficient isolation when we really don't). An added benefit >> is that we could replace a number of shell calls by Python code, which >> would simplify the code and increase performance. I'm actually the biggest fan of this solution (even more than Daniel's suggestion below) because it's the thing that is closest to reality. Security isn't a useful concept in a vacuum - it's something we do to prevent access to or damage resources that we don't want accessed by the wrong people. On compute nodes, the main valuable thing are the VMs themselves- and I'd expect the most interested target of an attack to be interested in manipulating, stealing data from or deleting the VMs. No amount of rootwrap or privileges are going to prevent nova-compute from performing unwanted actions on the VMs in its control - for the reason that it's job in life is to manipulate those things. Is it a security hole in the traditional distro sense - that we want to be able to install all of these things with apt-get or yum on a single server and have the actions of one service not affect the state of another? Sure. Is it in the real world? No. You're not going to use this to manage VMs on a laptop - you're going to use virtualbox or virt-manager. You're going to use nova-compute to manage compute hosts in a cloud - and in almost all circumstances the only thing that's going to be running on your compute hosts is going to be nova-compute. >> (3) intermediary solution where we would run as the nova user but run >> sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND). >> That would leave it up to distros to choose between a blanket sudoer or >> maintain their own filtering rules. I think it's a bit hypocritical >> though (pretend the distros could filter if they wanted it, when we >> dropped the towel on doing that ourselves). I'm also not convinced it's >> more secure than solution 2, and it prevents from reducing the number of >> shell-outs, which I think is a worthy idea. >> >> In all cases I would not drop the baby with the bath water, and keep >> rootwrap for all the cases where root rights are needed on a very >> specific set of commands (like neutron, or nova's api-metadata). The >> daemon mode should address the performance issue for the projects making >> a lot of calls. > > > (4) I think that ultimately we need to ditch rootwrap and provide a proper > privilege separated, form
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
> The first one is performance -- each call would spawn a Python
> interpreter which would then call the system command. This was fine when
> there were just a few calls here and there, not so much when it's called
> a hundred times in a row. During the Juno cycle, a daemon mode was added
> to solve this issue. It is significantly faster than running sudo
> directly (the often-suggested alternative). Projects still have to start
> adopting it though. Neutron and Cinder have started work to do that in Kilo.
>
> The second problem is the quality of the filter definitions. Rootwrap is
> a framework to enable isolation. It's only as good as the filters each
> project defines. Most of them rely on CommandFilters that do not check
> any argument, instead of using more powerful filters (which are arguably
> more painful to maintain). Developers routinely add filter definitions
> that basically remove any isolation that might have been there, like
> allowing blank dd, tee, chown or chmod.
I think this is really the key point which shows rootwrap as a concept
is broken by design IMHO. Root wrap is essentially trying to provide an
API for invoking privileged operations, but instead of actually designing
an explicit API for the operations, we done by implicit one based on
command args. From a security POV I think this approach is doomed to
failure, but command arg strings are fr to expressive a concept
to deal with.
> What solutions do we have ?
>
> (1) we could get our act together and audit and fix those filter
> definitions. Remove superfluous usage of root rights, make use of
> advanced filters for where we actually need them. We have been preaching
> for that at many many design summits. This is a lot of work though...
> There were such efforts in the past, but they were never completed for
> some types of nodes. Worse, the bad filter definitions kept coming back,
> since developers take shortcuts, reviewers may not have sufficient
> security awareness to detect crappy filter definitions, and I don't
> think we can design a gate test that would have such awareness.
>
> (2) bite the bullet and accept that some types of nodes actually need
> root rights for so many different things, they should just run as root
> anyway. I know a few distributions which won't be very pleased by such a
> prospect, but that would be a more honest approach (rather than claiming
> we provide efficient isolation when we really don't). An added benefit
> is that we could replace a number of shell calls by Python code, which
> would simplify the code and increase performance.
>
> (3) intermediary solution where we would run as the nova user but run
> sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
> That would leave it up to distros to choose between a blanket sudoer or
> maintain their own filtering rules. I think it's a bit hypocritical
> though (pretend the distros could filter if they wanted it, when we
> dropped the towel on doing that ourselves). I'm also not convinced it's
> more secure than solution 2, and it prevents from reducing the number of
> shell-outs, which I think is a worthy idea.
>
> In all cases I would not drop the baby with the bath water, and keep
> rootwrap for all the cases where root rights are needed on a very
> specific set of commands (like neutron, or nova's api-metadata). The
> daemon mode should address the performance issue for the projects making
> a lot of calls.
(4) I think that ultimately we need to ditch rootwrap and provide a proper
privilege separated, formal RPC mechanism for each project.
eg instead of having a rootwrap command, or rootwrap server attempting
to validate safety of
qemu-img create -f qcow2 /var/lib/nova/instances/instance1/disk.qcow2
we should have a nova-compute-worker daemon running as root, that accepts
an RPC command from nova-compute running unprivileged. eg
CreateImage("instane0001", "qcow2", "disk.qcow")
This immediately makes it trivial to validate that we're not trying to
trick qemu-img into overwriting some key system file.
This is certainly alot more work than trying to patchup rootwrap, but
it would provide a level of security that rootwrap can never achieve IMHO.
Regards,
Daniel
--
|: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap on root-intensive nodes
4) Write a small daemon that runs as root, accepting commands over a unix domain socket or similar. Easier to audit, less code running as root. On 4 February 2015 at 12:58, Thierry Carrez wrote: > Hi, > > This is the follow-up of the discussion we started yesterday on rootwrap > usage at the cross-project meeting. > > A bit of history to stage this story first. OpenStack nodes sometimes > need to run things with elevated privileges. Nova started out as calling > sudo shell commands to execute those, which basically allowed it to run > anything as root. Distributions didn't really like that, so packagers > started to ship sudoers files that restricted the commands Nova could > run with sudo. It created a maintenance nightmare, where the code and > the packaging were routinely out of sync. Rootwrap was designed so that > we would maintain the allowed commands in the code itself, which > facilitated maintenance. It would also allow more complex rules than > sudoers allowed, like the ability to only kill a certain executable > processes. > > Rootwrap was then adopted by Cinder and Neutron, and moved to oslo to > avoid code duplication. There were still two long-standing problems with > it, though. > > The first one is performance -- each call would spawn a Python > interpreter which would then call the system command. This was fine when > there were just a few calls here and there, not so much when it's called > a hundred times in a row. During the Juno cycle, a daemon mode was added > to solve this issue. It is significantly faster than running sudo > directly (the often-suggested alternative). Projects still have to start > adopting it though. Neutron and Cinder have started work to do that in > Kilo. > > The second problem is the quality of the filter definitions. Rootwrap is > a framework to enable isolation. It's only as good as the filters each > project defines. Most of them rely on CommandFilters that do not check > any argument, instead of using more powerful filters (which are arguably > more painful to maintain). Developers routinely add filter definitions > that basically remove any isolation that might have been there, like > allowing blank dd, tee, chown or chmod. > > Basically, some nodes run such a variety of commands as root that > maintaining proper isolation is a non-trivial amount of work: that's the > case for Nova's compute nodes and Cinder's volume nodes. I would argue > that for all other types of nodes, rootwrap is still an appropriate way > to provide isolation. But for nodes which need to make a very wide > variety of root calls, we pretend to isolate, but the filter definitions > are so full of holes that we actually don't. > > What solutions do we have ? > > (1) we could get our act together and audit and fix those filter > definitions. Remove superfluous usage of root rights, make use of > advanced filters for where we actually need them. We have been preaching > for that at many many design summits. This is a lot of work though... > There were such efforts in the past, but they were never completed for > some types of nodes. Worse, the bad filter definitions kept coming back, > since developers take shortcuts, reviewers may not have sufficient > security awareness to detect crappy filter definitions, and I don't > think we can design a gate test that would have such awareness. > > (2) bite the bullet and accept that some types of nodes actually need > root rights for so many different things, they should just run as root > anyway. I know a few distributions which won't be very pleased by such a > prospect, but that would be a more honest approach (rather than claiming > we provide efficient isolation when we really don't). An added benefit > is that we could replace a number of shell calls by Python code, which > would simplify the code and increase performance. > > (3) intermediary solution where we would run as the nova user but run > sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND). > That would leave it up to distros to choose between a blanket sudoer or > maintain their own filtering rules. I think it's a bit hypocritical > though (pretend the distros could filter if they wanted it, when we > dropped the towel on doing that ourselves). I'm also not convinced it's > more secure than solution 2, and it prevents from reducing the number of > shell-outs, which I think is a worthy idea. > > In all cases I would not drop the baby with the bath water, and keep > rootwrap for all the cases where root rights are needed on a very > specific set of commands (like neutron, or nova's api-metadata). The > daemon mode should address the performance issue for the projects making > a lot of calls. > > -- > Thierry Carrez (ttx) > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.or
