Re: [openstack-dev] [neutron][security][rootwrap] Proposal to replace rootwrap/sudo with privsep helper process (for neutron, but others too)
On Fri Feb 13 2015 at 10:35:49 PM Miguel Ángel Ajo majop...@redhat.com
wrote:
We have an ongoing effort in neutron to move to rootwrap-daemon.
https://review.openstack.org/#/q/status:open+project:openstack/neutron+branch:master+topic:bp/rootwrap-daemon-mode,n,z
Thanks for replying. I should have mentioned rootwrap-daemon in my
original post:
The main difference between this privsep proposal and rootwrap (in
daemon-mode or rootwrap classic) is that rootwrap is oriented around
command lines, and privsep is oriented around python function calls.
Using functions rather than command lines means we can move to using native
libraries more easily (as I've done here with pyroute2), which allows a
more closely integrated, performant and safer interface. It also means we
can describe more customised behaviour in the exposed function API, leading
to fewer cross-process calls and less scope for exposing unintended
behaviour.
To pick one example, starting up a long-running daemon (assuming we wanted
to do this) from the privsep process just works without any need to work
around the privsep mechanism.
To speed up multiple system calls, and be able to spawn daemons inside
namespaces.
Yep, rootwrap-daemon certainly removes the python rootwrap startup time;
privsep *also* removes the subprocess exec startup time for cases where we
have moved to native libraries (again, see the pyroute2 example).
I have to read a bit what are the good bad points of privsep.
The advantage of rootwrap-daemon, is that we don’t need to change all our
networking libraries across neutron,
and we kill the sudo/rootwrap spawn for every call, yet keeping
the rootwrap permission granularity.
The good news is that privsep and rootwrap (in either mode) can coexist
just fine. A staged migration to privsep might involve spawning the
privsep daemon via sudo on the first call to something that needs it. This
approach means we wouldn't require adding the privsep_daemon.start() call
early in any relevant main() - and the only downside is that we retain a
(single) dependency on sudo/sudoers.
- Gus
Miguel Ángel Ajo
On Friday, 13 de February de 2015 at 10:54, Angus Lees wrote:
On Fri Feb 13 2015 at 5:45:36 PM Eric Windisch e...@windisch.us wrote:
ᐧ
from neutron.agent.privileged.commands import ip_lib as priv_ip
def foo():
# Need to create a new veth interface pair - that usually requires
root/NET_ADMIN
priv_ip.CreateLink('veth', 'veth0', peer='veth1')
Because we now have elevated privileges directly (on the privileged daemon
side) without having to shell out through sudo, we can do all sorts of
nicer things like just using netlink directly to configure networking.
This avoids the overhead of executing subcommands, the ugliness (and
danger) of generating command lines and regex parsing output, and make us
less reliant on specific versions of command line tools (since the kernel
API should be very stable).
One of the advantages of spawning a new process is being able to use flags
to clone(2) and to set capabilities. This basically means to create
containers, by some definition. Anything you have in a privileged daemon
or privileged process ideally should reduce its privilege set for any
operation it performs. That might mean it clones itself and executes
Python, or it may execvp an executable, but either way, the new process
would have less-than-full-privilege.
For instance, writing a file might require root access, but does not need
the ability to load kernel modules. Changing network interfaces does not
need access to the filesystem, no more than changes to the filesystem needs
access to the network. The capabilities and namespaces mechanisms resolve
these security conundrums and simplify principle of least privilege.
Agreed wholeheartedly, and I'd appreciate your thoughts on how I'm using
capabilities in this change. The privsep daemon limits itself to a
particular set of capabilities (and drops root). The assumption is that
most OpenStack services commonly need the same small set of capabilities to
perform their duties (neutron - net_admin+sys_admin for example), so it
makes sense to reuse the same privileged process.
If we have a single service that somehow needs to frequently use a broad
swathe of capabilities then we might want to break it up further somehow
between the different internal aspects (multiple privsep helpers?) - but is
there such a case? There's also no problems with mixing privsep for
frequent operations with the existing sudo/rootwrap approach for
rare/awkward cases.
- Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
Re: [openstack-dev] [neutron][security][rootwrap] Proposal to replace rootwrap/sudo with privsep helper process (for neutron, but others too)
On Fri Feb 13 2015 at 5:45:36 PM Eric Windisch e...@windisch.us wrote:
ᐧ
from neutron.agent.privileged.commands import ip_lib as priv_ip
def foo():
# Need to create a new veth interface pair - that usually
requires root/NET_ADMIN
priv_ip.CreateLink('veth', 'veth0', peer='veth1')
Because we now have elevated privileges directly (on the privileged
daemon side) without having to shell out through sudo, we can do all sorts
of nicer things like just using netlink directly to configure networking.
This avoids the overhead of executing subcommands, the ugliness (and
danger) of generating command lines and regex parsing output, and make us
less reliant on specific versions of command line tools (since the kernel
API should be very stable).
One of the advantages of spawning a new process is being able to use flags
to clone(2) and to set capabilities. This basically means to create
containers, by some definition. Anything you have in a privileged daemon
or privileged process ideally should reduce its privilege set for any
operation it performs. That might mean it clones itself and executes
Python, or it may execvp an executable, but either way, the new process
would have less-than-full-privilege.
For instance, writing a file might require root access, but does not need
the ability to load kernel modules. Changing network interfaces does not
need access to the filesystem, no more than changes to the filesystem needs
access to the network. The capabilities and namespaces mechanisms resolve
these security conundrums and simplify principle of least privilege.
Agreed wholeheartedly, and I'd appreciate your thoughts on how I'm using
capabilities in this change. The privsep daemon limits itself to a
particular set of capabilities (and drops root). The assumption is that
most OpenStack services commonly need the same small set of capabilities to
perform their duties (neutron - net_admin+sys_admin for example), so it
makes sense to reuse the same privileged process.
If we have a single service that somehow needs to frequently use a broad
swathe of capabilities then we might want to break it up further somehow
between the different internal aspects (multiple privsep helpers?) - but is
there such a case? There's also no problems with mixing privsep for
frequent operations with the existing sudo/rootwrap approach for
rare/awkward cases.
- Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][security][rootwrap] Proposal to replace rootwrap/sudo with privsep helper process (for neutron, but others too)
We have an ongoing effort in neutron to move to rootwrap-daemon.
https://review.openstack.org/#/q/status:open+project:openstack/neutron+branch:master+topic:bp/rootwrap-daemon-mode,n,z
To speed up multiple system calls, and be able to spawn daemons inside
namespaces.
I have to read a bit what are the good bad points of privsep.
The advantage of rootwrap-daemon, is that we don’t need to change all our
networking libraries across neutron,
and we kill the sudo/rootwrap spawn for every call, yet keeping the rootwrap
permission granularity.
Miguel Ángel Ajo
On Friday, 13 de February de 2015 at 10:54, Angus Lees wrote:
On Fri Feb 13 2015 at 5:45:36 PM Eric Windisch e...@windisch.us
(mailto:e...@windisch.us) wrote:
ᐧ
from neutron.agent.privileged.commands import ip_lib as priv_ip
def foo():
# Need to create a new veth interface pair - that usually
requires root/NET_ADMIN
priv_ip.CreateLink('veth', 'veth0', peer='veth1')
Because we now have elevated privileges directly (on the privileged
daemon side) without having to shell out through sudo, we can do all
sorts of nicer things like just using netlink directly to configure
networking. This avoids the overhead of executing subcommands, the
ugliness (and danger) of generating command lines and regex parsing
output, and make us less reliant on specific versions of command line
tools (since the kernel API should be very stable).
One of the advantages of spawning a new process is being able to use flags
to clone(2) and to set capabilities. This basically means to create
containers, by some definition. Anything you have in a privileged daemon
or privileged process ideally should reduce its privilege set for any
operation it performs. That might mean it clones itself and executes
Python, or it may execvp an executable, but either way, the new process
would have less-than-full-privilege.
For instance, writing a file might require root access, but does not need
the ability to load kernel modules. Changing network interfaces does not
need access to the filesystem, no more than changes to the filesystem needs
access to the network. The capabilities and namespaces mechanisms resolve
these security conundrums and simplify principle of least privilege.
Agreed wholeheartedly, and I'd appreciate your thoughts on how I'm using
capabilities in this change. The privsep daemon limits itself to a
particular set of capabilities (and drops root). The assumption is that most
OpenStack services commonly need the same small set of capabilities to
perform their duties (neutron - net_admin+sys_admin for example), so it
makes sense to reuse the same privileged process.
If we have a single service that somehow needs to frequently use a broad
swathe of capabilities then we might want to break it up further somehow
between the different internal aspects (multiple privsep helpers?) - but is
there such a case? There's also no problems with mixing privsep for
frequent operations with the existing sudo/rootwrap approach for rare/awkward
cases.
- Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
(mailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][security][rootwrap] Proposal to replace rootwrap/sudo with privsep helper process (for neutron, but others too)
Angus Lees wrote: So inspired by the Rootwrap on root-intensive nodes thread, I went and wrote a proof-of-concept privsep daemon for neutron: https://review.openstack.org/#/c/155631 Nice work! Trying to check where the security model is actually weaker than the one provided by rootwrap here... If I read this correctly, one of the drawbacks compared to the rootwrap approach is that all the functions are accessible on any node running the privsep daemon. When properly packaged, the rootwrap system only has the relevant filter definition files present, which dramatically reduces what you allow to run as root there. For example, nova-metadata nodes can only run iptables to add a firewall rule, since that's the only filter definition file shipped on those nodes. That's a benefit of shipping allowed commands in the configuration, rather than in the code. With the proposed system, the nova-metadata node running with nova-privsep would have access to any function defined there, including the rather permissive file manipulations that nova-compute needs. Maybe one way to solve that would be to have several different privsep daemons in nova to run on the various types of nodes, each with their own set of allowed functions... Additionally, rootwrap was extremely lean in terms of dependencies -- basically only allowing Python stdlib module imports to reduce the envelope of code you need to trust to run things as root. Here you are reusing the Neutron agent framework, which has a pretty long list of module imports, which are all imported before the rights are dropped. That means you end up blindly trusting a widely larger envelope of modules. My suggestion would be to not run it as an agent but as a standalone daemon with an aggressively-limited set of module imports (ideally only stdlib). That's all I could spot with a cursory look :) Cheers, -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][security][rootwrap] Proposal to replace rootwrap/sudo with privsep helper process (for neutron, but others too)
So inspired by the Rootwrap on root-intensive nodes thread, I went and
wrote a proof-of-concept privsep daemon for neutron:
https://review.openstack.org/#/c/155631
There's nothing neutron-specific in the core mechanism and it could easily
be moved out into a common (oslo) library and reused across other projects.
The basic principles guiding a few design choices were:
- it had to be obvious what code would run with elevated privileges
- the interface between that and the rest of the system had to be easy to
understand and audit
- it had to be about as easy as just implementing a function to add new
functionality
The current code does the predictable things to get a privileged buddy
process: assumes you started as root, creates a socketpair, forks, keeps
limited perms and setuids the privileged process, then reads commands over
the socket. When the socket closes (for whatever reason), the daemon exits.
Currently it scrobbles around below the neutron.agent.privileged._commands
namespace and allows you to invoke any normal function that doesn't start
start with an underscore. I think I will change this to use some sort of
more explicit decorator, but you get the idea.
On the client side, it generates client stubs of all the same functions
below neutron.agent.privileged.commands at import-time. Using the daemon
is then as simple as just calling the client stub:
from neutron.agent.privileged import daemon as privsep_daemon
def main():
privsep_daemon.start()
...
from neutron.agent.privileged.commands import ip_lib as priv_ip
def foo():
# Need to create a new veth interface pair - that usually requires
root/NET_ADMIN
priv_ip.CreateLink('veth', 'veth0', peer='veth1')
Because we now have elevated privileges directly (on the privileged daemon
side) without having to shell out through sudo, we can do all sorts of
nicer things like just using netlink directly to configure networking.
This avoids the overhead of executing subcommands, the ugliness (and
danger) of generating command lines and regex parsing output, and make us
less reliant on specific versions of command line tools (since the kernel
API should be very stable).
I demonstrate some of that in the above change by a set of privileged ipset
functions that still call out to commands but don't use sh -c or sudo
anywhere (so are immune to shell metacharacters in arguments), and ip_lib
functions that use pyroute2 to just call netlink directly for network
interface create/delete/update.
Please discuss. I could have done this in a spec, but I felt the basic
concept and motivation was obvious and the code specifics were of such
importance that this was better explored in a poc change. I can post-facto
write a spec if it turns out folks would prefer that.
- Gus
(if you're curious, it took about a day to write the code, and then about 3
long days of debugging eventlet-related conflicts with the 3rd party
libraries I'd just pulled in. +1 to removing eventlet, particularly in
low-concurrent-queries agent processes where we can presumably just remove
it and use system threads without any further thought)
On Fri Feb 06 2015 at 5:52:44 PM Steven Dake (stdake) std...@cisco.com
wrote:
On 2/4/15, 10:24 AM, Daniel P. Berrange berra...@redhat.com wrote:
On Wed, Feb 04, 2015 at 09:10:06AM -0800, James E. Blair wrote:
Thierry Carrez thie...@openstack.org writes:
You make a good point when you mention traditional distro here. I
would argue that containers are slightly changing the rules of the
don't-run-as-root game.
Solution (2) aligns pretty well with container-powered OpenStack
deployments -- running compute nodes as root in a container (and
embracing abovementioned simplicity/performance gains) sounds like a
pretty strong combo.
This sounds at least a little like a suggestion that containers are a
substitute for the security provided by running non-root. The security
landscape around containers is complex, and while there are a lot of
benefits, I believe the general consensus is that uid 0 processes should
not be seen as fully isolated.
From https://docs.docker.com/articles/security/ :
Docker containers are, by default, quite secure; especially if you
take care of running your processes inside the containers as
non-privileged users (i.e., non-root).
Which is not to say that using containers is not a good idea, but
rather, if one does, one should avoid running as root (perhaps with
capabilities), and use selinux (or similar).
Yep, I've seen attempts by some folks to run nova-compute and libvirtd
and QEMU inside a docker container. Because of the inherantly privileged
nature of what Nova/libvirt/qemu need to do, you end up having to share
all the host namespaces with the docker container, except for the
filesystem
namespace and even that you need to bind mount a bunch of stuff over. As
a result the container isn't really offerring
Re: [openstack-dev] [neutron][security][rootwrap] Proposal to replace rootwrap/sudo with privsep helper process (for neutron, but others too)
On 13 Feb 2015 17:42, Angus Lees g...@inodes.org wrote: So inspired by the Rootwrap on root-intensive nodes thread, I went and wrote a proof-of-concept privsep daemon for neutron: https://review.openstack.org/#/c/155631 There's nothing neutron-specific in the core mechanism and it could easily be moved out into a common (oslo) library and reused across other projects. Bravo. More conceptual than a code review my questions are. msgpack rather than protobuf ? Given your previous experience there I'm just curious. Are you concerned that commands might call into less trusted areas of code? Would it make sense to have the privileged commands be separate somehow to avoid this? Rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][security][rootwrap] Proposal to replace rootwrap/sudo with privsep helper process (for neutron, but others too)
ᐧ
from neutron.agent.privileged.commands import ip_lib as priv_ip
def foo():
# Need to create a new veth interface pair - that usually requires
root/NET_ADMIN
priv_ip.CreateLink('veth', 'veth0', peer='veth1')
Because we now have elevated privileges directly (on the privileged daemon
side) without having to shell out through sudo, we can do all sorts of
nicer things like just using netlink directly to configure networking.
This avoids the overhead of executing subcommands, the ugliness (and
danger) of generating command lines and regex parsing output, and make us
less reliant on specific versions of command line tools (since the kernel
API should be very stable).
One of the advantages of spawning a new process is being able to use flags
to clone(2) and to set capabilities. This basically means to create
containers, by some definition. Anything you have in a privileged daemon
or privileged process ideally should reduce its privilege set for any
operation it performs. That might mean it clones itself and executes
Python, or it may execvp an executable, but either way, the new process
would have less-than-full-privilege.
For instance, writing a file might require root access, but does not need
the ability to load kernel modules. Changing network interfaces does not
need access to the filesystem, no more than changes to the filesystem needs
access to the network. The capabilities and namespaces mechanisms resolve
these security conundrums and simplify principle of least privilege.
Regards,
Eric Windisch
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][security][rootwrap] Proposal to replace rootwrap/sudo with privsep helper process (for neutron, but others too)
On Fri Feb 13 2015 at 4:05:33 PM Robert Collins robe...@robertcollins.net wrote: On 13 Feb 2015 17:42, Angus Lees g...@inodes.org wrote: So inspired by the Rootwrap on root-intensive nodes thread, I went and wrote a proof-of-concept privsep daemon for neutron: https://review.openstack.org/#/c/155631 There's nothing neutron-specific in the core mechanism and it could easily be moved out into a common (oslo) library and reused across other projects. Bravo. More conceptual than a code review my questions are. msgpack rather than protobuf ? Given your previous experience there I'm just curious. I have no educated preference between the two, and I didn't know of any high-performance precedent within openstack. msgpack was just the first thing I came across that seemed well supported, fast, and only handled dumb types (no object auto-vivifying features that might backfire on us). We could use json too if we wanted to avoid a new dependency, or presumably numerous other choices. Are you concerned that commands might call into less trusted areas of code? Would it make sense to have the privileged commands be separate somehow to avoid this? Hrm, not particularly, although we should explore any implications. If a standalone chunk of python imported other python libraries, then they may have a path that ends up with them able to be called - which I figure is similar to the current situation that also requires an explicit python import (or some other chain of object references). If there's a bug that lets you escape the python level and run arbitrary C code, then it won't matter what's already loaded and we only have the linux capabilities/permissions mechanisms to save us. In addition, the current simple fork/no-exec is also good for sharing most of the pages in memory - making the overhead extremely minimal. Oh, if you mean separate just in a filesystem/code organisation sense, rather than a Linux process sense, then yes I do think they should be in a separate place for ease of auditing. in my change above I have them all below a particular neutron.agent.privileged._commands prefix, and the communication assumes/restricts it to this. We can of course pick another namespace prefix, but I agree that even with some different decorator-based method, I don't think we should just have privileged commands scattered anywhere throughout our regular codebase. - Gus __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
