Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
Excerpts from Thierry Carrez's message of 2015-09-10 19:35:21 +0200: > Doug Hellmann wrote: > > Excerpts from Thierry Carrez's message of 2015-09-10 14:23:34 +0200: > >> My quick 2cents on this. Rootwrap was designed as a generic solution to > >> wrap privileged calls. That's why filter files are part of its > >> "configuration". The problem is, OpenStack needs a pretty precise set of > >> those filters to be "configured" to run properly. So it's configuration > >> for rootwrap, but not "configuration" for OpenStack. > > > > That makes them sound like data, not configuration. If that's the case, > > Python's pkgutil module is an existing API for putting a data file > > inside a library and then accessing it. Maybe we should look at moving > > the files to a place that lets us use that, instead of requiring any > > deployer-based configuration at all. Combining that with the "symbolic" > > suggestion from Sean, we would use the package name as the symbol and > > there would be a well-defined resource name within that package to use > > with pkgutil.get_data() [1]. > > That sounds promising. One trick is that it's the consuming application > that needs to ship the filters, not the library itself (so rootwrap > would have to look into nova resources, not rootwrap resources). Another > trick is that it should require root rights (not nova rights) to change > those resources, otherwise the security model is broken (the whole idea > of rootwrap being to restrict what a compromised nova user can do to the > system). If we put the data file inside the library, it will be installed where the code lives, so it should have the permission protection we need. The symbol names passed to rootwrap would be hard-coded inside the application making the call, so that would also be protected. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
Excerpts from Sean Dague's message of 2015-09-10 14:11:20 -0400: > On 09/10/2015 01:05 PM, Doug Hellmann wrote: > > Excerpts from Thierry Carrez's message of 2015-09-10 14:23:34 +0200: > >> Sean Dague wrote: > >>> Right now, they are all a bunch of files, they can be anywhere. And then > >>> you have other files that have to reference these files by path, which > >>> can be anywhere. We could just punt in that part and say "punt! every > >>> installer and configuration management install needs to solve this on > >>> their own." I'm not convinced that's a good answer. The os-brick filters > >>> aren't really config. If you change them all that happens is > >>> terribleness. Stuff stops working, and you don't know why. They are data > >>> to exchange with another process about how to function. Honestly, they > >>> should probably be python code that's imported by rootwrap. > >>> > >>> Much like the issues around clouds failing when you try to GET /v2 on > >>> the Nova API (because we have a bunch of knobs you have to align for SSL > >>> termination, and a bunch of deployers didn't), I don't think we should > >>> be satisfied with "there's a config for that!" when all that config > >>> means is that someone can break their configuration if they don't get it > >>> exactly right. > >> > >> My quick 2cents on this. Rootwrap was designed as a generic solution to > >> wrap privileged calls. That's why filter files are part of its > >> "configuration". The problem is, OpenStack needs a pretty precise set of > >> those filters to be "configured" to run properly. So it's configuration > >> for rootwrap, but not "configuration" for OpenStack. > > > > That makes them sound like data, not configuration. If that's the case, > > Python's pkgutil module is an existing API for putting a data file > > inside a library and then accessing it. Maybe we should look at moving > > the files to a place that lets us use that, instead of requiring any > > deployer-based configuration at all. Combining that with the "symbolic" > > suggestion from Sean, we would use the package name as the symbol and > > there would be a well-defined resource name within that package to use > > with pkgutil.get_data() [1]. > > > > Doug > > > > [1] https://docs.python.org/2.7/library/pkgutil.html#pkgutil.get_data > > That sounds reasonable as well. Monty and dstuft had sent us down the > get_resource direction instead using setuptools - > https://pythonhosted.org/setuptools/pkg_resources.html. If you think > that get_data is a better option, this would apply just as well there. We rely heavily on setuptools, so using that API is probably just as good. Frankly I don't know the implementation differences. The point is to use something that exists, though, rather than inventing one, and so either would satisfy me in that regard. > The important thing is: filters are tied to code, so should be able to > be looked up symbolicly based on the code level that's installed. Right, we still need to tell rootwrap what package names to look at, as Thierry pointed out. So the existing path option probably needs to be deprecated in favor of a new way of telling the daemon what packages to consult instead. Doug > > -Sean > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
On 09/10/2015 01:05 PM, Doug Hellmann wrote: > Excerpts from Thierry Carrez's message of 2015-09-10 14:23:34 +0200: >> Sean Dague wrote: >>> Right now, they are all a bunch of files, they can be anywhere. And then >>> you have other files that have to reference these files by path, which >>> can be anywhere. We could just punt in that part and say "punt! every >>> installer and configuration management install needs to solve this on >>> their own." I'm not convinced that's a good answer. The os-brick filters >>> aren't really config. If you change them all that happens is >>> terribleness. Stuff stops working, and you don't know why. They are data >>> to exchange with another process about how to function. Honestly, they >>> should probably be python code that's imported by rootwrap. >>> >>> Much like the issues around clouds failing when you try to GET /v2 on >>> the Nova API (because we have a bunch of knobs you have to align for SSL >>> termination, and a bunch of deployers didn't), I don't think we should >>> be satisfied with "there's a config for that!" when all that config >>> means is that someone can break their configuration if they don't get it >>> exactly right. >> >> My quick 2cents on this. Rootwrap was designed as a generic solution to >> wrap privileged calls. That's why filter files are part of its >> "configuration". The problem is, OpenStack needs a pretty precise set of >> those filters to be "configured" to run properly. So it's configuration >> for rootwrap, but not "configuration" for OpenStack. > > That makes them sound like data, not configuration. If that's the case, > Python's pkgutil module is an existing API for putting a data file > inside a library and then accessing it. Maybe we should look at moving > the files to a place that lets us use that, instead of requiring any > deployer-based configuration at all. Combining that with the "symbolic" > suggestion from Sean, we would use the package name as the symbol and > there would be a well-defined resource name within that package to use > with pkgutil.get_data() [1]. > > Doug > > [1] https://docs.python.org/2.7/library/pkgutil.html#pkgutil.get_data That sounds reasonable as well. Monty and dstuft had sent us down the get_resource direction instead using setuptools - https://pythonhosted.org/setuptools/pkg_resources.html. If you think that get_data is a better option, this would apply just as well there. The important thing is: filters are tied to code, so should be able to be looked up symbolicly based on the code level that's installed. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
Just out of curiosity, not 100% related to this thread, but other applications also bundle configuration files (for example heat templates @ https://github.com/openstack/magnum/tree/master/magnum/templates) Should there be some guidelines on how these config files are packaged and distributed for these projects as well (ie also use pkgutil...); probably a good idea to have one so that distributors/installers/packaging folk don't have to do something custom when packaging these applications/libraries. -Josh Thierry Carrez wrote: Doug Hellmann wrote: Excerpts from Thierry Carrez's message of 2015-09-10 14:23:34 +0200: My quick 2cents on this. Rootwrap was designed as a generic solution to wrap privileged calls. That's why filter files are part of its "configuration". The problem is, OpenStack needs a pretty precise set of those filters to be "configured" to run properly. So it's configuration for rootwrap, but not "configuration" for OpenStack. That makes them sound like data, not configuration. If that's the case, Python's pkgutil module is an existing API for putting a data file inside a library and then accessing it. Maybe we should look at moving the files to a place that lets us use that, instead of requiring any deployer-based configuration at all. Combining that with the "symbolic" suggestion from Sean, we would use the package name as the symbol and there would be a well-defined resource name within that package to use with pkgutil.get_data() [1]. That sounds promising. One trick is that it's the consuming application that needs to ship the filters, not the library itself (so rootwrap would have to look into nova resources, not rootwrap resources). Another trick is that it should require root rights (not nova rights) to change those resources, otherwise the security model is broken (the whole idea of rootwrap being to restrict what a compromised nova user can do to the system). __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
Doug Hellmann wrote: > Excerpts from Thierry Carrez's message of 2015-09-10 14:23:34 +0200: >> My quick 2cents on this. Rootwrap was designed as a generic solution to >> wrap privileged calls. That's why filter files are part of its >> "configuration". The problem is, OpenStack needs a pretty precise set of >> those filters to be "configured" to run properly. So it's configuration >> for rootwrap, but not "configuration" for OpenStack. > > That makes them sound like data, not configuration. If that's the case, > Python's pkgutil module is an existing API for putting a data file > inside a library and then accessing it. Maybe we should look at moving > the files to a place that lets us use that, instead of requiring any > deployer-based configuration at all. Combining that with the "symbolic" > suggestion from Sean, we would use the package name as the symbol and > there would be a well-defined resource name within that package to use > with pkgutil.get_data() [1]. That sounds promising. One trick is that it's the consuming application that needs to ship the filters, not the library itself (so rootwrap would have to look into nova resources, not rootwrap resources). Another trick is that it should require root rights (not nova rights) to change those resources, otherwise the security model is broken (the whole idea of rootwrap being to restrict what a compromised nova user can do to the system). -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
Sean Dague wrote: > On 09/10/2015 08:23 AM, Thierry Carrez wrote: >> Now another problem you're describing is that there is no single place >> where those filters end up, depending on the way the projects (or libs) >> are packaged and installed. And it's up to the distros to "fix" the >> filters_path in the configuration file so that it points to every single >> place where those end up. It's a problem (especially when you start to >> install things using multiple concurrent packaging systems), but it's >> not exactly new -- it's just that libraries shipping fliters file are >> even more likely to ship their filters somewhere weird. So maybe we can >> continue to live with that problem we always had, until the privsep >> system completely replaces rootwrap ? > > I do get this is where we came from. I feel like this doesn't really > address or understand that things are actually quite different when it > comes to libraries doing rootwrap. We've spent weeks attempting various > work arounds, and for Liberty just punted and said "os-brick, cinder, > and nova all must upgrade exactly at the same time". Because that's the > only solution that doesn't require pages of documentation that > installers will get wrong some times. > > I don't feel like that's an acceptable solution. And it also means that > "living" with it means that next cycle we're going to have to say "nova, > neutron, cinder, os-brick, and vif library must all upgrade at exactly > the same time". Which is clearly not a thing we want. Had we figured out > this rootwrap limitation early, os-brick would never have been put into > Nova because it makes the upgrade process demonstrably worse and more > fragile. Right. The only reason why packagers got it right the first time is because things were a lot simpler back then (Ubuntu was almost the only game in town) and I personally got involved and made sure they got it right. I suspect you're right when you assume that if we rely on them to push the right things at the right places at this stage in the cycle, that would probably go wrong in a lot of places. I also agree that os-brick is not worth it if the cost is crappy upgrades. I'm just unsure adding a layer of config on top will solve things. I kinda like the idea of not relying on the filesystem at all (as proposed by Doug) since the filesystem is so deployment-dependent. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
Excerpts from Thierry Carrez's message of 2015-09-10 14:23:34 +0200: > Sean Dague wrote: > > Right now, they are all a bunch of files, they can be anywhere. And then > > you have other files that have to reference these files by path, which > > can be anywhere. We could just punt in that part and say "punt! every > > installer and configuration management install needs to solve this on > > their own." I'm not convinced that's a good answer. The os-brick filters > > aren't really config. If you change them all that happens is > > terribleness. Stuff stops working, and you don't know why. They are data > > to exchange with another process about how to function. Honestly, they > > should probably be python code that's imported by rootwrap. > > > > Much like the issues around clouds failing when you try to GET /v2 on > > the Nova API (because we have a bunch of knobs you have to align for SSL > > termination, and a bunch of deployers didn't), I don't think we should > > be satisfied with "there's a config for that!" when all that config > > means is that someone can break their configuration if they don't get it > > exactly right. > > My quick 2cents on this. Rootwrap was designed as a generic solution to > wrap privileged calls. That's why filter files are part of its > "configuration". The problem is, OpenStack needs a pretty precise set of > those filters to be "configured" to run properly. So it's configuration > for rootwrap, but not "configuration" for OpenStack. That makes them sound like data, not configuration. If that's the case, Python's pkgutil module is an existing API for putting a data file inside a library and then accessing it. Maybe we should look at moving the files to a place that lets us use that, instead of requiring any deployer-based configuration at all. Combining that with the "symbolic" suggestion from Sean, we would use the package name as the symbol and there would be a well-defined resource name within that package to use with pkgutil.get_data() [1]. Doug [1] https://docs.python.org/2.7/library/pkgutil.html#pkgutil.get_data > > The way it was supposed to work out was that you would have a single > rootwrap on nodes and every component on that node needing filters would > drop them in some unique location. A library is just another component > needing filters, so os-brick could just deploy a few more filters on > nodes where it's installed. > > The trick is, to increase "security" we promoted usage of per-project > directories (so that Nova only has access to Nova privileged commands), > which translated into using a specific config file for Nova rootwrap > pointing to Nova filters. Now if we are willing to sacrifice that, we > could have a single directory per-node (/usr/share/rootwrap instead of > /usr/share/*/rootwrap) that makes most of the interpolation you're > describing unnecessary. > > Alternatively you could keep project-specific directories and have > os-brick drop symbolic links to its filters into both nova and > cinder-specific directories. It's slightly less flexible (since the lib > now has to know what consumes it) but keeps you from sacrificing "security". > > Now another problem you're describing is that there is no single place > where those filters end up, depending on the way the projects (or libs) > are packaged and installed. And it's up to the distros to "fix" the > filters_path in the configuration file so that it points to every single > place where those end up. It's a problem (especially when you start to > install things using multiple concurrent packaging systems), but it's > not exactly new -- it's just that libraries shipping fliters file are > even more likely to ship their filters somewhere weird. So maybe we can > continue to live with that problem we always had, until the privsep > system completely replaces rootwrap ? > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
On 09/10/2015 08:23 AM, Thierry Carrez wrote: > Sean Dague wrote: >> Right now, they are all a bunch of files, they can be anywhere. And then >> you have other files that have to reference these files by path, which >> can be anywhere. We could just punt in that part and say "punt! every >> installer and configuration management install needs to solve this on >> their own." I'm not convinced that's a good answer. The os-brick filters >> aren't really config. If you change them all that happens is >> terribleness. Stuff stops working, and you don't know why. They are data >> to exchange with another process about how to function. Honestly, they >> should probably be python code that's imported by rootwrap. >> >> Much like the issues around clouds failing when you try to GET /v2 on >> the Nova API (because we have a bunch of knobs you have to align for SSL >> termination, and a bunch of deployers didn't), I don't think we should >> be satisfied with "there's a config for that!" when all that config >> means is that someone can break their configuration if they don't get it >> exactly right. > > My quick 2cents on this. Rootwrap was designed as a generic solution to > wrap privileged calls. That's why filter files are part of its > "configuration". The problem is, OpenStack needs a pretty precise set of > those filters to be "configured" to run properly. So it's configuration > for rootwrap, but not "configuration" for OpenStack. > > The way it was supposed to work out was that you would have a single > rootwrap on nodes and every component on that node needing filters would > drop them in some unique location. A library is just another component > needing filters, so os-brick could just deploy a few more filters on > nodes where it's installed. > > The trick is, to increase "security" we promoted usage of per-project > directories (so that Nova only has access to Nova privileged commands), > which translated into using a specific config file for Nova rootwrap > pointing to Nova filters. Now if we are willing to sacrifice that, we > could have a single directory per-node (/usr/share/rootwrap instead of > /usr/share/*/rootwrap) that makes most of the interpolation you're > describing unnecessary. > > Alternatively you could keep project-specific directories and have > os-brick drop symbolic links to its filters into both nova and > cinder-specific directories. It's slightly less flexible (since the lib > now has to know what consumes it) but keeps you from sacrificing "security". > > Now another problem you're describing is that there is no single place > where those filters end up, depending on the way the projects (or libs) > are packaged and installed. And it's up to the distros to "fix" the > filters_path in the configuration file so that it points to every single > place where those end up. It's a problem (especially when you start to > install things using multiple concurrent packaging systems), but it's > not exactly new -- it's just that libraries shipping fliters file are > even more likely to ship their filters somewhere weird. So maybe we can > continue to live with that problem we always had, until the privsep > system completely replaces rootwrap ? I do get this is where we came from. I feel like this doesn't really address or understand that things are actually quite different when it comes to libraries doing rootwrap. We've spent weeks attempting various work arounds, and for Liberty just punted and said "os-brick, cinder, and nova all must upgrade exactly at the same time". Because that's the only solution that doesn't require pages of documentation that installers will get wrong some times. I don't feel like that's an acceptable solution. And it also means that "living" with it means that next cycle we're going to have to say "nova, neutron, cinder, os-brick, and vif library must all upgrade at exactly the same time". Which is clearly not a thing we want. Had we figured out this rootwrap limitation early, os-brick would never have been put into Nova because it makes the upgrade process demonstrably worse and more fragile. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
Sean Dague wrote: > Right now, they are all a bunch of files, they can be anywhere. And then > you have other files that have to reference these files by path, which > can be anywhere. We could just punt in that part and say "punt! every > installer and configuration management install needs to solve this on > their own." I'm not convinced that's a good answer. The os-brick filters > aren't really config. If you change them all that happens is > terribleness. Stuff stops working, and you don't know why. They are data > to exchange with another process about how to function. Honestly, they > should probably be python code that's imported by rootwrap. > > Much like the issues around clouds failing when you try to GET /v2 on > the Nova API (because we have a bunch of knobs you have to align for SSL > termination, and a bunch of deployers didn't), I don't think we should > be satisfied with "there's a config for that!" when all that config > means is that someone can break their configuration if they don't get it > exactly right. My quick 2cents on this. Rootwrap was designed as a generic solution to wrap privileged calls. That's why filter files are part of its "configuration". The problem is, OpenStack needs a pretty precise set of those filters to be "configured" to run properly. So it's configuration for rootwrap, but not "configuration" for OpenStack. The way it was supposed to work out was that you would have a single rootwrap on nodes and every component on that node needing filters would drop them in some unique location. A library is just another component needing filters, so os-brick could just deploy a few more filters on nodes where it's installed. The trick is, to increase "security" we promoted usage of per-project directories (so that Nova only has access to Nova privileged commands), which translated into using a specific config file for Nova rootwrap pointing to Nova filters. Now if we are willing to sacrifice that, we could have a single directory per-node (/usr/share/rootwrap instead of /usr/share/*/rootwrap) that makes most of the interpolation you're describing unnecessary. Alternatively you could keep project-specific directories and have os-brick drop symbolic links to its filters into both nova and cinder-specific directories. It's slightly less flexible (since the lib now has to know what consumes it) but keeps you from sacrificing "security". Now another problem you're describing is that there is no single place where those filters end up, depending on the way the projects (or libs) are packaged and installed. And it's up to the distros to "fix" the filters_path in the configuration file so that it points to every single place where those end up. It's a problem (especially when you start to install things using multiple concurrent packaging systems), but it's not exactly new -- it's just that libraries shipping fliters file are even more likely to ship their filters somewhere weird. So maybe we can continue to live with that problem we always had, until the privsep system completely replaces rootwrap ? -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
On 09/09/2015 07:16 PM, Doug Hellmann wrote: > Excerpts from Matt Riedemann's message of 2015-09-09 13:45:29 -0500: >> >> On 9/9/2015 1:04 PM, Doug Hellmann wrote: >>> Excerpts from Sean Dague's message of 2015-09-09 13:36:37 -0400: >> The problem with the static file paths in rootwrap.conf is that we don't >> know where those other library filter files are going to end up on the >> system when the library is installed. We could hard-code nova's >> rootwrap.conf filter_path to include "/etc/os-brick/rootwrap.d" but then > > I thought the configuration file passed to rootwrap was something the > deployer could change, which would let them fix the paths on their > system. Did I misunderstand what the argument was? > >> that means the deploy/config management tooling that installing this >> stuff needs to copy that directory structure from the os-brick install >> location (which we're finding non-deterministic, at least when using >> data_files with pbr) to the target location that rootwrap.conf cares about. >> >> That's why we were proposing adding things to rootwrap.conf that >> oslo.rootwrap can parse and process dynamically using the resource >> access stuff in pkg_resources, so we just say 'I want you to load the >> os-brick.filters file from the os-brick project, thanks.'. >> > > Doesn't that put the rootwrap config file for os-brick in a place the > deployer can't change it? Maybe they're not supposed to? If they're not, > then I agree that burying the actual file inside the library and using > something like pkgtools to get its contents makes more sense. Right now, they are all a bunch of files, they can be anywhere. And then you have other files that have to reference these files by path, which can be anywhere. We could just punt in that part and say "punt! every installer and configuration management install needs to solve this on their own." I'm not convinced that's a good answer. The os-brick filters aren't really config. If you change them all that happens is terribleness. Stuff stops working, and you don't know why. They are data to exchange with another process about how to function. Honestly, they should probably be python code that's imported by rootwrap. Much like the issues around clouds failing when you try to GET /v2 on the Nova API (because we have a bunch of knobs you have to align for SSL termination, and a bunch of deployers didn't), I don't think we should be satisfied with "there's a config for that!" when all that config means is that someone can break their configuration if they don't get it exactly right. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
Excerpts from Matt Riedemann's message of 2015-09-09 13:45:29 -0500: > > On 9/9/2015 1:04 PM, Doug Hellmann wrote: > > Excerpts from Sean Dague's message of 2015-09-09 13:36:37 -0400: > >> We've got a new pattern emerging where some of the key functionality in > >> services is moving into libraries that can be called from different > >> services. A good instance of this is os-brick, which has the setup / > >> config functionality for devices that sometimes need to be called by > >> cinder and sometimes need to be called by nova when setting up a guest. > >> Many of these actions need root access, so require rootwrap filters. > >> > >> The point of putting this logic into a library is that it's self > >> contained, and that it can be an upgrade unit that is distinct from the > >> upgrade unit of either nova or cinder. > >> > >> The problem rootwrap.conf. Projects ship an example rootwrap.conf > >> which specifies filter files like such: > >> > >> [DEFAULT] > >> # List of directories to load filter definitions from (separated by ','). > >> # These directories MUST all be only writeable by root ! > >> filters_path=/etc/nova/rootwrap.d,/usr/share/nova/rootwrap > >> > >> however, we'd really like to be loading those filters from files the > >> library controls, so they are in sync with the library functionality. > >> Knowing where those files are going to be turns out to be a really > >> interesting guessing game. And, for security reasons, having a super > >> large set of paths rootwrap is guessing at seems really unwise. > >> > >> It seems like what we really want is something more like this: > >> > >> [filters] > >> nova=compute,network > >> os-brick=os-brick > >> > >> Which would translate into a symbolic look up via: > >> > >> filter_file = resource_filename(project, '%s.filters' % filter) > >> ... read the filter file > > > > Right now rootwrap takes as input an oslo.config file, which it reads to > > find the filter_path config value, which it interprets as a directory > > containing a bunch of other INI files, which it then reads and merges > > together into a single set of filters. I'm not sure the symbolic lookup > > you're proposing is going to support that use of multiple files. Maybe > > it shouldn't? > > > > What about allowing filter_path to contain more than one directory > > to scan? That would let projects using os-brick pass their own path and > > os-brick's path, if it's different. > > > > Doug > > > >> > >> So that rootwrap would be referencing things symbolically instead of > >> static / filebased which is going to be different depending on install > >> method. > >> > >> > >> For liberty we just hacked around it and put the os-brick rules into > >> Nova and Cinder. It's late in the release, and a clear better path > >> forward wasn't out there. It does mean the upgrade of the two components > >> is a bit coupled in the fiber channel case. But it was the best we could > >> do. > >> > >> I'd like to get the discussion rolling about the proposed solution > >> above. It emerged from #openstack-cinder this morning as we attempted to > >> get some kind of workable solution and figure out what was next. We > >> should definitely do a summit session on this one to nail down the > >> details and the path forward. > >> > >> -Sean > >> > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > The problem with the static file paths in rootwrap.conf is that we don't > know where those other library filter files are going to end up on the > system when the library is installed. We could hard-code nova's > rootwrap.conf filter_path to include "/etc/os-brick/rootwrap.d" but then I thought the configuration file passed to rootwrap was something the deployer could change, which would let them fix the paths on their system. Did I misunderstand what the argument was? > that means the deploy/config management tooling that installing this > stuff needs to copy that directory structure from the os-brick install > location (which we're finding non-deterministic, at least when using > data_files with pbr) to the target location that rootwrap.conf cares about. > > That's why we were proposing adding things to rootwrap.conf that > oslo.rootwrap can parse and process dynamically using the resource > access stuff in pkg_resources, so we just say 'I want you to load the > os-brick.filters file from the os-brick project, thanks.'. > Doesn't that put the rootwrap config file for os-brick in a place the deployer can't change it? Maybe they're not supposed to? If they're not, then I agree that burying the actual file inside the library and using something like pkgtools to get its contents makes more sense. Doug
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
On Wed, Sep 09, 2015 at 03:33:36PM -0400, Sean Dague wrote: > On 09/09/2015 02:55 PM, Robert Collins wrote: > > On 10 September 2015 at 06:45, Matt Riedemann > > wrote: > >> > > So, I realise thats a bit sucky. My suggestion would be to just take > > the tactical approach of syncing things into each consuming tree - and > > dogpile onto the privsep daemon asap. This does look interesting, but I would be very hesitant to change everything right away to move from rootwrap to privsep, assuming privsep will land and be stable enough to use in time. > > syncing things to the consuming tree means that you've now coupled > upgrade of os-brick, cinder, and nova to be at the same time. Because > the code to use the filters is in os-brick, but the filters are in > cinder and nova. > > That's exactly the opposite direction from where we'd like to move. We > did that work around for Liberty, but that nearly completely makes > os-brick pointless if it now means cinder and nova must be in lockstep > all the time. Agreed. I would like to see a clean separation of these. The reason this is even a big issue right now is a command was added to os-brick's rootwrap that was not picked up by Nova and Cinder. It only affected fibre channel attached storage, so we didn't even realize there was an issue until the third party CI's of FC drivers started all failing. I do like the proposed approach of passing in the library to rootwrap and letting rootwrap take care of loading its filters. It does bring up some security questions, but as a consumer of a library I think it makes sense to tell rootwrap - hey I'm using this library over there, do what it says it needs to do. Sean (smcginnis) PS - pardon the mail client SNAFU just sent prior to this. Oops. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
On 10 September 2015 at 07:33, Sean Dague wrote: > On 09/09/2015 02:55 PM, Robert Collins wrote: >> On 10 September 2015 at 06:45, Matt Riedemann >> wrote: >>> >> >>> The problem with the static file paths in rootwrap.conf is that we don't >>> know where those other library filter files are going to end up on the >>> system when the library is installed. We could hard-code nova's >>> rootwrap.conf filter_path to include "/etc/os-brick/rootwrap.d" but then >>> that means the deploy/config management tooling that installing this stuff >>> needs to copy that directory structure from the os-brick install location >>> (which we're finding non-deterministic, at least when using data_files with >>> pbr) to the target location that rootwrap.conf cares about. >>> >>> That's why we were proposing adding things to rootwrap.conf that >>> oslo.rootwrap can parse and process dynamically using the resource access >>> stuff in pkg_resources, so we just say 'I want you to load the >>> os-brick.filters file from the os-brick project, thanks.'. >> >> So, I realise thats a bit sucky. My suggestion would be to just take >> the tactical approach of syncing things into each consuming tree - and >> dogpile onto the privsep daemon asap. > > syncing things to the consuming tree means that you've now coupled > upgrade of os-brick, cinder, and nova to be at the same time. Because > the code to use the filters is in os-brick, but the filters are in > cinder and nova. > > That's exactly the opposite direction from where we'd like to move. We > did that work around for Liberty, but that nearly completely makes > os-brick pointless if it now means cinder and nova must be in lockstep > all the time. > >> privsep is the outcome of Gus' experiments with having a Python API to >> talk a richer language than shell command lines to a privileged >> daemon, with one (or more) dedicated daemon processes per server >> process. It avoids all of the churn and difficulties in mapping >> complex things through the command line (none of our rootwrap files >> are actually secure). And its massively lower latency and better >> performing. >> >> https://review.openstack.org/#/c/204073/ > > If someone else wants to go down this path that's fine. I feel like > that's not an answer to the question at all, because it says that > instead of moving forward with a decoupling mechanism (and we want to do > this kind of library approach in the nova/neutron interaction next > cycle, so this isn't a one off) we have to go into a holding pattern and > completely tear up a piece of core infrastructure for N cycles. If it not helpful, its not helpful - sorry. My perspective was that we want privsep in deployments for M, so if the brick code can move to privsep during M, there's only one cycle during which rootwrap filters may need to be adjusted. > I'm no particular fan of rootwrap, but this suggestion seems pretty non > productive in solving the problems in front of us. Especially given how > deeply the calling interface is embedded in our programs today. AIUI the case here is that there is a library, and you call that. Which means the calling interface of rootwrap isn't broadly at issue - we only need to migrate the calls w/in brick to solve the issue. Anyhow, as you say, it may not be helpful - so I'll not kibbitz, EOF :). If I do have thoughts on the rootwrap thing specifically I'll write a separate reply. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
> Sent: Wednesday, September 09, 2015 at 2:33 PM > From: "Sean Dague" > To: "OpenStack Development Mailing List (not for usage questions)" > > Subject: Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC > > On 09/09/2015 02:55 PM, Robert Collins wrote: > > On 10 September 2015 at 06:45, Matt Riedemann > > wrote: > >> > > > >> The problem with the static file paths in rootwrap.conf is that we don't > >> know where those other library filter files are going to end up on the > >> system when the library is installed. We could hard-code nova's > >> rootwrap.conf filter_path to include "/etc/os-brick/rootwrap.d" but then > >> that means the deploy/config management tooling that installing this stuff > >> needs to copy that directory structure from the os-brick install location > >> (which we're finding non-deterministic, at least when using data_files with > >> pbr) to the target location that rootwrap.conf cares about. > >> > >> That's why we were proposing adding things to rootwrap.conf that > >> oslo.rootwrap can parse and process dynamically using the resource access > >> stuff in pkg_resources, so we just say 'I want you to load the > >> os-brick.filters file from the os-brick project, thanks.'. > > > > So, I realise thats a bit sucky. My suggestion would be to just take > > the tactical approach of syncing things into each consuming tree - and > > dogpile onto the privsep daemon asap. > > syncing things to the consuming tree means that you've now coupled > upgrade of os-brick, cinder, and nova to be at the same time. Because > the code to use the filters is in os-brick, but the filters are in > cinder and nova. > > That's exactly the opposite direction from where we'd like to move. We > did that work around for Liberty, but that nearly completely makes > os-brick pointless if it now means cinder and nova must be in lockstep > all the time. > > > privsep is the outcome of Gus' experiments with having a Python API to > > talk a richer language than shell command lines to a privileged > > daemon, with one (or more) dedicated daemon processes per server > > process. It avoids all of the churn and difficulties in mapping > > complex things through the command line (none of our rootwrap files > > are actually secure). And its massively lower latency and better > > performing. > > > > https://review.openstack.org/#/c/204073/ > > If someone else wants to go down this path that's fine. I feel like > that's not an answer to the question at all, because it says that > instead of moving forward with a decoupling mechanism (and we want to do > this kind of library approach in the nova/neutron interaction next > cycle, so this isn't a one off) we have to go into a holding pattern and > completely tear up a piece of core infrastructure for N cycles. > > I'm no particular fan of rootwrap, but this suggestion seems pretty non > productive in solving the problems in front of us. Especially given how > deeply the calling interface is embedded in our programs today. > > -Sean > > -- > Sean Dague > http://dague.net > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
On 09/09/2015 02:55 PM, Robert Collins wrote: > On 10 September 2015 at 06:45, Matt Riedemann > wrote: >> > >> The problem with the static file paths in rootwrap.conf is that we don't >> know where those other library filter files are going to end up on the >> system when the library is installed. We could hard-code nova's >> rootwrap.conf filter_path to include "/etc/os-brick/rootwrap.d" but then >> that means the deploy/config management tooling that installing this stuff >> needs to copy that directory structure from the os-brick install location >> (which we're finding non-deterministic, at least when using data_files with >> pbr) to the target location that rootwrap.conf cares about. >> >> That's why we were proposing adding things to rootwrap.conf that >> oslo.rootwrap can parse and process dynamically using the resource access >> stuff in pkg_resources, so we just say 'I want you to load the >> os-brick.filters file from the os-brick project, thanks.'. > > So, I realise thats a bit sucky. My suggestion would be to just take > the tactical approach of syncing things into each consuming tree - and > dogpile onto the privsep daemon asap. syncing things to the consuming tree means that you've now coupled upgrade of os-brick, cinder, and nova to be at the same time. Because the code to use the filters is in os-brick, but the filters are in cinder and nova. That's exactly the opposite direction from where we'd like to move. We did that work around for Liberty, but that nearly completely makes os-brick pointless if it now means cinder and nova must be in lockstep all the time. > privsep is the outcome of Gus' experiments with having a Python API to > talk a richer language than shell command lines to a privileged > daemon, with one (or more) dedicated daemon processes per server > process. It avoids all of the churn and difficulties in mapping > complex things through the command line (none of our rootwrap files > are actually secure). And its massively lower latency and better > performing. > > https://review.openstack.org/#/c/204073/ If someone else wants to go down this path that's fine. I feel like that's not an answer to the question at all, because it says that instead of moving forward with a decoupling mechanism (and we want to do this kind of library approach in the nova/neutron interaction next cycle, so this isn't a one off) we have to go into a holding pattern and completely tear up a piece of core infrastructure for N cycles. I'm no particular fan of rootwrap, but this suggestion seems pretty non productive in solving the problems in front of us. Especially given how deeply the calling interface is embedded in our programs today. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
On 10 September 2015 at 06:45, Matt Riedemann wrote: > > The problem with the static file paths in rootwrap.conf is that we don't > know where those other library filter files are going to end up on the > system when the library is installed. We could hard-code nova's > rootwrap.conf filter_path to include "/etc/os-brick/rootwrap.d" but then > that means the deploy/config management tooling that installing this stuff > needs to copy that directory structure from the os-brick install location > (which we're finding non-deterministic, at least when using data_files with > pbr) to the target location that rootwrap.conf cares about. > > That's why we were proposing adding things to rootwrap.conf that > oslo.rootwrap can parse and process dynamically using the resource access > stuff in pkg_resources, so we just say 'I want you to load the > os-brick.filters file from the os-brick project, thanks.'. So, I realise thats a bit sucky. My suggestion would be to just take the tactical approach of syncing things into each consuming tree - and dogpile onto the privsep daemon asap. privsep is the outcome of Gus' experiments with having a Python API to talk a richer language than shell command lines to a privileged daemon, with one (or more) dedicated daemon processes per server process. It avoids all of the churn and difficulties in mapping complex things through the command line (none of our rootwrap files are actually secure). And its massively lower latency and better performing. https://review.openstack.org/#/c/204073/ -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
On 9/9/2015 1:04 PM, Doug Hellmann wrote: Excerpts from Sean Dague's message of 2015-09-09 13:36:37 -0400: We've got a new pattern emerging where some of the key functionality in services is moving into libraries that can be called from different services. A good instance of this is os-brick, which has the setup / config functionality for devices that sometimes need to be called by cinder and sometimes need to be called by nova when setting up a guest. Many of these actions need root access, so require rootwrap filters. The point of putting this logic into a library is that it's self contained, and that it can be an upgrade unit that is distinct from the upgrade unit of either nova or cinder. The problem rootwrap.conf. Projects ship an example rootwrap.conf which specifies filter files like such: [DEFAULT] # List of directories to load filter definitions from (separated by ','). # These directories MUST all be only writeable by root ! filters_path=/etc/nova/rootwrap.d,/usr/share/nova/rootwrap however, we'd really like to be loading those filters from files the library controls, so they are in sync with the library functionality. Knowing where those files are going to be turns out to be a really interesting guessing game. And, for security reasons, having a super large set of paths rootwrap is guessing at seems really unwise. It seems like what we really want is something more like this: [filters] nova=compute,network os-brick=os-brick Which would translate into a symbolic look up via: filter_file = resource_filename(project, '%s.filters' % filter) ... read the filter file Right now rootwrap takes as input an oslo.config file, which it reads to find the filter_path config value, which it interprets as a directory containing a bunch of other INI files, which it then reads and merges together into a single set of filters. I'm not sure the symbolic lookup you're proposing is going to support that use of multiple files. Maybe it shouldn't? What about allowing filter_path to contain more than one directory to scan? That would let projects using os-brick pass their own path and os-brick's path, if it's different. Doug So that rootwrap would be referencing things symbolically instead of static / filebased which is going to be different depending on install method. For liberty we just hacked around it and put the os-brick rules into Nova and Cinder. It's late in the release, and a clear better path forward wasn't out there. It does mean the upgrade of the two components is a bit coupled in the fiber channel case. But it was the best we could do. I'd like to get the discussion rolling about the proposed solution above. It emerged from #openstack-cinder this morning as we attempted to get some kind of workable solution and figure out what was next. We should definitely do a summit session on this one to nail down the details and the path forward. -Sean __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev The problem with the static file paths in rootwrap.conf is that we don't know where those other library filter files are going to end up on the system when the library is installed. We could hard-code nova's rootwrap.conf filter_path to include "/etc/os-brick/rootwrap.d" but then that means the deploy/config management tooling that installing this stuff needs to copy that directory structure from the os-brick install location (which we're finding non-deterministic, at least when using data_files with pbr) to the target location that rootwrap.conf cares about. That's why we were proposing adding things to rootwrap.conf that oslo.rootwrap can parse and process dynamically using the resource access stuff in pkg_resources, so we just say 'I want you to load the os-brick.filters file from the os-brick project, thanks.'. -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [rootwrap] rootwrap and libraries - RFC
Excerpts from Sean Dague's message of 2015-09-09 13:36:37 -0400: > We've got a new pattern emerging where some of the key functionality in > services is moving into libraries that can be called from different > services. A good instance of this is os-brick, which has the setup / > config functionality for devices that sometimes need to be called by > cinder and sometimes need to be called by nova when setting up a guest. > Many of these actions need root access, so require rootwrap filters. > > The point of putting this logic into a library is that it's self > contained, and that it can be an upgrade unit that is distinct from the > upgrade unit of either nova or cinder. > > The problem rootwrap.conf. Projects ship an example rootwrap.conf > which specifies filter files like such: > > [DEFAULT] > # List of directories to load filter definitions from (separated by ','). > # These directories MUST all be only writeable by root ! > filters_path=/etc/nova/rootwrap.d,/usr/share/nova/rootwrap > > however, we'd really like to be loading those filters from files the > library controls, so they are in sync with the library functionality. > Knowing where those files are going to be turns out to be a really > interesting guessing game. And, for security reasons, having a super > large set of paths rootwrap is guessing at seems really unwise. > > It seems like what we really want is something more like this: > > [filters] > nova=compute,network > os-brick=os-brick > > Which would translate into a symbolic look up via: > > filter_file = resource_filename(project, '%s.filters' % filter) > ... read the filter file Right now rootwrap takes as input an oslo.config file, which it reads to find the filter_path config value, which it interprets as a directory containing a bunch of other INI files, which it then reads and merges together into a single set of filters. I'm not sure the symbolic lookup you're proposing is going to support that use of multiple files. Maybe it shouldn't? What about allowing filter_path to contain more than one directory to scan? That would let projects using os-brick pass their own path and os-brick's path, if it's different. Doug > > So that rootwrap would be referencing things symbolically instead of > static / filebased which is going to be different depending on install > method. > > > For liberty we just hacked around it and put the os-brick rules into > Nova and Cinder. It's late in the release, and a clear better path > forward wasn't out there. It does mean the upgrade of the two components > is a bit coupled in the fiber channel case. But it was the best we could do. > > I'd like to get the discussion rolling about the proposed solution > above. It emerged from #openstack-cinder this morning as we attempted to > get some kind of workable solution and figure out what was next. We > should definitely do a summit session on this one to nail down the > details and the path forward. > > -Sean > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [rootwrap] rootwrap and libraries - RFC
We've got a new pattern emerging where some of the key functionality in services is moving into libraries that can be called from different services. A good instance of this is os-brick, which has the setup / config functionality for devices that sometimes need to be called by cinder and sometimes need to be called by nova when setting up a guest. Many of these actions need root access, so require rootwrap filters. The point of putting this logic into a library is that it's self contained, and that it can be an upgrade unit that is distinct from the upgrade unit of either nova or cinder. The problem rootwrap.conf. Projects ship an example rootwrap.conf which specifies filter files like such: [DEFAULT] # List of directories to load filter definitions from (separated by ','). # These directories MUST all be only writeable by root ! filters_path=/etc/nova/rootwrap.d,/usr/share/nova/rootwrap however, we'd really like to be loading those filters from files the library controls, so they are in sync with the library functionality. Knowing where those files are going to be turns out to be a really interesting guessing game. And, for security reasons, having a super large set of paths rootwrap is guessing at seems really unwise. It seems like what we really want is something more like this: [filters] nova=compute,network os-brick=os-brick Which would translate into a symbolic look up via: filter_file = resource_filename(project, '%s.filters' % filter) ... read the filter file So that rootwrap would be referencing things symbolically instead of static / filebased which is going to be different depending on install method. For liberty we just hacked around it and put the os-brick rules into Nova and Cinder. It's late in the release, and a clear better path forward wasn't out there. It does mean the upgrade of the two components is a bit coupled in the fiber channel case. But it was the best we could do. I'd like to get the discussion rolling about the proposed solution above. It emerged from #openstack-cinder this morning as we attempted to get some kind of workable solution and figure out what was next. We should definitely do a summit session on this one to nail down the details and the path forward. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev