Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Thierry Carrez]

Sean Dague wrote:

On 07/04/2016 05:36 AM, Sean McGinnis wrote:

On Mon, Jul 04, 2016 at 01:59:09PM +0200, Thierry Carrez wrote:
[...]

The issue here is that oslo.rootwrap uses config files to determine
what to allow, but those are not really configuration files as far
as the application using them is concerned. Those must match the
code being executed.

So from Grenade perspective, those should really not be considered
configuration files, but application files.

[...]

+1

I have to agree with this perspective. They are config files, but they
are a special type of config file that is closely tied in to the code. I
think we should treat them as application files.

I say we allow these changes for grenade and move forward on this. I
think we all agree we want to move to privsep. As long as we document
this very clearly that these changes need to be made for upgrades, I'm
OK with that.

I would really like to be able to decided on this and move forward. I'm
afraid sticking with rootwrap for another cycle with just confuse things
and compound our issues.


So, can we just put them in python code inline then and abandon etc?

Special config files that we don't want anyone to touch, but we put in
/etc, aren't really a thing. You really can't have it both ways. Either
these are in the part of the filesystem where ops expect to change them,
or they are not.


Totally. As I said elsewhere in this thread, most distros are shipping 
them under /usr/share/, and oslo.rootwrap by default loads them 
from there as well. The only reason it (also) loads filters from 
/etc/ is that we wanted to let users *add* their own filters 
for their own extra plugin code. So I would argue that the right call is 
for devstack to deploy them in /usr/share/ (where they would 
just upgrade in place at the same time the code is updated) ?


Loading them from Python code inline would require more significant 
changes (including moving them in all projects, changing rootwrap to 
support inline filter loading and a min version bump everywhere...)


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Sean Dague]

On 07/04/2016 05:36 AM, Sean McGinnis wrote:

On Mon, Jul 04, 2016 at 01:59:09PM +0200, Thierry Carrez wrote:
[...]

The issue here is that oslo.rootwrap uses config files to determine
what to allow, but those are not really configuration files as far
as the application using them is concerned. Those must match the
code being executed.

So from Grenade perspective, those should really not be considered
configuration files, but application files.

[...]

+1

I have to agree with this perspective. They are config files, but they
are a special type of config file that is closely tied in to the code. I
think we should treat them as application files.

I say we allow these changes for grenade and move forward on this. I
think we all agree we want to move to privsep. As long as we document
this very clearly that these changes need to be made for upgrades, I'm
OK with that.

I would really like to be able to decided on this and move forward. I'm
afraid sticking with rootwrap for another cycle with just confuse things
and compound our issues.


So, can we just put them in python code inline then and abandon etc?

Special config files that we don't want anyone to touch, but we put in 
/etc, aren't really a thing. You really can't have it both ways. Either 
these are in the part of the filesystem where ops expect to change them, 
or they are not.


-Sean

--
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Thu, 7 Jul 2016 at 03:06 Matthew Treinish  wrote:

> On Wed, Jul 06, 2016 at 11:41:56AM -0500, Matt Riedemann wrote:
> > I just wonder how many deployments are actually relying on this, since as
> > noted elsewhere in this thread we don't really enforce this for all
> things,
> > only what happens to get tested in our CI system, e.g. the virtuozzo
> > rootwrap filters that don't have grenade testing.
>
> Sure, our testing coverage here is far from perfect, that's never been in
> dispute. It's always been best effort (which there has been limited in this
> space) like I'm not aware of anything doing any upgrade testing with
> virtuozzo enabled, or any of the other random ephemeral storage backends,
> **cough** ceph **cough**.  But, as I said before just because we don't
> catch all
> the issues isn't a reason to throw everything out the window.
>

So now we have identified some other examples recently added to the
codebase, that where not noticed by grenade for one reason or another.

Do we:
A) revert+postpone the virtuozzo changes until the next release?
B) add a releasenote saying you need to update the rootwrap filter first?

(Yes, this is a test)

It's boring, but not that hard to manually diff filters between releases -
I can do an audit if we'd like to build a list of other such changes.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Matthew Treinish]

On Wed, Jul 06, 2016 at 11:41:56AM -0500, Matt Riedemann wrote:
> On 7/6/2016 10:55 AM, Matthew Treinish wrote:
> > 
> > Well, for better or worse rootwrap filters are put in /etc and treated like 
> > a
> > config file. What you're essentially saying is that it shouldn't be config 
> > and
> > just be in code. I completely agree with that being what we want 
> > eventually, but
> > it's not how we advertise it today. Privsep sounds like it's our way of 
> > making
> > this migration. But, it doesn't change the status quo where it's this hybrid
> > config/code thing today, like policy was in nova before:
> > 
> > http://specs.openstack.org/openstack/nova-specs/specs/newton/approved/policy-in-code.html
> > 
> > (which has come up before as another tension point in the past during 
> > upgrades)
> > I don't think we should break what we're currently enforcing today because 
> > we
> > don't like the model we've built. We need to handle the migration to the new
> > better thing gracefully so we don't break people who are relying on our 
> > current
> > guarantees, regardless of how bad they are.
> > 
> > -Matt Treinish
> > 
> > 
> 
> I just wonder how many deployments are actually relying on this, since as
> noted elsewhere in this thread we don't really enforce this for all things,
> only what happens to get tested in our CI system, e.g. the virtuozzo
> rootwrap filters that don't have grenade testing.

Sure, our testing coverage here is far from perfect, that's never been in
dispute. It's always been best effort (which there has been limited in this
space) like I'm not aware of anything doing any upgrade testing with
virtuozzo enabled, or any of the other random ephemeral storage backends,
**cough** ceph **cough**.  But, as I said before just because we don't catch all
the issues isn't a reason to throw everything out the window.

> 
> Which is also why I'd like to get some operator perspective on this.
> 

I think what we'll find is the people who rely on this don't even realize it.
(which is kinda the point) I expect the people on the ops list are knowledgeable
enough and have enough experience to figure this kind of issue out and just
expect it during the course of an upgrade. This is more likely a trap for young
players who haven't even thought about this as being a potential issue before.
I don't think there is any disagreement we should move to something better in
this space. But, this is something we've said we would guarantee and I don't
think we should break that in the process of moving to the new better thing.

-Matt Treinish


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Matthew Treinish]

On Wed, Jul 06, 2016 at 06:20:30PM +0200, Thierry Carrez wrote:
> Matthew Treinish wrote:
> > > [...]
> > > Am I missing something else here?
> > 
> > Well, for better or worse rootwrap filters are put in /etc and treated like 
> > a
> > config file. What you're essentially saying is that it shouldn't be config 
> > and
> > just be in code. I completely agree with that being what we want 
> > eventually, but
> > it's not how we advertise it today.
> 
> Well, some (most ?) distros ship them as code rather than configuration,
> under /usr/share rather than under /etc. So one may argue that the issue is
> that devstack is installing them under /etc :)
> 

Devstack doesn't do anything special here, it just uses the project defaults.
For most cases that's what devstack strives to do that wherever possible. Your
issue is with nova and pretty much everything using rootwrap then. The fact that
most distros do this is just further indication that how we have things setup
today is the wrong way to handle this.

-Matt Treinish


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Clint Byrum]

Excerpts from Matthew Treinish's message of 2016-07-06 11:55:53 -0400:
> On Wed, Jul 06, 2016 at 10:34:49AM -0500, Matt Riedemann wrote:
> > On 6/27/2016 6:24 AM, Sean Dague wrote:
> > > On 06/26/2016 10:02 PM, Angus Lees wrote:
> > > > On Fri, 24 Jun 2016 at 20:48 Sean Dague  > > > > wrote:
> > > > 
> > > > On 06/24/2016 05:12 AM, Thierry Carrez wrote:
> > > > > I'm adding Possibility (0): change Grenade so that rootwrap
> > > > filters from
> > > > > N+1 are put in place before you upgrade.
> > > > 
> > > > If you do that as general course what you are saying is that every
> > > > installer and install process includes overwriting all of rootwrap
> > > > before every upgrade. Keep in mind we do upstream upgrade as 
> > > > offline,
> > > > which means that we've fully shut down the cloud. This would remove 
> > > > the
> > > > testing requirement that rootwrap configs were even compatible 
> > > > between N
> > > > and N+1. And you think this is theoretical, you should see the 
> > > > patches
> > > > I've gotten over the years to grenade because people didn't see an 
> > > > issue
> > > > with that at all. :)
> > > > 
> > > > I do get that people don't like the constraints we've self imposed, 
> > > > but
> > > > we've done that for very good reasons. The #1 complaint from 
> > > > operators,
> > > > for ever, has been the pain and danger of upgrading. That's why we 
> > > > are
> > > > still trademarking new Juno clouds. When you upgrade Apache, you 
> > > > don't
> > > > have to change your config files.
> > > > 
> > > > 
> > > > In case it got lost, I'm 100% on board with making upgrades safe and
> > > > straightforward, and I understand that grenade is merely a tool to help
> > > > us test ourselves against our process and not an enemy to be worked
> > > > around.  I'm an ops guy proud and true and hate you all for making
> > > > openstack hard to upgrade in the first place :P
> > > > 
> > > > Rootwrap configs need to be updated in line with new rootwrap-using code
> > > > - that's just the way the rootwrap security mechanism works, since the
> > > > security "trust" flows from the root-installed rootwrap config files.
> > > > 
> > > > I would like to clarify what our self-imposed upgrade rules are so that
> > > > I can design code within those constraints, and no-one is answering my
> > > > question so I'm just getting more confused as this thread progresses...
> > > > 
> > > > ***
> > > > What are we trying to impose on ourselves for upgrades for the present
> > > > and near future (ie: while rootwrap is still a thing)?
> > > > ***
> > > > 
> > > > A. Sean says above that we do "offline" upgrades, by which I _think_ he
> > > > means a host-by-host (or even global?) "turn everything (on the same
> > > > host/container) off, upgrade all files on disk for that host/container,
> > > > turn it all back on again".  If this is the model, then we can trivially
> > > > update rootwrap files during the "upgrade" step, and I don't see any
> > > > reason why we need to discuss anything further - except how we implement
> > > > this in grenade.
> > > > 
> > > > B. We need to support a mix of old + new code running on the same
> > > > host/container, running against the same config files (presumably
> > > > because we're updating service-by-service, or want to minimise the
> > > > service-unavailability during upgrades to literally just a process
> > > > restart).  So we need to think about how and when we stage config vs
> > > > code updates, and make sure that any overlap is appropriately allowed
> > > > for (expand-contract, etc).
> > > > 
> > > > C. We would like to just never upgrade rootwrap (or other config) files
> > > > ever again (implying a freeze in as_root command lines, effective ~a
> > > > year ago).  Any config update is an exception dealt with through
> > > > case-by-case process and release notes.
> > > > 
> > > > 
> > > > I feel like the grenade check currently implements (B) with a 6 month
> > > > lead time on config changes, but the "theory of upgrade" doc and our
> > > > verbal policy might actually be (C) (see this thread, eg), and Sean
> > > > above introduced the phrase "offline" which threw me completely into
> > > > thinking maybe we're aiming for (A).  You can see why I'm looking for
> > > > clarification  ;)
> > > 
> > > Ok, there is theory of what we are striving for, and there is what is
> > > viable to test consistently.
> > > 
> > > The thing we are shooting for is making the code Continuously
> > > Deployable. Which means the upgrade process should be "pip install -U
> > > $foo && $foo-manage db-sync" on the API surfaces and "pip install -U
> > > $foo; service restart" on everything else.
> > > 
> > > Logic we can put into the python install process is common logic shared
> > > by all deployment tools, and we can encode it in there. So all
> > > installers just get it.
> 

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Matt Riedemann]

On 7/6/2016 10:55 AM, Matthew Treinish wrote:


Well, for better or worse rootwrap filters are put in /etc and treated like a
config file. What you're essentially saying is that it shouldn't be config and
just be in code. I completely agree with that being what we want eventually, but
it's not how we advertise it today. Privsep sounds like it's our way of making
this migration. But, it doesn't change the status quo where it's this hybrid
config/code thing today, like policy was in nova before:

http://specs.openstack.org/openstack/nova-specs/specs/newton/approved/policy-in-code.html

(which has come up before as another tension point in the past during upgrades)
I don't think we should break what we're currently enforcing today because we
don't like the model we've built. We need to handle the migration to the new
better thing gracefully so we don't break people who are relying on our current
guarantees, regardless of how bad they are.

-Matt Treinish




I just wonder how many deployments are actually relying on this, since 
as noted elsewhere in this thread we don't really enforce this for all 
things, only what happens to get tested in our CI system, e.g. the 
virtuozzo rootwrap filters that don't have grenade testing.


Which is also why I'd like to get some operator perspective on this.

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Thierry Carrez]

Matthew Treinish wrote:

[...]
Am I missing something else here?


Well, for better or worse rootwrap filters are put in /etc and treated like a
config file. What you're essentially saying is that it shouldn't be config and
just be in code. I completely agree with that being what we want eventually, but
it's not how we advertise it today.


Well, some (most ?) distros ship them as code rather than configuration, 
under /usr/share rather than under /etc. So one may argue that the issue 
is that devstack is installing them under /etc :)


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Matthew Treinish]

On Wed, Jul 06, 2016 at 10:34:49AM -0500, Matt Riedemann wrote:
> On 6/27/2016 6:24 AM, Sean Dague wrote:
> > On 06/26/2016 10:02 PM, Angus Lees wrote:
> > > On Fri, 24 Jun 2016 at 20:48 Sean Dague  > > > wrote:
> > > 
> > > On 06/24/2016 05:12 AM, Thierry Carrez wrote:
> > > > I'm adding Possibility (0): change Grenade so that rootwrap
> > > filters from
> > > > N+1 are put in place before you upgrade.
> > > 
> > > If you do that as general course what you are saying is that every
> > > installer and install process includes overwriting all of rootwrap
> > > before every upgrade. Keep in mind we do upstream upgrade as offline,
> > > which means that we've fully shut down the cloud. This would remove 
> > > the
> > > testing requirement that rootwrap configs were even compatible 
> > > between N
> > > and N+1. And you think this is theoretical, you should see the patches
> > > I've gotten over the years to grenade because people didn't see an 
> > > issue
> > > with that at all. :)
> > > 
> > > I do get that people don't like the constraints we've self imposed, 
> > > but
> > > we've done that for very good reasons. The #1 complaint from 
> > > operators,
> > > for ever, has been the pain and danger of upgrading. That's why we are
> > > still trademarking new Juno clouds. When you upgrade Apache, you don't
> > > have to change your config files.
> > > 
> > > 
> > > In case it got lost, I'm 100% on board with making upgrades safe and
> > > straightforward, and I understand that grenade is merely a tool to help
> > > us test ourselves against our process and not an enemy to be worked
> > > around.  I'm an ops guy proud and true and hate you all for making
> > > openstack hard to upgrade in the first place :P
> > > 
> > > Rootwrap configs need to be updated in line with new rootwrap-using code
> > > - that's just the way the rootwrap security mechanism works, since the
> > > security "trust" flows from the root-installed rootwrap config files.
> > > 
> > > I would like to clarify what our self-imposed upgrade rules are so that
> > > I can design code within those constraints, and no-one is answering my
> > > question so I'm just getting more confused as this thread progresses...
> > > 
> > > ***
> > > What are we trying to impose on ourselves for upgrades for the present
> > > and near future (ie: while rootwrap is still a thing)?
> > > ***
> > > 
> > > A. Sean says above that we do "offline" upgrades, by which I _think_ he
> > > means a host-by-host (or even global?) "turn everything (on the same
> > > host/container) off, upgrade all files on disk for that host/container,
> > > turn it all back on again".  If this is the model, then we can trivially
> > > update rootwrap files during the "upgrade" step, and I don't see any
> > > reason why we need to discuss anything further - except how we implement
> > > this in grenade.
> > > 
> > > B. We need to support a mix of old + new code running on the same
> > > host/container, running against the same config files (presumably
> > > because we're updating service-by-service, or want to minimise the
> > > service-unavailability during upgrades to literally just a process
> > > restart).  So we need to think about how and when we stage config vs
> > > code updates, and make sure that any overlap is appropriately allowed
> > > for (expand-contract, etc).
> > > 
> > > C. We would like to just never upgrade rootwrap (or other config) files
> > > ever again (implying a freeze in as_root command lines, effective ~a
> > > year ago).  Any config update is an exception dealt with through
> > > case-by-case process and release notes.
> > > 
> > > 
> > > I feel like the grenade check currently implements (B) with a 6 month
> > > lead time on config changes, but the "theory of upgrade" doc and our
> > > verbal policy might actually be (C) (see this thread, eg), and Sean
> > > above introduced the phrase "offline" which threw me completely into
> > > thinking maybe we're aiming for (A).  You can see why I'm looking for
> > > clarification  ;)
> > 
> > Ok, there is theory of what we are striving for, and there is what is
> > viable to test consistently.
> > 
> > The thing we are shooting for is making the code Continuously
> > Deployable. Which means the upgrade process should be "pip install -U
> > $foo && $foo-manage db-sync" on the API surfaces and "pip install -U
> > $foo; service restart" on everything else.
> > 
> > Logic we can put into the python install process is common logic shared
> > by all deployment tools, and we can encode it in there. So all
> > installers just get it.
> > 
> > The challenge is there is no facility for config file management in
> > python native packaging. Which means that software which *depends* on
> > config files for new or even working features now moves from the camp of
> > CDable to manual upgrade needed. What you need to do is in 

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Matt Riedemann]

On 6/27/2016 6:24 AM, Sean Dague wrote:

On 06/26/2016 10:02 PM, Angus Lees wrote:

On Fri, 24 Jun 2016 at 20:48 Sean Dague > wrote:

On 06/24/2016 05:12 AM, Thierry Carrez wrote:
> I'm adding Possibility (0): change Grenade so that rootwrap
filters from
> N+1 are put in place before you upgrade.

If you do that as general course what you are saying is that every
installer and install process includes overwriting all of rootwrap
before every upgrade. Keep in mind we do upstream upgrade as offline,
which means that we've fully shut down the cloud. This would remove the
testing requirement that rootwrap configs were even compatible between N
and N+1. And you think this is theoretical, you should see the patches
I've gotten over the years to grenade because people didn't see an issue
with that at all. :)

I do get that people don't like the constraints we've self imposed, but
we've done that for very good reasons. The #1 complaint from operators,
for ever, has been the pain and danger of upgrading. That's why we are
still trademarking new Juno clouds. When you upgrade Apache, you don't
have to change your config files.


In case it got lost, I'm 100% on board with making upgrades safe and
straightforward, and I understand that grenade is merely a tool to help
us test ourselves against our process and not an enemy to be worked
around.  I'm an ops guy proud and true and hate you all for making
openstack hard to upgrade in the first place :P

Rootwrap configs need to be updated in line with new rootwrap-using code
- that's just the way the rootwrap security mechanism works, since the
security "trust" flows from the root-installed rootwrap config files.

I would like to clarify what our self-imposed upgrade rules are so that
I can design code within those constraints, and no-one is answering my
question so I'm just getting more confused as this thread progresses...

***
What are we trying to impose on ourselves for upgrades for the present
and near future (ie: while rootwrap is still a thing)?
***

A. Sean says above that we do "offline" upgrades, by which I _think_ he
means a host-by-host (or even global?) "turn everything (on the same
host/container) off, upgrade all files on disk for that host/container,
turn it all back on again".  If this is the model, then we can trivially
update rootwrap files during the "upgrade" step, and I don't see any
reason why we need to discuss anything further - except how we implement
this in grenade.

B. We need to support a mix of old + new code running on the same
host/container, running against the same config files (presumably
because we're updating service-by-service, or want to minimise the
service-unavailability during upgrades to literally just a process
restart).  So we need to think about how and when we stage config vs
code updates, and make sure that any overlap is appropriately allowed
for (expand-contract, etc).

C. We would like to just never upgrade rootwrap (or other config) files
ever again (implying a freeze in as_root command lines, effective ~a
year ago).  Any config update is an exception dealt with through
case-by-case process and release notes.


I feel like the grenade check currently implements (B) with a 6 month
lead time on config changes, but the "theory of upgrade" doc and our
verbal policy might actually be (C) (see this thread, eg), and Sean
above introduced the phrase "offline" which threw me completely into
thinking maybe we're aiming for (A).  You can see why I'm looking for
clarification  ;)


Ok, there is theory of what we are striving for, and there is what is
viable to test consistently.

The thing we are shooting for is making the code Continuously
Deployable. Which means the upgrade process should be "pip install -U
$foo && $foo-manage db-sync" on the API surfaces and "pip install -U
$foo; service restart" on everything else.

Logic we can put into the python install process is common logic shared
by all deployment tools, and we can encode it in there. So all
installers just get it.

The challenge is there is no facility for config file management in
python native packaging. Which means that software which *depends* on
config files for new or even working features now moves from the camp of
CDable to manual upgrade needed. What you need to do is in releasenotes,
not in code that's shipped with your software. Release notes are not
scriptable.

So, we've said, doing that has to be the exception and not the rule.
It's also the same reasoning behind our deprecation phase for all config
options. Things move from working (in N), to working with warnings (in
N+1), to not working (in N+2). Which allows people to CD across this
boundary, and do config file fixing in their Config Management tools
*post* upgrade.


rootwrap filters aren't config options, but I get the feeling we're 
shoe-horning grenade to treat them as such.


I get why 

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Matt Riedemann]

On 7/3/2016 10:25 PM, Angus Lees wrote:


I see there are already a few other additions to the rootwrap filters in
nova/cinder (the comments suggest (nova) libvirt/imagebackend.py,
(cinder) remotefs.py, and (both) vzstorage.py).  The various
privsep-only suggestions about fallback strategies don't help in these
other examples.  Any corresponding code changes that rely on these new
filters will also need to be reverted and resubmitted during next cycle
- or do what usually happens and slip under the radar as they are not
exercised by grenade.


This is a good point - there were a couple of rootwrap filters added to 
nova already for virtuozzo features (vz volume attach support and 
rescue/resize support using the prl_disk_tool binary). These would fail 
grenade if we ran it with resize and the virtuozzo config with libvirt.


It seems a bit crazy to me to have to land rootwrap filters 6 months 
ahead of the code that uses them though, which is why I didn't block 
those changes from getting in.




 - Gus


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



I haven't noticed anyone from the operators community weigh in on this 
thread, but I'm very curious to how they handle rootwrap filters when 
doing upgrades. I might start a separate thread in the operators list 
about that.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Sean McGinnis]

On Mon, Jul 04, 2016 at 01:59:09PM +0200, Thierry Carrez wrote:
[...]
> The issue here is that oslo.rootwrap uses config files to determine
> what to allow, but those are not really configuration files as far
> as the application using them is concerned. Those must match the
> code being executed.
> 
> So from Grenade perspective, those should really not be considered
> configuration files, but application files.
[...]

+1

I have to agree with this perspective. They are config files, but they
are a special type of config file that is closely tied in to the code. I
think we should treat them as application files.

I say we allow these changes for grenade and move forward on this. I
think we all agree we want to move to privsep. As long as we document
this very clearly that these changes need to be made for upgrades, I'm
OK with that.

I would really like to be able to decided on this and move forward. I'm
afraid sticking with rootwrap for another cycle with just confuse things
and compound our issues.

Sean


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Thierry Carrez]

Angus Lees wrote:

Ack. Ok .. so what's the additional difficulty around config files?  It
sounds like we can replace all the config files with something
completely different during the update phase, if we wanted to do so.
Indeed, it sounds like there isn't even a need to manage a deprecation
period for config files since there will never be mismatched code+config
(good, means fewer poorly tested legacy combinations in code).

Specifically, it seems grenade in both doc and code currently describes
something quite a bit stricter than this.  I think what we want is more
like "use devstack to deploy old; run/test; **use devstack to deploy
new** but pointing at existing DB/state_path from old; run/test,
interact with things created with old, etc".

A solution to our immediate rootwrap issue would be to just copy over
the rootwrap configs from 'new' during upgrade always, and this
shouldn't even be controversial.  I can't read body language over email,
so .. is everyone ok with this?  Why is this not what everyone was
jumping to suggest before now?


The issue here is that oslo.rootwrap uses config files to determine what 
to allow, but those are not really configuration files as far as the 
application using them is concerned. Those must match the code being 
executed.


So from Grenade perspective, those should really not be considered 
configuration files, but application files. They should be updated 
together with the code. Some distributions ship them in 
/usr/share/$project IIRC to reinforce that point. If anyone adds their 
own filters they would/should do that in separate files (and probably 
separate directories).


So you can hardcode that subtle distinction in Grenade (copy over the 
rootwrap config files from new). We could maybe also move (during 
Newton) rootwrap "config files" into a location that looks less like 
config files and more like code (and make rootwrap magically find that 
location). That way on upgrade rootwrap could use the Mitaka config 
files + the Newton code-like location (allowing both old and new 
commands to run until the Mitaka files are cleaned up). That might be an 
option if we don't want to special-case rootwrap files (the way they 
should always have been special-cased) in Grenade. Making rootwrap find 
that new location without requiring altering its configuration is a bit 
tricky though.


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Sat, 2 Jul 2016 at 01:02 Matt Riedemann 
wrote:

> On 6/28/2016 4:56 PM, Sean Dague wrote:
> > On 06/28/2016 01:46 AM, Angus Lees wrote:
> >> Ok, thanks for the in-depth explanation.
> >>
> >> My take away is that we need to file any rootwrap updates as exceptions
> >> for now (so releasenotes and grenade scripts).
> >
> > That is definitely the fall back if there is no better idea. However, we
> > should try really hard to figure out if there is a non manual way
> > through this. Even if that means some compat code that we keep for a
> > release to just bridge the gap.
> >
> > -Sean
> >
>
> Walter had this for os-brick:
>
> https://review.openstack.org/#/c/329586/
>
> That would fallback to rootwrap if privsep doesn't work / not available.
> That could be a workaround for upgrading with os-brick for Newton, with
> a big fat warning logged if we use it, and then drop it in Ocata and
> require privsep.
>

Yes, this is basically a version of "submit the rootwrap filter, then wait
6 months before submitting the code that needs it".   If we don't wish to
use the exception mechanism (or adjust the policy to upgrade conf before
code as I described earlier), then we can certainly do this.  Rather than
log a big fat warning if we use privsep, we may as well just revert the
privsep change for os-brick and then resubmit it next cycle.

This thread topic isn't actually about privsep however, although a
migration to privsep will mostly mitigate this in the future which is
perhaps why it is causing topic collisions for everyone.

I see there are already a few other additions to the rootwrap filters in
nova/cinder (the comments suggest (nova) libvirt/imagebackend.py, (cinder)
remotefs.py, and (both) vzstorage.py).  The various privsep-only
suggestions about fallback strategies don't help in these other examples.  Any
corresponding code changes that rely on these new filters will also need to
be reverted and resubmitted during next cycle - or do what usually happens
and slip under the radar as they are not exercised by grenade.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Matt Riedemann]

On 6/28/2016 4:56 PM, Sean Dague wrote:

On 06/28/2016 01:46 AM, Angus Lees wrote:

Ok, thanks for the in-depth explanation.

My take away is that we need to file any rootwrap updates as exceptions
for now (so releasenotes and grenade scripts).


That is definitely the fall back if there is no better idea. However, we
should try really hard to figure out if there is a non manual way
through this. Even if that means some compat code that we keep for a
release to just bridge the gap.

-Sean



Walter had this for os-brick:

https://review.openstack.org/#/c/329586/

That would fallback to rootwrap if privsep doesn't work / not available. 
That could be a workaround for upgrading with os-brick for Newton, with 
a big fat warning logged if we use it, and then drop it in Ocata and 
require privsep.


I'm not sure about os-vif, we weren't using that in Mitaka so it doesn't 
suffer from the same mitaka->newton upgrade issue, but will we get into 
any problems with newton->ocata? I know there was a change to devstack 
to configure nova to use privsep for os-vif:


https://review.openstack.org/#/c/327199/

And the os-vif integration change in nova has a rootwrap change for 
using privsep + os-vif:


https://review.openstack.org/#/c/269672/25/etc/nova/rootwrap.d/compute.filters

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Sean Dague]

On 06/28/2016 01:46 AM, Angus Lees wrote:

Ok, thanks for the in-depth explanation.

My take away is that we need to file any rootwrap updates as exceptions
for now (so releasenotes and grenade scripts).


That is definitely the fall back if there is no better idea. However, we 
should try really hard to figure out if there is a non manual way 
through this. Even if that means some compat code that we keep for a 
release to just bridge the gap.


-Sean

--
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

Ok, thanks for the in-depth explanation.

My take away is that we need to file any rootwrap updates as exceptions for
now (so releasenotes and grenade scripts).

 - Gus

On Mon, 27 Jun 2016 at 21:25 Sean Dague  wrote:

> On 06/26/2016 10:02 PM, Angus Lees wrote:
> > On Fri, 24 Jun 2016 at 20:48 Sean Dague  > > wrote:
> >
> > On 06/24/2016 05:12 AM, Thierry Carrez wrote:
> > > I'm adding Possibility (0): change Grenade so that rootwrap
> > filters from
> > > N+1 are put in place before you upgrade.
> >
> > If you do that as general course what you are saying is that every
> > installer and install process includes overwriting all of rootwrap
> > before every upgrade. Keep in mind we do upstream upgrade as offline,
> > which means that we've fully shut down the cloud. This would remove
> the
> > testing requirement that rootwrap configs were even compatible
> between N
> > and N+1. And you think this is theoretical, you should see the
> patches
> > I've gotten over the years to grenade because people didn't see an
> issue
> > with that at all. :)
> >
> > I do get that people don't like the constraints we've self imposed,
> but
> > we've done that for very good reasons. The #1 complaint from
> operators,
> > for ever, has been the pain and danger of upgrading. That's why we
> are
> > still trademarking new Juno clouds. When you upgrade Apache, you
> don't
> > have to change your config files.
> >
> >
> > In case it got lost, I'm 100% on board with making upgrades safe and
> > straightforward, and I understand that grenade is merely a tool to help
> > us test ourselves against our process and not an enemy to be worked
> > around.  I'm an ops guy proud and true and hate you all for making
> > openstack hard to upgrade in the first place :P
> >
> > Rootwrap configs need to be updated in line with new rootwrap-using code
> > - that's just the way the rootwrap security mechanism works, since the
> > security "trust" flows from the root-installed rootwrap config files.
> >
> > I would like to clarify what our self-imposed upgrade rules are so that
> > I can design code within those constraints, and no-one is answering my
> > question so I'm just getting more confused as this thread progresses...
> >
> > ***
> > What are we trying to impose on ourselves for upgrades for the present
> > and near future (ie: while rootwrap is still a thing)?
> > ***
> >
> > A. Sean says above that we do "offline" upgrades, by which I _think_ he
> > means a host-by-host (or even global?) "turn everything (on the same
> > host/container) off, upgrade all files on disk for that host/container,
> > turn it all back on again".  If this is the model, then we can trivially
> > update rootwrap files during the "upgrade" step, and I don't see any
> > reason why we need to discuss anything further - except how we implement
> > this in grenade.
> >
> > B. We need to support a mix of old + new code running on the same
> > host/container, running against the same config files (presumably
> > because we're updating service-by-service, or want to minimise the
> > service-unavailability during upgrades to literally just a process
> > restart).  So we need to think about how and when we stage config vs
> > code updates, and make sure that any overlap is appropriately allowed
> > for (expand-contract, etc).
> >
> > C. We would like to just never upgrade rootwrap (or other config) files
> > ever again (implying a freeze in as_root command lines, effective ~a
> > year ago).  Any config update is an exception dealt with through
> > case-by-case process and release notes.
> >
> >
> > I feel like the grenade check currently implements (B) with a 6 month
> > lead time on config changes, but the "theory of upgrade" doc and our
> > verbal policy might actually be (C) (see this thread, eg), and Sean
> > above introduced the phrase "offline" which threw me completely into
> > thinking maybe we're aiming for (A).  You can see why I'm looking for
> > clarification  ;)
>
> Ok, there is theory of what we are striving for, and there is what is
> viable to test consistently.
>
> The thing we are shooting for is making the code Continuously
> Deployable. Which means the upgrade process should be "pip install -U
> $foo && $foo-manage db-sync" on the API surfaces and "pip install -U
> $foo; service restart" on everything else.
>
> Logic we can put into the python install process is common logic shared
> by all deployment tools, and we can encode it in there. So all
> installers just get it.
>
> The challenge is there is no facility for config file management in
> python native packaging. Which means that software which *depends* on
> config files for new or even working features now moves from the camp of
> CDable to manual upgrade needed. What you need to do is in releasenotes,
> not in code that's shipped with your software. Release notes are not

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Sean Dague]

On 06/26/2016 10:02 PM, Angus Lees wrote:
> On Fri, 24 Jun 2016 at 20:48 Sean Dague  > wrote:
> 
> On 06/24/2016 05:12 AM, Thierry Carrez wrote:
> > I'm adding Possibility (0): change Grenade so that rootwrap
> filters from
> > N+1 are put in place before you upgrade.
> 
> If you do that as general course what you are saying is that every
> installer and install process includes overwriting all of rootwrap
> before every upgrade. Keep in mind we do upstream upgrade as offline,
> which means that we've fully shut down the cloud. This would remove the
> testing requirement that rootwrap configs were even compatible between N
> and N+1. And you think this is theoretical, you should see the patches
> I've gotten over the years to grenade because people didn't see an issue
> with that at all. :)
> 
> I do get that people don't like the constraints we've self imposed, but
> we've done that for very good reasons. The #1 complaint from operators,
> for ever, has been the pain and danger of upgrading. That's why we are
> still trademarking new Juno clouds. When you upgrade Apache, you don't
> have to change your config files.
> 
> 
> In case it got lost, I'm 100% on board with making upgrades safe and
> straightforward, and I understand that grenade is merely a tool to help
> us test ourselves against our process and not an enemy to be worked
> around.  I'm an ops guy proud and true and hate you all for making
> openstack hard to upgrade in the first place :P
> 
> Rootwrap configs need to be updated in line with new rootwrap-using code
> - that's just the way the rootwrap security mechanism works, since the
> security "trust" flows from the root-installed rootwrap config files.
> 
> I would like to clarify what our self-imposed upgrade rules are so that
> I can design code within those constraints, and no-one is answering my
> question so I'm just getting more confused as this thread progresses...
> 
> ***
> What are we trying to impose on ourselves for upgrades for the present
> and near future (ie: while rootwrap is still a thing)?
> ***
> 
> A. Sean says above that we do "offline" upgrades, by which I _think_ he
> means a host-by-host (or even global?) "turn everything (on the same
> host/container) off, upgrade all files on disk for that host/container,
> turn it all back on again".  If this is the model, then we can trivially
> update rootwrap files during the "upgrade" step, and I don't see any
> reason why we need to discuss anything further - except how we implement
> this in grenade.
> 
> B. We need to support a mix of old + new code running on the same
> host/container, running against the same config files (presumably
> because we're updating service-by-service, or want to minimise the
> service-unavailability during upgrades to literally just a process
> restart).  So we need to think about how and when we stage config vs
> code updates, and make sure that any overlap is appropriately allowed
> for (expand-contract, etc).
> 
> C. We would like to just never upgrade rootwrap (or other config) files
> ever again (implying a freeze in as_root command lines, effective ~a
> year ago).  Any config update is an exception dealt with through
> case-by-case process and release notes.
> 
> 
> I feel like the grenade check currently implements (B) with a 6 month
> lead time on config changes, but the "theory of upgrade" doc and our
> verbal policy might actually be (C) (see this thread, eg), and Sean
> above introduced the phrase "offline" which threw me completely into
> thinking maybe we're aiming for (A).  You can see why I'm looking for
> clarification  ;)

Ok, there is theory of what we are striving for, and there is what is
viable to test consistently.

The thing we are shooting for is making the code Continuously
Deployable. Which means the upgrade process should be "pip install -U
$foo && $foo-manage db-sync" on the API surfaces and "pip install -U
$foo; service restart" on everything else.

Logic we can put into the python install process is common logic shared
by all deployment tools, and we can encode it in there. So all
installers just get it.

The challenge is there is no facility for config file management in
python native packaging. Which means that software which *depends* on
config files for new or even working features now moves from the camp of
CDable to manual upgrade needed. What you need to do is in releasenotes,
not in code that's shipped with your software. Release notes are not
scriptable.

So, we've said, doing that has to be the exception and not the rule.
It's also the same reasoning behind our deprecation phase for all config
options. Things move from working (in N), to working with warnings (in
N+1), to not working (in N+2). Which allows people to CD across this
boundary, and do config file fixing in their Config Management tools
*post* upgrade.

Our testing, like all testing, 

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Mon, 27 Jun 2016 at 12:59 Tony Breeds  wrote:

> On Mon, Jun 27, 2016 at 02:02:35AM +, Angus Lees wrote:
>
> > ***
> > What are we trying to impose on ourselves for upgrades for the present
> and
> > near future (ie: while rootwrap is still a thing)?
> > ***
> >
> > A. Sean says above that we do "offline" upgrades, by which I _think_ he
> > means a host-by-host (or even global?) "turn everything (on the same
> > host/container) off, upgrade all files on disk for that host/container,
> > turn it all back on again".  If this is the model, then we can trivially
> > update rootwrap files during the "upgrade" step, and I don't see any
> reason
> > why we need to discuss anything further - except how we implement this in
> > grenade.
>
> Yes this one.  You must upgrade everything in the host/container to the
> same
> release at the same time.  For example we do NOT support running
> nova@liberty
> and cinder@mitaka.
>

Ack.  Ok .. so what's the additional difficulty around config files?  It
sounds like we can replace all the config files with something completely
different during the update phase, if we wanted to do so.  Indeed, it
sounds like there isn't even a need to manage a deprecation period for
config files since there will never be mismatched code+config (good, means
fewer poorly tested legacy combinations in code).

Specifically, it seems grenade in both doc and code currently describes
something quite a bit stricter than this.  I think what we want is more
like "use devstack to deploy old; run/test; **use devstack to deploy new**
but pointing at existing DB/state_path from old; run/test, interact with
things created with old, etc".

A solution to our immediate rootwrap issue would be to just copy over the
rootwrap configs from 'new' during upgrade always, and this shouldn't even
be controversial.  I can't read body language over email, so .. is everyone
ok with this?  Why is this not what everyone was jumping to suggest before
now?

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Tony Breeds]

On Mon, Jun 27, 2016 at 02:02:35AM +, Angus Lees wrote:

> ***
> What are we trying to impose on ourselves for upgrades for the present and
> near future (ie: while rootwrap is still a thing)?
> ***
> 
> A. Sean says above that we do "offline" upgrades, by which I _think_ he
> means a host-by-host (or even global?) "turn everything (on the same
> host/container) off, upgrade all files on disk for that host/container,
> turn it all back on again".  If this is the model, then we can trivially
> update rootwrap files during the "upgrade" step, and I don't see any reason
> why we need to discuss anything further - except how we implement this in
> grenade.

Yes this one.  You must upgrade everything in the host/container to the same
release at the same time.  For example we do NOT support running nova@liberty
and cinder@mitaka.

> B. We need to support a mix of old + new code running on the same
> host/container, running against the same config files (presumably because
> we're updating service-by-service, or want to minimise the
> service-unavailability during upgrades to literally just a process
> restart).  So we need to think about how and when we stage config vs code
> updates, and make sure that any overlap is appropriately allowed for
> (expand-contract, etc).

During the Austin summit we clearly said this wasn't a thing we did.  The
discussion was mostly centered around code but if we're running code that
needs filter $x then it's reasonable to have to install that filter at the
same time.

I can't find those points in the etherpad sessions where I thought it was
discussed.

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Fri, 24 Jun 2016 at 19:13 Thierry Carrez  wrote:

> In summary, I think the choice is between (1)+(4) and doing (4)
> directly. How doable is (4) in the timeframe we have ? Do we all agree
> that (4) is the endgame ?
>

I don't make predictions about development timelines within OpenStack.

Yes, I think (4) should be our endgame, and is certainly the most
future-proof approach.  I haven't thought through in detail nor spoken with
the ops community at all about how disruptive such a transition might be.

Thinking just now, I suspect any such transition needs a minimum of one
(probably 2) release cycles.  Theoretically:

- introduce top-of-main code that drops to unpriv uid if root (could
theoretically be done in current N cycle)
- change required deployment method during upgrade to release N+1
- In N+1 cycle require starting as root (or similar)
- In N+1 change top-of-main code to fork-privsep-then-drop instead of
drop-then-later-use-sudo
- So by the time N+2 is deployed we're done.

If we want to give more notice and manage a longer deprecation cycle before
requiring a start-as-root deployment (and I think we want both these
things), then expand as appropriate.

I suspect the change to deployment method will be easier to communicate and
less unpleasant if we do "every" openstack service in the one
tear-the-bandaid-off cycle rather than drag this out over years.  This of
course requires substantial cross-project consensus, coordination, and
timing.   Once the laughing subsides and we assume that isn't going to
happen, then we will need something machine-readable and in-tree that
correctly communicates the desired per-service approach at that point in
time - perhaps we could publish (eg) systemd unit files for our services
and adjust those to User=root/not-root as services switch to the new method
over time.  This requires communicating when in the upgrade cycle systemd
unit files (or the implied equivalents) should be updated, so goto 10 ;)

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Fri, 24 Jun 2016 at 20:48 Sean Dague  wrote:

> On 06/24/2016 05:12 AM, Thierry Carrez wrote:
> > I'm adding Possibility (0): change Grenade so that rootwrap filters from
> > N+1 are put in place before you upgrade.
>
> If you do that as general course what you are saying is that every
> installer and install process includes overwriting all of rootwrap
> before every upgrade. Keep in mind we do upstream upgrade as offline,
> which means that we've fully shut down the cloud. This would remove the
> testing requirement that rootwrap configs were even compatible between N
> and N+1. And you think this is theoretical, you should see the patches
> I've gotten over the years to grenade because people didn't see an issue
> with that at all. :)
>
> I do get that people don't like the constraints we've self imposed, but
> we've done that for very good reasons. The #1 complaint from operators,
> for ever, has been the pain and danger of upgrading. That's why we are
> still trademarking new Juno clouds. When you upgrade Apache, you don't
> have to change your config files.
>

In case it got lost, I'm 100% on board with making upgrades safe and
straightforward, and I understand that grenade is merely a tool to help us
test ourselves against our process and not an enemy to be worked around.
I'm an ops guy proud and true and hate you all for making openstack hard to
upgrade in the first place :P

Rootwrap configs need to be updated in line with new rootwrap-using code -
that's just the way the rootwrap security mechanism works, since the
security "trust" flows from the root-installed rootwrap config files.

I would like to clarify what our self-imposed upgrade rules are so that I
can design code within those constraints, and no-one is answering my
question so I'm just getting more confused as this thread progresses...

***
What are we trying to impose on ourselves for upgrades for the present and
near future (ie: while rootwrap is still a thing)?
***

A. Sean says above that we do "offline" upgrades, by which I _think_ he
means a host-by-host (or even global?) "turn everything (on the same
host/container) off, upgrade all files on disk for that host/container,
turn it all back on again".  If this is the model, then we can trivially
update rootwrap files during the "upgrade" step, and I don't see any reason
why we need to discuss anything further - except how we implement this in
grenade.

B. We need to support a mix of old + new code running on the same
host/container, running against the same config files (presumably because
we're updating service-by-service, or want to minimise the
service-unavailability during upgrades to literally just a process
restart).  So we need to think about how and when we stage config vs code
updates, and make sure that any overlap is appropriately allowed for
(expand-contract, etc).

C. We would like to just never upgrade rootwrap (or other config) files
ever again (implying a freeze in as_root command lines, effective ~a year
ago).  Any config update is an exception dealt with through case-by-case
process and release notes.


I feel like the grenade check currently implements (B) with a 6 month lead
time on config changes, but the "theory of upgrade" doc and our verbal
policy might actually be (C) (see this thread, eg), and Sean above
introduced the phrase "offline" which threw me completely into thinking
maybe we're aiming for (A).  You can see why I'm looking for clarification
 ;)

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Jeremy Stanley]

On 2016-06-24 06:47:08 -0400 (-0400), Sean Dague wrote:
[...]
> When you upgrade Apache, you don't have to change your config
> files.
[...]

*cough* (2.4) *cough*

But it still highlights your point. There's been basically one
Apache transition in recent history where a majority of people
running it needed to adjust their configs, and many of us considered
that a significant hassle.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Fri, 24 Jun 2016 at 21:04 Sean Dague  wrote:

> On 06/24/2016 05:19 AM, Daniel P. Berrange wrote:
> > On Fri, Jun 24, 2016 at 11:12:27AM +0200, Thierry Carrez wrote:
> >> No perfect answer here... I'm hesitating between (0), (1) and (4). (4)
> is
> >> IMHO the right solution, but it's a larger change for downstream. (1)
> is a
> >> bit of a hack, where we basically hardcode in rootwrap that it's being
> >> transitioned to privsep. That's fine, but only if we get rid of rootwrap
> >> soon. So only if we have a plan for (4) anyway. Option (0) is a bit of a
> >> hard sell for upgrade procedures -- if we need to take a hit in that
> area,
> >> let's do (4) directly...
> >>
> >> In summary, I think the choice is between (1)+(4) and doing (4)
> directly.
> >> How doable is (4) in the timeframe we have ? Do we all agree that (4)
> is the
> >> endgame ?
> >
> > We've already merged change to privsep to allow nova/cinder/etc to
> > initialize the default helper command to use rootwrap:
> >
> >
> https://github.com/openstack/oslo.privsep/commit/9bf606327d156de52c9418d5784cd7f29e243487
> >
> > So we just need new release of privsep & add code to nova to initialize
> > it and we're sorted.
>
> Actually, I don't think so. Matt ran that test scenario, and we're
> missing the rootwrap rule that lets privsep-helper run as root. So we
> fail to start the daemon from the unpriv nova compute process post upgrade.
>

Right, I thought that recent privsep change would address this, but we're
having this conversation because it turns out that simply updating code
only is not sufficient.  I (and I presume all the other members of the
earlier review/email discussion) had just assumed that we already had a
sensible process for making orderly changes to rootwrap files.

As I've since learned however, the current openstack upgrade process
doesn't talk about updating the rootwrap files, ever.  The only reason
we've been able to make reasonable progress since this was instituted is
either through the grenade exception mechanism, or by slipping things
through in drivers that are not tested by grenade.  Hence me opening this
fun can of worms for broader discussion in this thread.  You're welcome :-)

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Sean Dague]

On 06/24/2016 05:19 AM, Daniel P. Berrange wrote:
> On Fri, Jun 24, 2016 at 11:12:27AM +0200, Thierry Carrez wrote:
>> No perfect answer here... I'm hesitating between (0), (1) and (4). (4) is
>> IMHO the right solution, but it's a larger change for downstream. (1) is a
>> bit of a hack, where we basically hardcode in rootwrap that it's being
>> transitioned to privsep. That's fine, but only if we get rid of rootwrap
>> soon. So only if we have a plan for (4) anyway. Option (0) is a bit of a
>> hard sell for upgrade procedures -- if we need to take a hit in that area,
>> let's do (4) directly...
>>
>> In summary, I think the choice is between (1)+(4) and doing (4) directly.
>> How doable is (4) in the timeframe we have ? Do we all agree that (4) is the
>> endgame ?
> 
> We've already merged change to privsep to allow nova/cinder/etc to
> initialize the default helper command to use rootwrap:
> 
>   
> https://github.com/openstack/oslo.privsep/commit/9bf606327d156de52c9418d5784cd7f29e243487
> 
> So we just need new release of privsep & add code to nova to initialize
> it and we're sorted.

Actually, I don't think so. Matt ran that test scenario, and we're
missing the rootwrap rule that lets privsep-helper run as root. So we
fail to start the daemon from the unpriv nova compute process post upgrade.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Sean Dague]

On 06/24/2016 05:12 AM, Thierry Carrez wrote:
> Angus Lees wrote:
>> [...]
>> None of these are great, but:
>>
>> Possibility 1:  Backdoor rootwrap
>>
>> However if we assume rootwrap already exists then we _could_ rollout a
>> new version of oslo.rootwrap that contains a backdoor that allows
>> privsep-helper to be run as root for any context, without the need to
>> install a new rootwrap filter.
>>
>> Disclaimers:
>>
>> - It wouldn't work for virtualenvs, because the "privsep-helper"
>> executable won't be in sudo's usual PATH.
>>
>> - Retro-fitting something like that to rootwrap feels like it's skirting
>> close to some sort of ethical contract we've made with admins regarding
>> rootwrap's featureset.  Not saying we shouldn't do it, just that we
>> should think about how an operator is going to feel about that.
>>
>>
>> Possibility 2: Wider rootwrap filter
>>
>> In the past, I've been proposing rootwrap filters that match only
>> specific privsep "privileged contexts" by name.  On further reflection,
>> if we're assuming the existing python modules installed into root's
>> python path are already trustworthy (and we _are_ assuming that), then
>> it might also be reasonable to trust *any* privsep entrypoint declared
>> within that module path.  This gives a larger attack surface to think
>> about (particularly if python libraries including privsep decorators
>> were installed for some reason other than providing privsep entry
>> points), but there's no reason why this is _necessarily_ an issue.
>>
>> This allows us to get to a single rootwrap filter per-project (or
>> rather, "per-rootwrap") since projects use separate rootwrap config
>> directories - so we would still have to do a thing once per project.
>>
>>
>> Possibility 3: Skip rootwrap, use just sudo
>>
>> sudoers isn't very expressive - but we could install a new rootwrap-like
>> wrapper into sudoers once system-wide, which includes some sort of logic
>> to start privsep-helpers.  This could be as simple as a small shell
>> script.  The advantage this has over rootwrap is that it would contain
>> some sort of system-wide config, rather than per-project.
>>
>> Downsides
>>
>> - Would still need to be installed once system-wide.
>>
>> - Would need to be configured per-virtualenv, since otherwise we have no
>> way to know which virtualenvs should be given root powers.
>>
>>
>> Possibility 4: Run as root initially
>>
>> Another option would be to follow the usual Unix daemon model: Start the
>> process with all required privileges, and avoid sudo/rootwrap entirely.
>>
>> In this version, we take a once-off hit to tell everyone to start
>> running their OpenStack agents as root (probably from init/systemd), and
>> right at the top of main() we fork() the privsep-helper and then drop to
>> a regular uid.  No sudo or rootwrap ever (although the unprivileged code
>> could continue to use it while we clean up all the legacy code).
>>
>> A glorious future, but still a big per-project deployment change that
>> has to be managed somehow.
> 
> I'm adding Possibility (0): change Grenade so that rootwrap filters from
> N+1 are put in place before you upgrade.

If you do that as general course what you are saying is that every
installer and install process includes overwriting all of rootwrap
before every upgrade. Keep in mind we do upstream upgrade as offline,
which means that we've fully shut down the cloud. This would remove the
testing requirement that rootwrap configs were even compatible between N
and N+1. And you think this is theoretical, you should see the patches
I've gotten over the years to grenade because people didn't see an issue
with that at all. :)

I do get that people don't like the constraints we've self imposed, but
we've done that for very good reasons. The #1 complaint from operators,
for ever, has been the pain and danger of upgrading. That's why we are
still trademarking new Juno clouds. When you upgrade Apache, you don't
have to change your config files.

As an asside, one of our major inhibitors to smooth upgrade are these
giant conf trees which exist for users to modify in /etc, but not in a
really clear override way: policy, paste, rootwrap. Policy is being
solved right now by moving defaults to code (in Nova, but all the
infrastructure is in oslo.policy, and should be a quick transition for
other projects once we've proved it out) and keeping etc as only
overrides. Morgan and I talked about the same kind of approach with
paste, he's going to sniff that one out. Once we get to privsep, I think
we have it solved for rootwrap. But that transition is hard. Because the
existing system was designed without thinking about the upgrade
implications.

> No perfect answer here... I'm hesitating between (0), (1) and (4). (4)
> is IMHO the right solution, but it's a larger change for downstream. (1)
> is a bit of a hack, where we basically hardcode in rootwrap that it's
> being transitioned to privsep. That's fine, but only if we get rid of
> 

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Daniel P. Berrange]

On Fri, Jun 24, 2016 at 11:12:27AM +0200, Thierry Carrez wrote:
> Angus Lees wrote:
> > [...]
> > None of these are great, but:
> > 
> > Possibility 1:  Backdoor rootwrap
> > 
> > However if we assume rootwrap already exists then we _could_ rollout a
> > new version of oslo.rootwrap that contains a backdoor that allows
> > privsep-helper to be run as root for any context, without the need to
> > install a new rootwrap filter.
> > 
> > Disclaimers:
> > 
> > - It wouldn't work for virtualenvs, because the "privsep-helper"
> > executable won't be in sudo's usual PATH.
> > 
> > - Retro-fitting something like that to rootwrap feels like it's skirting
> > close to some sort of ethical contract we've made with admins regarding
> > rootwrap's featureset.  Not saying we shouldn't do it, just that we
> > should think about how an operator is going to feel about that.
> > 
> > 
> > Possibility 2: Wider rootwrap filter
> > 
> > In the past, I've been proposing rootwrap filters that match only
> > specific privsep "privileged contexts" by name.  On further reflection,
> > if we're assuming the existing python modules installed into root's
> > python path are already trustworthy (and we _are_ assuming that), then
> > it might also be reasonable to trust *any* privsep entrypoint declared
> > within that module path.  This gives a larger attack surface to think
> > about (particularly if python libraries including privsep decorators
> > were installed for some reason other than providing privsep entry
> > points), but there's no reason why this is _necessarily_ an issue.
> > 
> > This allows us to get to a single rootwrap filter per-project (or
> > rather, "per-rootwrap") since projects use separate rootwrap config
> > directories - so we would still have to do a thing once per project.
> > 
> > 
> > Possibility 3: Skip rootwrap, use just sudo
> > 
> > sudoers isn't very expressive - but we could install a new rootwrap-like
> > wrapper into sudoers once system-wide, which includes some sort of logic
> > to start privsep-helpers.  This could be as simple as a small shell
> > script.  The advantage this has over rootwrap is that it would contain
> > some sort of system-wide config, rather than per-project.
> > 
> > Downsides
> > 
> > - Would still need to be installed once system-wide.
> > 
> > - Would need to be configured per-virtualenv, since otherwise we have no
> > way to know which virtualenvs should be given root powers.
> > 
> > 
> > Possibility 4: Run as root initially
> > 
> > Another option would be to follow the usual Unix daemon model: Start the
> > process with all required privileges, and avoid sudo/rootwrap entirely.
> > 
> > In this version, we take a once-off hit to tell everyone to start
> > running their OpenStack agents as root (probably from init/systemd), and
> > right at the top of main() we fork() the privsep-helper and then drop to
> > a regular uid.  No sudo or rootwrap ever (although the unprivileged code
> > could continue to use it while we clean up all the legacy code).
> > 
> > A glorious future, but still a big per-project deployment change that
> > has to be managed somehow.
> 
> I'm adding Possibility (0): change Grenade so that rootwrap filters from N+1
> are put in place before you upgrade.
> 
> No perfect answer here... I'm hesitating between (0), (1) and (4). (4) is
> IMHO the right solution, but it's a larger change for downstream. (1) is a
> bit of a hack, where we basically hardcode in rootwrap that it's being
> transitioned to privsep. That's fine, but only if we get rid of rootwrap
> soon. So only if we have a plan for (4) anyway. Option (0) is a bit of a
> hard sell for upgrade procedures -- if we need to take a hit in that area,
> let's do (4) directly...
> 
> In summary, I think the choice is between (1)+(4) and doing (4) directly.
> How doable is (4) in the timeframe we have ? Do we all agree that (4) is the
> endgame ?

We've already merged change to privsep to allow nova/cinder/etc to
initialize the default helper command to use rootwrap:

  
https://github.com/openstack/oslo.privsep/commit/9bf606327d156de52c9418d5784cd7f29e243487

So we just need new release of privsep & add code to nova to initialize
it and we're sorted.

We can also revert the changes we made to devstack to update nova.conf
for privsep too.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Thierry Carrez]

Angus Lees wrote:

[...]
None of these are great, but:

Possibility 1:  Backdoor rootwrap

However if we assume rootwrap already exists then we _could_ rollout a
new version of oslo.rootwrap that contains a backdoor that allows
privsep-helper to be run as root for any context, without the need to
install a new rootwrap filter.

Disclaimers:

- It wouldn't work for virtualenvs, because the "privsep-helper"
executable won't be in sudo's usual PATH.

- Retro-fitting something like that to rootwrap feels like it's skirting
close to some sort of ethical contract we've made with admins regarding
rootwrap's featureset.  Not saying we shouldn't do it, just that we
should think about how an operator is going to feel about that.


Possibility 2: Wider rootwrap filter

In the past, I've been proposing rootwrap filters that match only
specific privsep "privileged contexts" by name.  On further reflection,
if we're assuming the existing python modules installed into root's
python path are already trustworthy (and we _are_ assuming that), then
it might also be reasonable to trust *any* privsep entrypoint declared
within that module path.  This gives a larger attack surface to think
about (particularly if python libraries including privsep decorators
were installed for some reason other than providing privsep entry
points), but there's no reason why this is _necessarily_ an issue.

This allows us to get to a single rootwrap filter per-project (or
rather, "per-rootwrap") since projects use separate rootwrap config
directories - so we would still have to do a thing once per project.


Possibility 3: Skip rootwrap, use just sudo

sudoers isn't very expressive - but we could install a new rootwrap-like
wrapper into sudoers once system-wide, which includes some sort of logic
to start privsep-helpers.  This could be as simple as a small shell
script.  The advantage this has over rootwrap is that it would contain
some sort of system-wide config, rather than per-project.

Downsides

- Would still need to be installed once system-wide.

- Would need to be configured per-virtualenv, since otherwise we have no
way to know which virtualenvs should be given root powers.


Possibility 4: Run as root initially

Another option would be to follow the usual Unix daemon model: Start the
process with all required privileges, and avoid sudo/rootwrap entirely.

In this version, we take a once-off hit to tell everyone to start
running their OpenStack agents as root (probably from init/systemd), and
right at the top of main() we fork() the privsep-helper and then drop to
a regular uid.  No sudo or rootwrap ever (although the unprivileged code
could continue to use it while we clean up all the legacy code).

A glorious future, but still a big per-project deployment change that
has to be managed somehow.


I'm adding Possibility (0): change Grenade so that rootwrap filters from 
N+1 are put in place before you upgrade.


No perfect answer here... I'm hesitating between (0), (1) and (4). (4) 
is IMHO the right solution, but it's a larger change for downstream. (1) 
is a bit of a hack, where we basically hardcode in rootwrap that it's 
being transitioned to privsep. That's fine, but only if we get rid of 
rootwrap soon. So only if we have a plan for (4) anyway. Option (0) is a 
bit of a hard sell for upgrade procedures -- if we need to take a hit in 
that area, let's do (4) directly...


In summary, I think the choice is between (1)+(4) and doing (4) 
directly. How doable is (4) in the timeframe we have ? Do we all agree 
that (4) is the endgame ?


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Fri, 24 Jun 2016 at 00:40 Sean Dague  wrote:

> On 06/23/2016 10:07 AM, Sean McGinnis wrote:
> > On Thu, Jun 23, 2016 at 12:19:34AM +, Angus Lees wrote:
> >> So how does rootwrap fit into the "theory of upgrade"?
> >>
> >> The doc talks about deprecating config, but is silent on when new
> required
> >> config (rootwrap filters) should be installed.  By virtue of the way the
> >> grenade code works (install N from scratch, attempt to run code from
> N+1),
> >> we effectively have a policy that any new configs are installed at some
> >> nebulous time *after* the N+1 code is deployed.  In particular, this
> means
> >> a new rootwrap filter needs to be merged a whole release before that
> >> rootwrap entry can be used - and anything else is treated as an
> "exception"
> >> (see for example the nova from-* scripts which have basically updated
> >> rootwrap for each release).
> >>
> >> --
> >>
> >> Stepping back, I feel like an "expand-contract" style upgrade process
> >> involving rootwrap should look something like
> >> 0. Update rootwrap to be the union of N and N+1 rootwrap filters,
> >> 1. Rolling update from N to N+1 code,
> >> 2. Remove N-only rootwrap entries.
> >>
> >> We could make that a bit easier for deployers by having a sensible
> >> deprecation policy that requires our own rootwrap filters for each
> release
> >> are already the union of this release and the last (which indeed is
> already
> >> our policy aiui), and then it would just look like:
> >> 0. Install rootwrap filters from N+1
> >> 1. Rolling update to N+1 code
> >
> > I think effectively this is what we've ended up with in the past.
> >
> > We've had this issue for some time. There have been several releases
> > where either Cinder drivers or os-brick changes have needed to add
> > rootwrap changes. Theoretically we _should_ have hit these problems long
> > ago.
> >
> > I think the only reason it hasn't come up before is that these changes
> > are usually for vendor storage backends. So they never got hit in
> > grenade tests since those use LVM. We have third party CI, but grenade
> > tests are not a part of that.
> >
> > The switch to privsep now has really highlighted this gap. I think we
> > need to make this implied constraint clear and have it documented. To
> > upgrade we will need to make sure the rootwrap filters are in place
> > _before_ we perform any upgrades.
>
> Are we going to have to do this for every service individually as it
> moves to privsep? Or is there a way we can do it common once, take the
> hit, and everyone moves forward?
>
> For instance, can we get oslo.rootwrap to make an exception, in code,
> for privsep-helper? Thereby not having to touch a config file in etc to
> roll forward.
>

Obviously, if we had nothing already in place then: no, there would be
nothing that our python code could do that would suddenly allow it to run
commands as root without some additional (sudo or similar) config by the
sysadmin (good!).

None of these are great, but:

Possibility 1:  Backdoor rootwrap

However if we assume rootwrap already exists then we _could_ rollout a new
version of oslo.rootwrap that contains a backdoor that allows
privsep-helper to be run as root for any context, without the need to
install a new rootwrap filter.

Disclaimers:

- It wouldn't work for virtualenvs, because the "privsep-helper" executable
won't be in sudo's usual PATH.

- Retro-fitting something like that to rootwrap feels like it's skirting
close to some sort of ethical contract we've made with admins regarding
rootwrap's featureset.  Not saying we shouldn't do it, just that we should
think about how an operator is going to feel about that.


Possibility 2: Wider rootwrap filter

In the past, I've been proposing rootwrap filters that match only specific
privsep "privileged contexts" by name.  On further reflection, if we're
assuming the existing python modules installed into root's python path are
already trustworthy (and we _are_ assuming that), then it might also be
reasonable to trust *any* privsep entrypoint declared within that module
path.  This gives a larger attack surface to think about (particularly if
python libraries including privsep decorators were installed for some
reason other than providing privsep entry points), but there's no reason
why this is _necessarily_ an issue.

This allows us to get to a single rootwrap filter per-project (or rather,
"per-rootwrap") since projects use separate rootwrap config directories -
so we would still have to do a thing once per project.


Possibility 3: Skip rootwrap, use just sudo

sudoers isn't very expressive - but we could install a new rootwrap-like
wrapper into sudoers once system-wide, which includes some sort of logic to
start privsep-helpers.  This could be as simple as a small shell script.
The advantage this has over rootwrap is that it would contain some sort of
system-wide config, rather than per-project.

Downsides

- Would still need to be installed 

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Sean Dague]

On 06/23/2016 10:07 AM, Sean McGinnis wrote:
> On Thu, Jun 23, 2016 at 12:19:34AM +, Angus Lees wrote:
>> So how does rootwrap fit into the "theory of upgrade"?
>>
>> The doc talks about deprecating config, but is silent on when new required
>> config (rootwrap filters) should be installed.  By virtue of the way the
>> grenade code works (install N from scratch, attempt to run code from N+1),
>> we effectively have a policy that any new configs are installed at some
>> nebulous time *after* the N+1 code is deployed.  In particular, this means
>> a new rootwrap filter needs to be merged a whole release before that
>> rootwrap entry can be used - and anything else is treated as an "exception"
>> (see for example the nova from-* scripts which have basically updated
>> rootwrap for each release).
>>
>> --
>>
>> Stepping back, I feel like an "expand-contract" style upgrade process
>> involving rootwrap should look something like
>> 0. Update rootwrap to be the union of N and N+1 rootwrap filters,
>> 1. Rolling update from N to N+1 code,
>> 2. Remove N-only rootwrap entries.
>>
>> We could make that a bit easier for deployers by having a sensible
>> deprecation policy that requires our own rootwrap filters for each release
>> are already the union of this release and the last (which indeed is already
>> our policy aiui), and then it would just look like:
>> 0. Install rootwrap filters from N+1
>> 1. Rolling update to N+1 code
> 
> I think effectively this is what we've ended up with in the past.
> 
> We've had this issue for some time. There have been several releases
> where either Cinder drivers or os-brick changes have needed to add
> rootwrap changes. Theoretically we _should_ have hit these problems long
> ago.
> 
> I think the only reason it hasn't come up before is that these changes
> are usually for vendor storage backends. So they never got hit in
> grenade tests since those use LVM. We have third party CI, but grenade
> tests are not a part of that.
> 
> The switch to privsep now has really highlighted this gap. I think we
> need to make this implied constraint clear and have it documented. To
> upgrade we will need to make sure the rootwrap filters are in place
> _before_ we perform any upgrades.

Are we going to have to do this for every service individually as it
moves to privsep? Or is there a way we can do it common once, take the
hit, and everyone moves forward?

For instance, can we get oslo.rootwrap to make an exception, in code,
for privsep-helper? Thereby not having to touch a config file in etc to
roll forward.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Sean McGinnis]

On Thu, Jun 23, 2016 at 12:19:34AM +, Angus Lees wrote:
> So how does rootwrap fit into the "theory of upgrade"?
> 
> The doc talks about deprecating config, but is silent on when new required
> config (rootwrap filters) should be installed.  By virtue of the way the
> grenade code works (install N from scratch, attempt to run code from N+1),
> we effectively have a policy that any new configs are installed at some
> nebulous time *after* the N+1 code is deployed.  In particular, this means
> a new rootwrap filter needs to be merged a whole release before that
> rootwrap entry can be used - and anything else is treated as an "exception"
> (see for example the nova from-* scripts which have basically updated
> rootwrap for each release).
> 
> --
> 
> Stepping back, I feel like an "expand-contract" style upgrade process
> involving rootwrap should look something like
> 0. Update rootwrap to be the union of N and N+1 rootwrap filters,
> 1. Rolling update from N to N+1 code,
> 2. Remove N-only rootwrap entries.
> 
> We could make that a bit easier for deployers by having a sensible
> deprecation policy that requires our own rootwrap filters for each release
> are already the union of this release and the last (which indeed is already
> our policy aiui), and then it would just look like:
> 0. Install rootwrap filters from N+1
> 1. Rolling update to N+1 code

I think effectively this is what we've ended up with in the past.

We've had this issue for some time. There have been several releases
where either Cinder drivers or os-brick changes have needed to add
rootwrap changes. Theoretically we _should_ have hit these problems long
ago.

I think the only reason it hasn't come up before is that these changes
are usually for vendor storage backends. So they never got hit in
grenade tests since those use LVM. We have third party CI, but grenade
tests are not a part of that.

The switch to privsep now has really highlighted this gap. I think we
need to make this implied constraint clear and have it documented. To
upgrade we will need to make sure the rootwrap filters are in place
_before_ we perform any upgrades.

> 
> --
> 
> So: We obviously need to update rootwrap filters at *some* point, and we
> should actually decide when that is.
> 
> We can stick with the current de-facto "config after code" grenade policy
> where we use the rootwrap filters from N for code from N+1, but this
> implies a 6-month lag on any new rootwrap-using code.  I hereby propose we
> choose a "config before code" where we use the rootwrap filters for N+1 for
> code from N+1.  This implies that upgrading the rootwrap filters is a
> necessary precondition for a code upgrade.

+1

> 
> In practice (for grenade) this just means copying any rootwrap filters from
> the new branch into place before attempting code upgrade.  I presume there
> would also be a corresponding ops docs impact - but I haven't seen our
> current published upgrade procedure.

I think we definitely should have this documented somewhere. Not that
most folks will read that documentation, but at least we have it out
there.

> 
> Thoughts?
> 
>  - Gus

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

So how does rootwrap fit into the "theory of upgrade"?

The doc talks about deprecating config, but is silent on when new required
config (rootwrap filters) should be installed.  By virtue of the way the
grenade code works (install N from scratch, attempt to run code from N+1),
we effectively have a policy that any new configs are installed at some
nebulous time *after* the N+1 code is deployed.  In particular, this means
a new rootwrap filter needs to be merged a whole release before that
rootwrap entry can be used - and anything else is treated as an "exception"
(see for example the nova from-* scripts which have basically updated
rootwrap for each release).

--

Stepping back, I feel like an "expand-contract" style upgrade process
involving rootwrap should look something like
0. Update rootwrap to be the union of N and N+1 rootwrap filters,
1. Rolling update from N to N+1 code,
2. Remove N-only rootwrap entries.

We could make that a bit easier for deployers by having a sensible
deprecation policy that requires our own rootwrap filters for each release
are already the union of this release and the last (which indeed is already
our policy aiui), and then it would just look like:
0. Install rootwrap filters from N+1
1. Rolling update to N+1 code

--

So: We obviously need to update rootwrap filters at *some* point, and we
should actually decide when that is.

We can stick with the current de-facto "config after code" grenade policy
where we use the rootwrap filters from N for code from N+1, but this
implies a 6-month lag on any new rootwrap-using code.  I hereby propose we
choose a "config before code" where we use the rootwrap filters for N+1 for
code from N+1.  This implies that upgrading the rootwrap filters is a
necessary precondition for a code upgrade.

In practice (for grenade) this just means copying any rootwrap filters from
the new branch into place before attempting code upgrade.  I presume there
would also be a corresponding ops docs impact - but I haven't seen our
current published upgrade procedure.

Thoughts?

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev