Re: [openstack-dev] [cinder] [nova] os-brick privsep failures and an upgrade strategy?

[Angus Lees]

On Thu, 21 Jul 2016 at 09:27 Sean Dague  wrote:

> On 07/12/2016 06:25 AM, Matt Riedemann wrote:
> 
> > We probably aren't doing anything while Sean Dague is on vacation. He's
> > back next week and we have the nova/cinder meetups, so I'm planning on
> > talking about the grenade issue in person and hopefully we'll have a
> > plan by the end of next week to move forward.
>
> After some discussions at the Nova midcycle we threw together an
> approach where we just always allow privsep-helper from oslo.rootwrap.
>
> https://review.openstack.org/344450


Were these discussions captured anywhere?  I thought we'd discussed
alternatives on os-dev, reached a conclusion, implemented the changes(*),
and verified the results all a month ago - and that we were just awaiting
nova approval.  So I'm surprised to see this sudden change in direction...

(*) Changes:
https://review.openstack.org/#/c/329769/
https://review.openstack.org/#/c/332610/
mriedem's verification: https://review.openstack.org/#/c/331885/

 - Gus

We did a sniff test of this, and it worked to roll over the upgrade
> boundary, without an etc change, and work with osbrick 1.4.0 (currently
> blacklisted because of the upgrade issue). While I realize it wasn't the
> favorite approach by many it works. It's 3 lines of functional change.
> If we land this, release, and bump the minimum, we've got the upgrade
> issue solved in this cycle.
>
> Please take a look and see if we can agree to this path forward.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> --
> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
> filtering.http://www.mailguard.com.au/mg
> Click here to report this message as spam:
> https://console.mailguard.com.au/ras/1OSGOh3pqW/kb4I7l49SLBdqHGpZpoHi/0.82
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Thu, 7 Jul 2016 at 03:06 Matthew Treinish  wrote:

> On Wed, Jul 06, 2016 at 11:41:56AM -0500, Matt Riedemann wrote:
> > I just wonder how many deployments are actually relying on this, since as
> > noted elsewhere in this thread we don't really enforce this for all
> things,
> > only what happens to get tested in our CI system, e.g. the virtuozzo
> > rootwrap filters that don't have grenade testing.
>
> Sure, our testing coverage here is far from perfect, that's never been in
> dispute. It's always been best effort (which there has been limited in this
> space) like I'm not aware of anything doing any upgrade testing with
> virtuozzo enabled, or any of the other random ephemeral storage backends,
> **cough** ceph **cough**.  But, as I said before just because we don't
> catch all
> the issues isn't a reason to throw everything out the window.
>

So now we have identified some other examples recently added to the
codebase, that where not noticed by grenade for one reason or another.

Do we:
A) revert+postpone the virtuozzo changes until the next release?
B) add a releasenote saying you need to update the rootwrap filter first?

(Yes, this is a test)

It's boring, but not that hard to manually diff filters between releases -
I can do an audit if we'd like to build a list of other such changes.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Sat, 2 Jul 2016 at 01:02 Matt Riedemann <mrie...@linux.vnet.ibm.com>
wrote:

> On 6/28/2016 4:56 PM, Sean Dague wrote:
> > On 06/28/2016 01:46 AM, Angus Lees wrote:
> >> Ok, thanks for the in-depth explanation.
> >>
> >> My take away is that we need to file any rootwrap updates as exceptions
> >> for now (so releasenotes and grenade scripts).
> >
> > That is definitely the fall back if there is no better idea. However, we
> > should try really hard to figure out if there is a non manual way
> > through this. Even if that means some compat code that we keep for a
> > release to just bridge the gap.
> >
> > -Sean
> >
>
> Walter had this for os-brick:
>
> https://review.openstack.org/#/c/329586/
>
> That would fallback to rootwrap if privsep doesn't work / not available.
> That could be a workaround for upgrading with os-brick for Newton, with
> a big fat warning logged if we use it, and then drop it in Ocata and
> require privsep.
>

Yes, this is basically a version of "submit the rootwrap filter, then wait
6 months before submitting the code that needs it".   If we don't wish to
use the exception mechanism (or adjust the policy to upgrade conf before
code as I described earlier), then we can certainly do this.  Rather than
log a big fat warning if we use privsep, we may as well just revert the
privsep change for os-brick and then resubmit it next cycle.

This thread topic isn't actually about privsep however, although a
migration to privsep will mostly mitigate this in the future which is
perhaps why it is causing topic collisions for everyone.

I see there are already a few other additions to the rootwrap filters in
nova/cinder (the comments suggest (nova) libvirt/imagebackend.py, (cinder)
remotefs.py, and (both) vzstorage.py).  The various privsep-only
suggestions about fallback strategies don't help in these other examples.  Any
corresponding code changes that rely on these new filters will also need to
be reverted and resubmitted during next cycle - or do what usually happens
and slip under the radar as they are not exercised by grenade.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

Ok, thanks for the in-depth explanation.

My take away is that we need to file any rootwrap updates as exceptions for
now (so releasenotes and grenade scripts).

 - Gus

On Mon, 27 Jun 2016 at 21:25 Sean Dague <s...@dague.net> wrote:

> On 06/26/2016 10:02 PM, Angus Lees wrote:
> > On Fri, 24 Jun 2016 at 20:48 Sean Dague <s...@dague.net
> > <mailto:s...@dague.net>> wrote:
> >
> > On 06/24/2016 05:12 AM, Thierry Carrez wrote:
> > > I'm adding Possibility (0): change Grenade so that rootwrap
> > filters from
> > > N+1 are put in place before you upgrade.
> >
> > If you do that as general course what you are saying is that every
> > installer and install process includes overwriting all of rootwrap
> > before every upgrade. Keep in mind we do upstream upgrade as offline,
> > which means that we've fully shut down the cloud. This would remove
> the
> > testing requirement that rootwrap configs were even compatible
> between N
> > and N+1. And you think this is theoretical, you should see the
> patches
> > I've gotten over the years to grenade because people didn't see an
> issue
> > with that at all. :)
> >
> > I do get that people don't like the constraints we've self imposed,
> but
> > we've done that for very good reasons. The #1 complaint from
> operators,
> > for ever, has been the pain and danger of upgrading. That's why we
> are
> > still trademarking new Juno clouds. When you upgrade Apache, you
> don't
> > have to change your config files.
> >
> >
> > In case it got lost, I'm 100% on board with making upgrades safe and
> > straightforward, and I understand that grenade is merely a tool to help
> > us test ourselves against our process and not an enemy to be worked
> > around.  I'm an ops guy proud and true and hate you all for making
> > openstack hard to upgrade in the first place :P
> >
> > Rootwrap configs need to be updated in line with new rootwrap-using code
> > - that's just the way the rootwrap security mechanism works, since the
> > security "trust" flows from the root-installed rootwrap config files.
> >
> > I would like to clarify what our self-imposed upgrade rules are so that
> > I can design code within those constraints, and no-one is answering my
> > question so I'm just getting more confused as this thread progresses...
> >
> > ***
> > What are we trying to impose on ourselves for upgrades for the present
> > and near future (ie: while rootwrap is still a thing)?
> > ***
> >
> > A. Sean says above that we do "offline" upgrades, by which I _think_ he
> > means a host-by-host (or even global?) "turn everything (on the same
> > host/container) off, upgrade all files on disk for that host/container,
> > turn it all back on again".  If this is the model, then we can trivially
> > update rootwrap files during the "upgrade" step, and I don't see any
> > reason why we need to discuss anything further - except how we implement
> > this in grenade.
> >
> > B. We need to support a mix of old + new code running on the same
> > host/container, running against the same config files (presumably
> > because we're updating service-by-service, or want to minimise the
> > service-unavailability during upgrades to literally just a process
> > restart).  So we need to think about how and when we stage config vs
> > code updates, and make sure that any overlap is appropriately allowed
> > for (expand-contract, etc).
> >
> > C. We would like to just never upgrade rootwrap (or other config) files
> > ever again (implying a freeze in as_root command lines, effective ~a
> > year ago).  Any config update is an exception dealt with through
> > case-by-case process and release notes.
> >
> >
> > I feel like the grenade check currently implements (B) with a 6 month
> > lead time on config changes, but the "theory of upgrade" doc and our
> > verbal policy might actually be (C) (see this thread, eg), and Sean
> > above introduced the phrase "offline" which threw me completely into
> > thinking maybe we're aiming for (A).  You can see why I'm looking for
> > clarification  ;)
>
> Ok, there is theory of what we are striving for, and there is what is
> viable to test consistently.
>
> The thing we are shooting for is making the code Continuously
> Deployable. Which means the upgrade process should be "pip install -U
> $foo && $foo-manage db-sync" on the API surfaces and "pi

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Mon, 27 Jun 2016 at 12:59 Tony Breeds <t...@bakeyournoodle.com> wrote:

> On Mon, Jun 27, 2016 at 02:02:35AM +0000, Angus Lees wrote:
>
> > ***
> > What are we trying to impose on ourselves for upgrades for the present
> and
> > near future (ie: while rootwrap is still a thing)?
> > ***
> >
> > A. Sean says above that we do "offline" upgrades, by which I _think_ he
> > means a host-by-host (or even global?) "turn everything (on the same
> > host/container) off, upgrade all files on disk for that host/container,
> > turn it all back on again".  If this is the model, then we can trivially
> > update rootwrap files during the "upgrade" step, and I don't see any
> reason
> > why we need to discuss anything further - except how we implement this in
> > grenade.
>
> Yes this one.  You must upgrade everything in the host/container to the
> same
> release at the same time.  For example we do NOT support running
> nova@liberty
> and cinder@mitaka.
>

Ack.  Ok .. so what's the additional difficulty around config files?  It
sounds like we can replace all the config files with something completely
different during the update phase, if we wanted to do so.  Indeed, it
sounds like there isn't even a need to manage a deprecation period for
config files since there will never be mismatched code+config (good, means
fewer poorly tested legacy combinations in code).

Specifically, it seems grenade in both doc and code currently describes
something quite a bit stricter than this.  I think what we want is more
like "use devstack to deploy old; run/test; **use devstack to deploy new**
but pointing at existing DB/state_path from old; run/test, interact with
things created with old, etc".

A solution to our immediate rootwrap issue would be to just copy over the
rootwrap configs from 'new' during upgrade always, and this shouldn't even
be controversial.  I can't read body language over email, so .. is everyone
ok with this?  Why is this not what everyone was jumping to suggest before
now?

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Fri, 24 Jun 2016 at 19:13 Thierry Carrez  wrote:

> In summary, I think the choice is between (1)+(4) and doing (4)
> directly. How doable is (4) in the timeframe we have ? Do we all agree
> that (4) is the endgame ?
>

I don't make predictions about development timelines within OpenStack.

Yes, I think (4) should be our endgame, and is certainly the most
future-proof approach.  I haven't thought through in detail nor spoken with
the ops community at all about how disruptive such a transition might be.

Thinking just now, I suspect any such transition needs a minimum of one
(probably 2) release cycles.  Theoretically:

- introduce top-of-main code that drops to unpriv uid if root (could
theoretically be done in current N cycle)
- change required deployment method during upgrade to release N+1
- In N+1 cycle require starting as root (or similar)
- In N+1 change top-of-main code to fork-privsep-then-drop instead of
drop-then-later-use-sudo
- So by the time N+2 is deployed we're done.

If we want to give more notice and manage a longer deprecation cycle before
requiring a start-as-root deployment (and I think we want both these
things), then expand as appropriate.

I suspect the change to deployment method will be easier to communicate and
less unpleasant if we do "every" openstack service in the one
tear-the-bandaid-off cycle rather than drag this out over years.  This of
course requires substantial cross-project consensus, coordination, and
timing.   Once the laughing subsides and we assume that isn't going to
happen, then we will need something machine-readable and in-tree that
correctly communicates the desired per-service approach at that point in
time - perhaps we could publish (eg) systemd unit files for our services
and adjust those to User=root/not-root as services switch to the new method
over time.  This requires communicating when in the upgrade cycle systemd
unit files (or the implied equivalents) should be updated, so goto 10 ;)

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Fri, 24 Jun 2016 at 20:48 Sean Dague  wrote:

> On 06/24/2016 05:12 AM, Thierry Carrez wrote:
> > I'm adding Possibility (0): change Grenade so that rootwrap filters from
> > N+1 are put in place before you upgrade.
>
> If you do that as general course what you are saying is that every
> installer and install process includes overwriting all of rootwrap
> before every upgrade. Keep in mind we do upstream upgrade as offline,
> which means that we've fully shut down the cloud. This would remove the
> testing requirement that rootwrap configs were even compatible between N
> and N+1. And you think this is theoretical, you should see the patches
> I've gotten over the years to grenade because people didn't see an issue
> with that at all. :)
>
> I do get that people don't like the constraints we've self imposed, but
> we've done that for very good reasons. The #1 complaint from operators,
> for ever, has been the pain and danger of upgrading. That's why we are
> still trademarking new Juno clouds. When you upgrade Apache, you don't
> have to change your config files.
>

In case it got lost, I'm 100% on board with making upgrades safe and
straightforward, and I understand that grenade is merely a tool to help us
test ourselves against our process and not an enemy to be worked around.
I'm an ops guy proud and true and hate you all for making openstack hard to
upgrade in the first place :P

Rootwrap configs need to be updated in line with new rootwrap-using code -
that's just the way the rootwrap security mechanism works, since the
security "trust" flows from the root-installed rootwrap config files.

I would like to clarify what our self-imposed upgrade rules are so that I
can design code within those constraints, and no-one is answering my
question so I'm just getting more confused as this thread progresses...

***
What are we trying to impose on ourselves for upgrades for the present and
near future (ie: while rootwrap is still a thing)?
***

A. Sean says above that we do "offline" upgrades, by which I _think_ he
means a host-by-host (or even global?) "turn everything (on the same
host/container) off, upgrade all files on disk for that host/container,
turn it all back on again".  If this is the model, then we can trivially
update rootwrap files during the "upgrade" step, and I don't see any reason
why we need to discuss anything further - except how we implement this in
grenade.

B. We need to support a mix of old + new code running on the same
host/container, running against the same config files (presumably because
we're updating service-by-service, or want to minimise the
service-unavailability during upgrades to literally just a process
restart).  So we need to think about how and when we stage config vs code
updates, and make sure that any overlap is appropriately allowed for
(expand-contract, etc).

C. We would like to just never upgrade rootwrap (or other config) files
ever again (implying a freeze in as_root command lines, effective ~a year
ago).  Any config update is an exception dealt with through case-by-case
process and release notes.


I feel like the grenade check currently implements (B) with a 6 month lead
time on config changes, but the "theory of upgrade" doc and our verbal
policy might actually be (C) (see this thread, eg), and Sean above
introduced the phrase "offline" which threw me completely into thinking
maybe we're aiming for (A).  You can see why I'm looking for clarification
 ;)

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Fri, 24 Jun 2016 at 21:04 Sean Dague  wrote:

> On 06/24/2016 05:19 AM, Daniel P. Berrange wrote:
> > On Fri, Jun 24, 2016 at 11:12:27AM +0200, Thierry Carrez wrote:
> >> No perfect answer here... I'm hesitating between (0), (1) and (4). (4)
> is
> >> IMHO the right solution, but it's a larger change for downstream. (1)
> is a
> >> bit of a hack, where we basically hardcode in rootwrap that it's being
> >> transitioned to privsep. That's fine, but only if we get rid of rootwrap
> >> soon. So only if we have a plan for (4) anyway. Option (0) is a bit of a
> >> hard sell for upgrade procedures -- if we need to take a hit in that
> area,
> >> let's do (4) directly...
> >>
> >> In summary, I think the choice is between (1)+(4) and doing (4)
> directly.
> >> How doable is (4) in the timeframe we have ? Do we all agree that (4)
> is the
> >> endgame ?
> >
> > We've already merged change to privsep to allow nova/cinder/etc to
> > initialize the default helper command to use rootwrap:
> >
> >
> https://github.com/openstack/oslo.privsep/commit/9bf606327d156de52c9418d5784cd7f29e243487
> >
> > So we just need new release of privsep & add code to nova to initialize
> > it and we're sorted.
>
> Actually, I don't think so. Matt ran that test scenario, and we're
> missing the rootwrap rule that lets privsep-helper run as root. So we
> fail to start the daemon from the unpriv nova compute process post upgrade.
>

Right, I thought that recent privsep change would address this, but we're
having this conversation because it turns out that simply updating code
only is not sufficient.  I (and I presume all the other members of the
earlier review/email discussion) had just assumed that we already had a
sensible process for making orderly changes to rootwrap files.

As I've since learned however, the current openstack upgrade process
doesn't talk about updating the rootwrap files, ever.  The only reason
we've been able to make reasonable progress since this was instituted is
either through the grenade exception mechanism, or by slipping things
through in drivers that are not tested by grenade.  Hence me opening this
fun can of worms for broader discussion in this thread.  You're welcome :-)

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

On Fri, 24 Jun 2016 at 00:40 Sean Dague <s...@dague.net> wrote:

> On 06/23/2016 10:07 AM, Sean McGinnis wrote:
> > On Thu, Jun 23, 2016 at 12:19:34AM +0000, Angus Lees wrote:
> >> So how does rootwrap fit into the "theory of upgrade"?
> >>
> >> The doc talks about deprecating config, but is silent on when new
> required
> >> config (rootwrap filters) should be installed.  By virtue of the way the
> >> grenade code works (install N from scratch, attempt to run code from
> N+1),
> >> we effectively have a policy that any new configs are installed at some
> >> nebulous time *after* the N+1 code is deployed.  In particular, this
> means
> >> a new rootwrap filter needs to be merged a whole release before that
> >> rootwrap entry can be used - and anything else is treated as an
> "exception"
> >> (see for example the nova from-* scripts which have basically updated
> >> rootwrap for each release).
> >>
> >> --
> >>
> >> Stepping back, I feel like an "expand-contract" style upgrade process
> >> involving rootwrap should look something like
> >> 0. Update rootwrap to be the union of N and N+1 rootwrap filters,
> >> 1. Rolling update from N to N+1 code,
> >> 2. Remove N-only rootwrap entries.
> >>
> >> We could make that a bit easier for deployers by having a sensible
> >> deprecation policy that requires our own rootwrap filters for each
> release
> >> are already the union of this release and the last (which indeed is
> already
> >> our policy aiui), and then it would just look like:
> >> 0. Install rootwrap filters from N+1
> >> 1. Rolling update to N+1 code
> >
> > I think effectively this is what we've ended up with in the past.
> >
> > We've had this issue for some time. There have been several releases
> > where either Cinder drivers or os-brick changes have needed to add
> > rootwrap changes. Theoretically we _should_ have hit these problems long
> > ago.
> >
> > I think the only reason it hasn't come up before is that these changes
> > are usually for vendor storage backends. So they never got hit in
> > grenade tests since those use LVM. We have third party CI, but grenade
> > tests are not a part of that.
> >
> > The switch to privsep now has really highlighted this gap. I think we
> > need to make this implied constraint clear and have it documented. To
> > upgrade we will need to make sure the rootwrap filters are in place
> > _before_ we perform any upgrades.
>
> Are we going to have to do this for every service individually as it
> moves to privsep? Or is there a way we can do it common once, take the
> hit, and everyone moves forward?
>
> For instance, can we get oslo.rootwrap to make an exception, in code,
> for privsep-helper? Thereby not having to touch a config file in etc to
> roll forward.
>

Obviously, if we had nothing already in place then: no, there would be
nothing that our python code could do that would suddenly allow it to run
commands as root without some additional (sudo or similar) config by the
sysadmin (good!).

None of these are great, but:

Possibility 1:  Backdoor rootwrap

However if we assume rootwrap already exists then we _could_ rollout a new
version of oslo.rootwrap that contains a backdoor that allows
privsep-helper to be run as root for any context, without the need to
install a new rootwrap filter.

Disclaimers:

- It wouldn't work for virtualenvs, because the "privsep-helper" executable
won't be in sudo's usual PATH.

- Retro-fitting something like that to rootwrap feels like it's skirting
close to some sort of ethical contract we've made with admins regarding
rootwrap's featureset.  Not saying we shouldn't do it, just that we should
think about how an operator is going to feel about that.


Possibility 2: Wider rootwrap filter

In the past, I've been proposing rootwrap filters that match only specific
privsep "privileged contexts" by name.  On further reflection, if we're
assuming the existing python modules installed into root's python path are
already trustworthy (and we _are_ assuming that), then it might also be
reasonable to trust *any* privsep entrypoint declared within that module
path.  This gives a larger attack surface to think about (particularly if
python libraries including privsep decorators were installed for some
reason other than providing privsep entry points), but there's no reason
why this is _necessarily_ an issue.

This allows us to get to a single rootwrap filter per-project (or rather,
"per-rootwrap") since projects use separate rootwrap config directories -
so we would still have to do a thi

[openstack-dev] [grenade] upgrades vs rootwrap

[Angus Lees]

So how does rootwrap fit into the "theory of upgrade"?

The doc talks about deprecating config, but is silent on when new required
config (rootwrap filters) should be installed.  By virtue of the way the
grenade code works (install N from scratch, attempt to run code from N+1),
we effectively have a policy that any new configs are installed at some
nebulous time *after* the N+1 code is deployed.  In particular, this means
a new rootwrap filter needs to be merged a whole release before that
rootwrap entry can be used - and anything else is treated as an "exception"
(see for example the nova from-* scripts which have basically updated
rootwrap for each release).

--

Stepping back, I feel like an "expand-contract" style upgrade process
involving rootwrap should look something like
0. Update rootwrap to be the union of N and N+1 rootwrap filters,
1. Rolling update from N to N+1 code,
2. Remove N-only rootwrap entries.

We could make that a bit easier for deployers by having a sensible
deprecation policy that requires our own rootwrap filters for each release
are already the union of this release and the last (which indeed is already
our policy aiui), and then it would just look like:
0. Install rootwrap filters from N+1
1. Rolling update to N+1 code

--

So: We obviously need to update rootwrap filters at *some* point, and we
should actually decide when that is.

We can stick with the current de-facto "config after code" grenade policy
where we use the rootwrap filters from N for code from N+1, but this
implies a 6-month lag on any new rootwrap-using code.  I hereby propose we
choose a "config before code" where we use the rootwrap filters for N+1 for
code from N+1.  This implies that upgrading the rootwrap filters is a
necessary precondition for a code upgrade.

In practice (for grenade) this just means copying any rootwrap filters from
the new branch into place before attempting code upgrade.  I presume there
would also be a corresponding ops docs impact - but I haven't seen our
current published upgrade procedure.

Thoughts?

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] [nova] os-brick privsep failures and an upgrade strategy?

[Angus Lees]

On Wed, 22 Jun 2016 at 05:59 Matt Riedemann 
wrote:

> Angus, what should we be looking at from the privsep side for debugging
> this?
>

The line above the screen-n-cpu.txt.gz failure you linked to is:
2016-06-21 16:21:30.994

1840 WARNING oslo.privsep.daemon [-] privsep log:
/usr/local/bin/nova-rootwrap: Unauthorized command: privsep-helper
--config-file /etc/nova/nova.conf --privsep_context
os_brick.privileged.default --privsep_sock_path /tmp/tmpV5w2VC/privsep.sock
(no filter matched)

 .. so nova-rootwrap is rejecting the privsep-helper command line because
no filter matched.  This indicates the nova compute.filters file has not
been updated, or is incorrect.


As was later pointed out by mtreinish, grenade is attempting to run the
newton code against mitaka configs, and this includes using mitaka rootwrap
filters.   Unfortunately, the change to add privsep to nova's rootwrap
filters wasn't approved until the newton cycle (so that all the os-brick
privsep-related changes could be approved together), and so this doesn't
Just Work.

Digging in further, it appears that there *is* a mechanism in grenade to
upgrade rootwrap filters between major releases, but this needs to be
explicitly updated for each project+release and hasn't been for
nova+mitaka->newton.  I'm not sure how this is *meant* to work, since the
grenade "theory of upgrade" doesn't mention when configs should be updated
- the only mechanism provided is an "exception ... used sparingly."

Anyway, I added an upgrade step for nova mitaka->newton that updates
rootwrap filters appropriately(*).  Again, I'm not sure what this
communicates to deployers compared to cinder (which *did* have the updated
rootwrap filter merged in mitaka, but of course that update still needs to
be installed at some point).
(*) https://review.openstack.org/#/c/332610

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] [nova] os-brick privsep failures and an upgrade strategy?

[Angus Lees]

oslo.privsep change: https://review.openstack.org/#/c/329766/
And the nova change that uses it: https://review.openstack.org/#/c/329769

In particular I'm unsure if os-brick/os-vif is even loaded at this point in
nova-compute main().  Does anyone know when that actually happens or shall
I go exploring?

 - Gus

On Wed, 15 Jun 2016 at 11:43 Sean Dague <s...@dague.net> wrote:

> On 06/14/2016 06:11 PM, Angus Lees wrote:
> > Yep (3) is quite possible, and the only reason it doesn't just do this
> > already is because there's no way to find the name of the rootwrap
> > command to use (from any library, privsep or os-brick) - and I was never
> > very happy with the current need to specify a command line in
> > oslo.config purely for this lame reason.
> >
> > As Sean points out, all the others involve some sort of configuration
> > change preceding the code.  I had imagined rollouts would work by
> > pushing out the harmless conf or sudoers change first, but hadn't
> > appreciated the strict change phases imposed by grenade (and ourselves).
> >
> > If all "end-application" devs are happy calling something like (3)
> > before the first privileged operation occurs, then we should be good.  I
> > might even take the opportunity to phrase it as a general privsep.init()
> > function, and then we can use it for any other top-of-main()
> > privilege-setup steps that need to be taken in the future.
>
> That sounds promising. It would be fine to emit a warning if it only was
> using the default, asking people to make a configuration change to make
> it go away. We're totally good with things functioning with warnings
> after transitions, that ops can adjust during their timetable.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> --
> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
> filtering.http://www.mailguard.com.au/mg
> Click here to report this message as spam:
> https://console.mailguard.com.au/ras/1ODUv4oqIN/4x80DVYpDOULTM59jB3mdH/0.82
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] [nova] os-brick privsep failures and an upgrade strategy?

[Angus Lees]

On Tue, 14 Jun 2016 at 23:04 Daniel P. Berrange  wrote:

> On Tue, Jun 14, 2016 at 07:49:54AM -0400, Sean Dague wrote:
>
> [snip]
>

Urgh, thanks for the in-depth analysis :/

> The crux of the problem is that os-brick 1.4 and privsep can't be used
> > without a config file change during the upgrade. Which violates our
> > policy, because it breaks rolling upgrades.
>
> os-vif support is going to face exactly the same problem. We just followed
> os-brick's lead by adding a change to devstack to explicitly set the
> required config options in nova.conf to change privsep to use rootwrap
> instead of plain sudo.
>
> Basically every single user of privsep is likely to face the same
> problem.
>
> > So... we have a few options:
> >
> > 1) make an exception here with release notes, because it's the only way
> > to move forward.
>
> That's quite user hostile I think.
>
> > 2) have some way for os-brick to use either mode for a transition period
> > (depending on whether privsep is configured to work)
>
> I'm not sure that's viable - at least for os-vif we started from
> a clean slate to assume use of privsep, so we won't be able to have
> any optional fallback to non-privsep mode.
>
> > 3) Something else ?
>
> 3) Add an API to oslo.privsep that lets us configure the default
>command to launch the helper. Nova would invoke this on startup
>
>   privsep.set_default_helper("sudo nova-rootwrap ")
>
> 4) Have oslo.privsep install a sudo rule that grants permission
>to run privsep-helper, without needing rootwrap.
>
> 5) Have each user of privsep install a sudo rule to grants
>permission to run privsep-helper with just their specific
>entry point context, without needing rootwrap
>
> Any of 3/4/5 work out of the box, but I'm probably favouring
> option 4, then 5, then 3.
>
>
Yep (3) is quite possible, and the only reason it doesn't just do this
already is because there's no way to find the name of the rootwrap command
to use (from any library, privsep or os-brick) - and I was never very happy
with the current need to specify a command line in oslo.config purely for
this lame reason.

As Sean points out, all the others involve some sort of configuration
change preceding the code.  I had imagined rollouts would work by pushing
out the harmless conf or sudoers change first, but hadn't appreciated the
strict change phases imposed by grenade (and ourselves).

If all "end-application" devs are happy calling something like (3) before
the first privileged operation occurs, then we should be good.  I might
even take the opportunity to phrase it as a general privsep.init()
function, and then we can use it for any other top-of-main()
privilege-setup steps that need to be taken in the future.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Initial oslo.privsep conversion?

[Angus Lees]

One of the challenges with nova (and I'm working from some earlier
conversations, not a first-hand reading of the code) is that we can't
restrict file operations to any particular corner of the filesystem,
because the location of the libvirt data is stored (only) in the database,
and the database is writeable by "unprivileged" nova code.  My
understanding is that it's considered a feature that the libvirt data
directory can be changed at some point, and old instances will continue to
operate in their old location just fine.

There's a number of ways to improve that (restrict to a list of configured
dirs; limit access to dirs owned by a particular (non-root) uid/gid; etc)
but any translation of the nova file-manipulation code to something more
secure will rapidly run up against this "so how do we work out what should
actually be allowed?" policy discussion.  The conclusion of that discussion
will probably require broader nova changes than simply adopting privsep -
just fyi.

 - Gus

On Fri, 10 Jun 2016 at 09:51 Tony Breeds  wrote:

> On Fri, Jun 10, 2016 at 08:24:34AM +1000, Michael Still wrote:
> > On Fri, Jun 10, 2016 at 7:18 AM, Tony Breeds 
> > wrote:
> >
> > > On Wed, Jun 08, 2016 at 08:10:47PM -0500, Matt Riedemann wrote:
> > >
> > > > Agreed, but it's the worked example part that we don't have yet,
> > > > chicken/egg. So we can drop the hammer on all new things until
> someone
> > > does
> > > > it, which sucks, or hope that someone volunteers to work the first
> > > example.
> > >
> > > I'll work with gus to find a good example in nova and have patches up
> > > before
> > > the mid-cycle.  We can discuss next steps then.
> > >
> >
> > Sorry to be a pain, but I'd really like that example to be non-trivial if
> > possible. One of the advantages of privsep is that we can push the logic
> > down closer to the privileged code, instead of just doing something
> "close"
> > and then parsing. I think reinforcing that idea in the sample code is
> > important.
>
> I think *any* change will show that.  I wanted to pick something
> achievable in
> the short timeframe.
>
> The example I'm thinking of is nova/virt/libvirt/utils.py:update_mtime()
>
>  * It will provide a lot of the boiler plate
>  * Show that we can now now replace an exec with pure python code.
>  * Show how you need to retrieve data from a trusted source on the
> priviledged
>side
>  * Migrate testing
>  * Remove an entry from compute.filters
>
> Once that's implace chown() in the same file is probably a quick fix.
>
> Is it super helpful? does it have a measurable impact on performance,
> security?
> The answer is probably "no"
>
> I still think it has value.
>
> Handling qemu-img is probably best done by creating os-qemu (or similar)
> and
> designing from the ground up with provsep in mind.  Glance and Cinder would
> benefit from that also.  That howveer is waaay to big for this cycle.
>
> Yours Tony.
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> --
> Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
> filtering.http://www.mailguard.com.au/mg
> Click here to report this message as spam:
> https://console.mailguard.com.au/ras/1OBOU7cIz8/WIVtx4TniRa9AoBchPxBc/0.62
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] supporting Go

[Angus Lees]

No, it doesn't.  Several applications written in go are already packaged
for Debian (for example).

Indeed the equivalent of "installing from master/pip" (ie: not using distro
packages) is _much_ easier in go, since there is no need for the equivalent
of venvs.  Like all compiled languages, there is a separate compile step
required.

 - Gus

On Tue, 10 May 2016 at 16:44 Tim Bell  wrote:

> From: Rayson Ho 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Tuesday 10 May 2016 at 01:43
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
>
> Subject: Re: [openstack-dev] [tc] supporting Go
>
> Go is a production language used by Google, Dropbox, many many web
> startups, and in fact Fortune 500 companies.
>
> Using a package manager won't buy us anything, and like Clint raised, the
> Linux distros are way too slow in picking up new Go releases. In fact, the
> standard way of installing Rust also does not use a package manager:
>
> https://www.rust-lang.org/downloads.html
>
>
> > I have nothing against golang in particular but I strongly believe that
> mixing 2 languages within a project is always the wrong decision
>
> It would be nice if we only need to write code in one language. But in the
> real world the "nicer" & "easier" languages like Python & Perl are also the
> slower ones. I used to work for an investment bank, and our system was
> developed in Perl, with performance critical part rewritten in C/C++, so
> there really is nothing wrong with mixing languages. (But if you ask me, I
> would strongly prefer Go than C++.)
>
> Rayson
>
>
> I hope that the packaging technologies are considered as part of the TC
> evaluation of a new language. While many alternative approaches are
> available, a language which could not be packaged into RPM or DEB would be
> an additional burden for distro builders and deployers.
>
> Does Go present any additional work compared to Python in this area ?
>
> Tim
>
>
> ==
> Open Grid Scheduler - The Official Open Source Grid Engine
> http://gridscheduler.sourceforge.net/
> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
>
>
>
>
> >
> > If you want to write code in a language that's not Python, go start
> another project. Don't call it OpenStack. If it ends up being a better
> implementation than the reference OpenStack Swift implementation, it will
> win anyways and perhaps Swift will start to look more like the rest of the
> projects in OpenStack with a standardized API and multiple plugable
> implementations.
> >
> > -Ben Swartzlander
> >
> >
> >> Also worth noting, is that go is not a "language runtime" but a compiler
> >> (that happens to statically link in a runtime to the binaries it
> >> produces...).
> >>
> >> The point here though, is that the versions of Python that OpenStack
> >> has traditionally supported have been directly tied to what the Linux
> >> distributions carry in their repositories (case in point, Python 2.6
> >> was dropped from most things as soon as RHEL7 was available with Python
> >> 2.7). With Go, there might need to be similar restrictions.
> >>
> >>
> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> Message protected by MailGuard: e-mail anti-virus, anti-spam and content
> filtering.
> http://www.mailguard.com.au/mg
>
> Report this message as spam
> 
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] supporting Go

[Angus Lees]

On Tue, 10 May 2016 at 05:19 Edward Leafe  wrote:

> On May 9, 2016, at 1:58 PM, Hayes, Graham  wrote:
>
> > This is not a "Go seems cool - lets go try that" decision from us - we
> > know we have a performance problem with one of our components, and we
> > have come to the conclusion that Go (or something like it) is the
> > solution.
>
> Whenever I hear claims that Python is “too slow for X”, I wonder what’s so
> special about X that makes it so much more demanding than, say, serving up
> YouTube. YouTube is written nearly entirely in Python, and has been for
> many, many years. The only parts that aren’t are those that were identified
> as particular performance bottlenecks, such as some parts of the encoding
> process. These were then written in C, which is drop-in compatible with
> Python using ctypes.
>

(Just in case there is any confusion here, no part of serving a youtube
*video* to users involves python.  I believe the *web UI* is/was generated
using python (and then cached aggressively, etc) - but I'm less familiar
with the HTML parts of youtube)

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron] os-vif status report

[Angus Lees]

In case it wasn't already assumed, anyone is welcome to contact me directly
(irc: gus, email, or in Austin) if they have questions or want help with
privsep integration work.  It's early days still and the docs aren't
extensive (ahem).

os-brick privsep change just recently merged (yay), and I have the bulk of
the neutron ip_lib conversion almost ready for review, so os-vif is a good
candidate to focus on for this cycle.

 - Gus

On Thu, 14 Apr 2016 at 01:52 Daniel P. Berrange  wrote:

> I won't be present at the forthcoming Austin summit, so to prepare other
> people in case there are f2f discussions, this is a rough status report
> on the os-vif progress
>
>
> os-vif core
> ---
>
> NB by os-vif core, I mean the python packages in the os_vif/ namespace.
>
> The object model for describing the various different VIF backend
> configurations is defined well enough that it should cover all the
> VIF types currently used by Nova libvirt driver, and probably all
> those needed by other virt drivers. The only exception is that we
> do not have a representation for the vmware 'dvs' VIF type. There's
> no real reason why not, other than the fact that we're concentrating
> on converting the libvirt nova driver first. These are dealt with
> by the os_vif.objects.VIFBase object and its subclasses.
>
>
> We now have an object model for describing client host capabilities.
> This is dealt with by the os_vif.objects.HostInfo versioned object
> and things is used. Currently this object provides details of all
> the os-vif plugins that are installed on the host, and which VIF
> configs objects each supports.  The intent is that the HostInfo
> object is serialized to JSON, and passed to Neutron by Nova when
> creating a port.  This allows Neutron to dynamically decide which
> plugin and which VIF config it wants to use for creating the port.
>
>
> The os_vif.PluginBase class which all plugins must inherit from
> has been enhanced so that plugins can declare configuration
> parameters they wish to support. This allows config options for
> the plugins to be included directly in the nova.conf file in
> a dedicated section per plugin. For example, the linux bridge
> plugin will have its parameters in a "[os_vif_linux_bridge]"
> section in nova.conf.  This lets us setup the deprecations
> correctly, so that when upgrading from older Nova, existing
> settings in nova.conf still apply to the plugins provided
> by os-vif.
>
>
> os-vif reference plugins
> 
>
> Originally the intention was that all plugins would live outside
> of the os-vif package. During discussions at the Nova mid-cycle
> meeting there was a strong preference to have the linux bridge
> and openvswitch plugin implementations be distributed as part of
> the os-vif package directly.
>
> As such we now have 'vif_plug_linux_bridge' and 'vif_plug_ovs'
> python packages as part of the os-vif module. Note that these
> are *not* under the os_vif python namespace, as the intention
> was to keep their code structured as if they were separate,
> so we can easily split them out again in future in we need to.
>
> Both the linux bridge and ovs plugins have now been converted
> over to use oslo.privsep instead of rootwrap for all the places
> where they need to run privileged commands.
>
>
> os-vif extra plugins
> 
>
> Jay has had GIT repositories created to hold the plugins for all
> the other VIF types the libvirt driver needs to support to have
> feature parity with Mitaka and earlier. AFAIK, no one has done
> any work to actually get the code for these working. This is not
> a blocker, since the way the Nova integration is written allows
> us to incrementally convert each VIF type over to use os-vif, so
> we avoid need for a "big bang".
>
>
> os-vif Nova integration
> ---
>
> I have a patch up for review against Nova that converts the libvirt
> driver to use os-vif. It only does the conversion for linux bridge
> and openvswitch, all other vif types fallback to using the current
> code, as mentioned above.  The unit tests for this pass locally,
> but I've not been able to verify its working correctly when run for
> real. There's almost certainly privsep related integration tasks to
> shake out - possibly as little as just installing the rootwrap filter
> needed to allow use of privsep. My focus right now is ironing this
> out so that I can verify linux bridge + ovs work with os-vif.
>
>
> There is a new job defined in the experimental queue that tests that
> can verify Nova against os-vif git master so we can get forwarning
> if something in os-vif will cause Nova to break. This should also
> let us verify that the integration is actually working in Nova CI
> before allowing it to actually merge.
>
>
> os-vif Neutron integration
> --
>
> As mentioned earlier we now have a HostInfo versioned object defined
> in os-vif which Nova will populate. We need to extend 

Re: [openstack-dev] [os-brick][nova][cinder] os-brick/privsep change is done and awaiting your review

[Angus Lees]

On Sat, 19 Mar 2016 at 06:27 Matt Riedemann 
wrote:

> I stared pretty hard at the nova rootwrap filter change today [1] and
> tried to keep that in my head along with the devstack change and the
> changes to os-brick (which depend on the devstack/cinder/nova changes).
> And with reading the privsep-helper command code in privsep itself.
>
> I realize this is a bridge to fix the tightly couple lockstep upgrade
> issue between cinder and nova, but it would be super helpful, at least
> for me, to chart out how that nova rootwrap filter change fits into the
> bigger picture, like what calls what and how, where are things used, etc.
>
> I see devstack passing on the os-brick change so I'm inclined to almost
> blindly approve to just keep moving, but I'd feel bad about that. Would
> it be possible to flow chart this out somewhere?
>

Sorry for all the confusion Matt.  I obviously explained it poorly in my
gerrit reply to you and I presume also in the parts of the oslo spec that
you've read, so I'll try another explanation here:

privsep fundamentally involves two processes - the regular (nova, whatever)
unprivileged code, and a companion Unix process running with some sort of
elevated privileges (different uid/gid, extra Linux capabilities,
whatever).  These two processes talk to each other over a Unix socket in
the obvious way.

*Conceptually* the companion privileged process is a fork from the
unprivileged process - in that the python environment (oslo.config, etc)
tries to be as similar as possible and writing code that runs in the
privileged process looks just like python defined in the original process
but with a particular decorator.

privsep has two modes of setting up this split-process-with-ipc-channel
arrangement:
- One is to use a true fork(), which follows the traditional Unix daemon
model of starting with full privileges (from init or whatever) and then
dropping privileges later - this avoids sudo, is more secure (imo), and is
a whole lot simpler in the privsep code, but requires a change to the way
OpenStack services are deployed, and a function call at the top of main()
before dropping privileges.
- The second is to invoke sudo or sudo+rootwrap from the unprivileged
process to run the "privsep-helper" command that you see in this change.
This requires no changes to the way OpenStack services are deployed, so is
the method I'm recommending atm.  (We may never actually use the fork()
method tbh given how slowly things change in OpenStack.)  It is completely
inconsequential whether this uses sudo or sudo+rootwrap - it just affects
whether you need to add a line to sudoers or rootwrap filters.  I chose
rootwrap filter here because I thought we had greater precedent for that
type of change.

So hopefully that makes the overall picture clear:  We need this nova
rootwrap filter addition so privsep-helper can use sudo+rootwrap to become
root, so it can switch to the right set of elevated privileges, so we can
run the relevant privsep-decorated privileged functions in that privileged
environment.

I also have a concern in there about how the privsep-helper rootwrap
> command in nova is only using the os-brick context. What happens if
> os-vif and nova need to share common rootwrap commands? At the midcycle
> Jay and Daniel said there weren't any coming up soon, but if that
> happens, how do we handle it?


privsep is able to have different "privileged contexts", which can each run
as different uids and with different Linux capabilities.  In practice each
context has its own privileged process, and if we're using the
sudo(+rootwrap) and privsep-helper method, then each context will want its
own line in sudoers or rootwrap filters.
It is expected that most OpenStack services would only have one or maybe
two different contexts, but nova may end up with a few more because it has
its fingers in so many different pies.  So yes, we'll want another entry
similar to this one for os-vif - presumably os-vif will want CAP_NET_ADMIN,
whereas os-brick wants various storage device-related capabilities.


Again, I'm disappointed the relevant section of the privsep spec failed to
explain the above sufficiently - if this conversation helps clarify it for
you, *please* suggest some better wording for the spec.  It seems
(understandably!) no-one wants to approve even the smallest self-contained
privsep-related change without understanding the entire overall process, so
I feel like I've had the above conversation about 10 times now.  It would
greatly improve everyone's productivity if we can get the spec (or some new
doc) to a place where it can become the place where people learn about
privsep, and they don't have to wait for me to reply with poorly summarised
versions.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [os-brick][nova][cinder] os-brick/privsep change is done and awaiting your review

[Angus Lees]

On Fri, 26 Feb 2016 at 01:28 John Garbutt  wrote:

> Agreed with the concerns here, but I thought these are the same we
> raised at the midcycle.
>

Yes, afaik everything is exactly as we discussed and following the
direction we agreed at Nova+CInder mid-cycles.

In hindsight, we probably should also have nominated 2x cores from each of
cinder/nova who were willing to be aware of the situation and review the
resulting change - before actually embarking on the work.  As it is, the
clock is striking noon and the street suddenly contains nothing but
tumbleweeds :-P

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-brick][nova][cinder] os-brick/privsep change is done and awaiting your review

[Angus Lees]

(Reposting my reply to your gerrit comment here as well - conversation will
probably be easier here than in gerrit)

On Thu, 25 Feb 2016 at 00:07 Duncan Thomas <duncan.tho...@gmail.com> wrote:

> My (negative) feedback is on the review - I'm really not sure that this
> matches what I understood the vision of privsep to be at all.
>
> - If this is the vision / the new vision then I think it is majorly
> flawed.
>
> - If it is skipping the vision in the name of expediency of
> implementation, then I think it has gone too far in that direction and
> we've better off holding off one more cycle and putting it in next cycle
> instead with a touch more purity of vision.
>
> Apologies since you've clearly put work into it, and I should have
> provided such feedback earlier.
>

Yes, I agree with your concerns 100% and I'm actually quite glad that you
also understand that this isn't a very satisfying use of privsep.

Better uses of privsep would look more like
https://review.openstack.org/#/c/258252/ - but they're slow to write since
they typically involve quite a bit of refactoring of code in order to move
the trust boundary to a useful point up the call stack.  For os-brick in
particular, I found it quite difficult/risky performing such code refactors
without an easy way to actually test the affected low-level device
manipulation.

At the nova mid-cycle (and again in the nova-cinder VC conversation you
were part of), it was decided that the difficulties in merging the os-brick
rootwrap filters into nova (and I presume cinder) were too urgent to wait
for such a slow os-brick refactoring process.  The conclusion we reached
was that we would do a quick+dirty rootwrap drop-in replacement that
effectively just ran commands as root for Mitaka, and then we would come
back and refactor various codepaths away from that mechanism over time.
This is that first quick+dirty change.
I tried to capture that in the commit description, but evidently did a poor
job - if the above makes it any clearer to you, I'd welcome any suggested
rewording for the commit description.

TL;DR: what this buys us is the ability to use new/different command lines
in os-brick without having to go through a rootwrap filters merge in
downstream projects (and it is also that first baby step towards a
technology that will allow something better in the future).


Please continue to ask questions, since I know very few people have
actually looked at any of privsep nor the os-brick change until now.

 - Gus


> On 24 February 2016 at 14:59, Michał Dulko <michal.du...@intel.com> wrote:
>
>> On 02/24/2016 04:51 AM, Angus Lees wrote:
>> > Re: https://review.openstack.org/#/c/277224
>> >
>> > Most of the various required changes have flushed out by now, and this
>> > change now passes the dsvm-full integration tests(*).
>> >
>> > (*) well, the experimental job anyway.  It still relies on a
>> > merged-but-not-yet-released change in oslo.privsep so gate + 3rd party
>> > won't pass until that happens.
>> >
>> > What?
>> > This change replaces os-brick's use of rootwrap with a quick+dirty
>> > privsep-based drop-in replacement.  Privsep doesn't actually provide
>> > much security isolation when used in this way, but it *does* now run
>> > commands with CAP_SYS_ADMIN (still uid=0/gid=0) rather than full root
>> > superpowers.  The big win from a practical point of view is that it
>> > also means os-brick's rootwrap filters file is essentially deleted and
>> > no longer has to be manually merged with downstream projects.
>> >
>> > Code changes required in nova/cinder:
>> > There is one change each to nova+cinder to add the relevant
>> > privsep-helper command to rootwrap filters, and a devstack change to
>> > add a nova.conf/cinder.conf setting.  That's it - this is otherwise a
>> > backwards/forwards compatible change for nova+cinder.
>> >
>> > Deployment changes required in nova/cinder:
>> > A new "privsep_rootwrap.helper_command" needs to be defined in
>> > nova/cinder.conf (default is something sensible using sudo), and
>> > rootwrap filters or sudoers updated depending on the exact command
>> > chosen.  Be aware that any commands will now be run with CAP_SYS_ADMIN
>> > (only), and if that's insufficient for your hardware/drivers it can be
>> > tweaked with other oslo_config options.
>> >
>> > Risks:
>> > The end-result is still just running the same commands as before, via
>> > a different path - so there's not a lot of adventurousness here.  The
>> > big behavioural change is CAP_SYS_ADMIN, and (as highlighted above)
>> > it's conceivable that the driver

[openstack-dev] [os-brick][nova][cinder] os-brick/privsep change is done and awaiting your review

[Angus Lees]

Re: https://review.openstack.org/#/c/277224

Most of the various required changes have flushed out by now, and this
change now passes the dsvm-full integration tests(*).

(*) well, the experimental job anyway.  It still relies on a
merged-but-not-yet-released change in oslo.privsep so gate + 3rd party
won't pass until that happens.

What?
This change replaces os-brick's use of rootwrap with a quick+dirty
privsep-based drop-in replacement.  Privsep doesn't actually provide much
security isolation when used in this way, but it *does* now run commands
with CAP_SYS_ADMIN (still uid=0/gid=0) rather than full root superpowers.
The big win from a practical point of view is that it also means os-brick's
rootwrap filters file is essentially deleted and no longer has to be
manually merged with downstream projects.

Code changes required in nova/cinder:
There is one change each to nova+cinder to add the relevant privsep-helper
command to rootwrap filters, and a devstack change to add a
nova.conf/cinder.conf setting.  That's it - this is otherwise a
backwards/forwards compatible change for nova+cinder.

Deployment changes required in nova/cinder:
A new "privsep_rootwrap.helper_command" needs to be defined in
nova/cinder.conf (default is something sensible using sudo), and rootwrap
filters or sudoers updated depending on the exact command chosen.  Be aware
that any commands will now be run with CAP_SYS_ADMIN (only), and if that's
insufficient for your hardware/drivers it can be tweaked with other
oslo_config options.

Risks:
The end-result is still just running the same commands as before, via a
different path - so there's not a lot of adventurousness here.  The big
behavioural change is CAP_SYS_ADMIN, and (as highlighted above) it's
conceivable that the driver for some exotic os-brick/cinder hardware out
there wants something more than that.

Work remaining:
- global-requirements change needed (for os-brick) once the latest
oslo.privsep release is made
- cinder/nova/devstack changes need to be merged
- after the above, the os-brick gate integration jobs will be able to pass,
and it can be merged
- If we want to *force* the new version of os-brick, we then need an
appropriate global-requirements os-brick bump
- Documentation, release notes, etc

I'll continue chewing through those remaining work items, but essentially
this is now in your combined hands to prioritise for mitaka as you deem
appropriate.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Angus Lees]

On Tue, 1 Dec 2015 at 00:20 Thierry Carrez  wrote:

> Sean Dague wrote:
> > Right, so to me this seems that privsep just needs a NULL mode, and
> > we're done. If oslo.rootrwap was never supported on windows, I don't
> > think privsep really needs to be in a real way.
>
> +1
>

Agreed.  I need to make it safe to import, and then execute any wrapped
functions in-process instead of across a privsep IPC channel.

This is easy enough, except for verifying whether I've caught all the
import issues.  At this point I think my easiest path is to wait until I
have a nova -> os_brick -> privsep change stack, and then go look at the
nova hyper-v test results.

If someone (hyper-v folks?) with a suitable windows development environment
available locally wants to jump in before then, I would certainly
appreciate the assistance.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Angus Lees]

So I spent a day yesterday trying to get to the point where I could just
run a no-change "tox" successfully on windows.  Unfortunately I gave up
when I realised I still had several days of downloading+building things
ahead of me and clearly I was doing it the hard way :(

Could you point me to the "dev environment how-to" doc for hyper-v work?
I'm thinking of something like
http://docs.openstack.org/developer/nova/development.environment.html#setup
with
simple cut+paste instructions for the totally-windows-clueless like me ;)
 Or perhaps a pre-prepared disk image, if the Microsoft license allows such
a thing.  In particular, the details on
https://wiki.openstack.org/wiki/Hyper-V look out of date (Grizzly, and
several docs.openstack.org links are broken), and the links to things like
https://cloudbase.it/hyper-v-nova-compute-installer-unattended-setup/ look
like deployment rather than development environments (?).

In particular, I think(?) I should be careful *not* to install too much of
a cygwin-style environment, since I think(?) that might no longer be
representative of the environment in which hyper-v is expected to operate.
If I'm wrong, and cygwin/msys is ok, then it looks like there are several
free-software-on-windows "distributions" that would make life a whole lot
simpler (by frankly being less like Windows).  Some guidance from people
who understand the issues here would be appreciated.

 - Gus

On Tue, 24 Nov 2015 at 22:01 Claudiu Belu <cb...@cloudbasesolutions.com>
wrote:

> Hello,
>
> Thanks Dims for raising the concern and Angus for reaching out. :)
>
> Most of the time, python development on Windows is not too far off from
> Linux. But the two systems are quite different, which imply different
> modules (fcntl, pwd, grp modules do not exist in Windows) or different
> implementations of some modules (multiprocessing uses Popen instead of
> os.fork, os module is quite different) or some socket options and signals
> are different in Windows.
>
> 1.a. As I've said earlier, some modules do not exist in Windows. All, or
> at least most standard modules document the fact that they are strictly for
> Linux. [1][2][3]
> b. At the very least, running the unit tests in a Windows environment can
> at least detect simple problems (e.g. imports). Secondly, there is a
> Hyper-V / Windows CI running on some of the OpenStack projects (nova,
> neutron, networking_hyperv, cinder) that can be checked before merging.
>
> 2. This is a bit more complicated question. Well, for functions, you could
> have separate modules for Linux specific functions and Windows specific
> functions. This has been done before: [4] As for object-oriented
> implementations, I'd suggest having the system-specific calls be done in
> private methods, which will be overriden by Windows / Linux subclasses with
> their specific implementations. We've done something like this before, but
> solutions were pretty much straight-forward; it might not be as simple for
> oslo_privsep, since it is very Linux-specific.
>
> 3. Typically, the OpenStack services on Hyper-V / Windows are run with
> users that have enough privileges to do their job. For example, the
> nova-compute service is run with a user that has Hyper-V Admin privileges
> and is not necessarily in the "Administrator" user group. We haven't used
> rootwrap in our usecases, it is disabled by default, plus, oslo.rootwrap
> imports pwd, which doesn't exist in Windows.
>
> [1] https://docs.python.org/2/library/fcntl.html
> [2] https://docs.python.org/2/library/pwd.html
> [3] https://docs.python.org/2/library/grp.html
> [4]
> https://github.com/openstack/neutron/blob/master/neutron/agent/common/utils.py
> [5]
> https://github.com/openstack/oslo.rootwrap/blob/master/oslo_rootwrap/wrapper.py#L19
>
> If you have any further questions, feel free to ask. :)
>
> Best regards,
> Claudiu Belu
>
>
> --
> *From:* Angus Lees [g...@inodes.org]
> *Sent:* Tuesday, November 24, 2015 6:18 AM
> *To:* OpenStack Development Mailing List (not for usage questions);
> Claudiu Belu
> *Subject:* [hyper-v] oslo.privsep vs Windows
>
> Dims has just raised[1] the excellent concern that oslo.privsep will need
> to at least survive on Windows, because hyper-v.  I have no real experience
> coding on windows (I wrote a windows C program once, but I only ever ran it
> under wine ;) and certainly none within an OpenStack/python context:
>
> 1) How can I test whatever I'm working on to see if I have mistakenly
> introduced something Linux-specific?  Surely this is a challenge common
> across every project in the nova/oslo/hyper-v stack.
>
> 2) What predicate should I use to guard the inevitable Linux-specific or
> Windows-specific code bra

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Angus Lees]

So seriously - how do you set up a dev environment for hyper-v?
It doesn't have to be written up all nice and pretty, just a 30s outline in
an email would be really useful :-/

 - Gus

On Thu, 26 Nov 2015 at 13:13 Alessandro Pilotti <
apilo...@cloudbasesolutions.com> wrote:

> Angus,
>
> "I'm afraid this has to be easy for me or I'm just not going to be able to
> sustain the effort required :-/ )"
>
> What a tragedy, I’m s sorry that life is so terribly bitter and
> requires some effort to support people with views different from yours! :-)
>
> Just kidding, we’re here to help, but we obviously won’t do your work.
> Given that Oslo code needs to be portable, I was just trying to give you
> some alternative that doesn’t require you to do useless manual work during
> development.
>
> 1) Unit tests are not the answer, as hopefully they mock out most
> significant underlying OS features. Sure, you will catch the basic issues,
> like importing a module that doesn’t exist on Windows, but you cannot rely
> on them as a proof of portability.
> 2) Integration testing on the other side provide by definition a proof of
> reliability on the target system (in the limits of the tests coverage at
> least) and CI testing needs eventually to be there on Windows as well for
> oslo.. This is obvious not a way to develop, but a
> “guard”, against issues. In other words, regardless of your opinion on
> Windows, the Windows CI needs to be there, why not adding it now for this
> project?
>
> Getting to development practices, when designing features, hopefully
> portability gets evaluated _before_ starting coding, so you should not need
> too many manual runs on every platform before being confident enough to
> send a patch to Gerrit and wait for CI results. And for that you don’t need
> tox or anything fancy, just a Python environment and some very basic OS
> skills.
>
> When in doubt, an option is to drop by on #openstack-hyper-v and ask. But
> take care, you might find open minded people, make sure you're at ease with
> this. ;-)
>
> Alessandro
>
>
> From: Angus Lees <g...@inodes.org>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Thursday 26 November 2015 at 03:23
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>, Claudiu Belu <
> cb...@cloudbasesolutions.com>
>
> Subject: Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows
>
> Thanks for the suggestion, and having a CI bot running on oslo.privsep
> would be a good thing once the basic code works on Windows - I would have
> expected it to be already running on all oslo projects (that the hyper-v
> hypervisor does/might depend on) tbh.
>
> But that's a clumsy way to actually develop something.  I _know_ privsep
> won't work currently on windows (imports pwd/grp for starters), and having
> to add print statements + submit + push-to-gerrit + wait for a CI bot + dig
> through CI bot logs + repeat is a pretty terrible workflow ;-)
>
> (I already have very little empathy for anyone choosing to run Windows
> just to provide yet-another-x86-hypervisor when far easier alternatives are
> available, and I won't donate a disproportionate amount of my time to a
> well-funded company who has historically actively undermined the
> intellectual commons when _I_ have far more rewarding alternatives
> available.  I'm afraid this has to be easy for me or I'm just not going to
> be able to sustain the effort required :-/ )
>
>
>  - Gus
>
> On Thu, 26 Nov 2015 at 11:56 Alessandro Pilotti <
> apilo...@cloudbasesolutions.com> wrote:
>
>> Hi Angus,
>>
>> First thanks for your concern on code portability! It still happens that
>> we have to ask to revert patches on Oslo projects due to some Linux
>> specific code that we discover only when the actual Oslo modules are used
>> by Nova, Neutron, Cinder or other projects. Typically a running Windows CI
>> suddenly turns red on every patch and that’s how we get to find out. We’re
>> working on adding many more projects under Windows CI tests, so at some
>> point all the relevant Oslo ones will be covered and we’ll be able to
>> prevent those situations before the code gets merged.
>>
>> What about preparing some basic integration tests for oslo.privsep that
>> we could add to our Windows CI infrastructure? This would give you peace of
>> mind during development on Linux without needing to test manually all your
>> patches on Windows, knowing that the Windows CI will give an error if the
>> patch contains non portable code (or code that doesn’t

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Angus Lees]

Thanks for the suggestion, and having a CI bot running on oslo.privsep
would be a good thing once the basic code works on Windows - I would have
expected it to be already running on all oslo projects (that the hyper-v
hypervisor does/might depend on) tbh.

But that's a clumsy way to actually develop something.  I _know_ privsep
won't work currently on windows (imports pwd/grp for starters), and having
to add print statements + submit + push-to-gerrit + wait for a CI bot + dig
through CI bot logs + repeat is a pretty terrible workflow ;-)

(I already have very little empathy for anyone choosing to run Windows just
to provide yet-another-x86-hypervisor when far easier alternatives are
available, and I won't donate a disproportionate amount of my time to a
well-funded company who has historically actively undermined the
intellectual commons when _I_ have far more rewarding alternatives
available.  I'm afraid this has to be easy for me or I'm just not going to
be able to sustain the effort required :-/ )

 - Gus

On Thu, 26 Nov 2015 at 11:56 Alessandro Pilotti <
apilo...@cloudbasesolutions.com> wrote:

> Hi Angus,
>
> First thanks for your concern on code portability! It still happens that
> we have to ask to revert patches on Oslo projects due to some Linux
> specific code that we discover only when the actual Oslo modules are used
> by Nova, Neutron, Cinder or other projects. Typically a running Windows CI
> suddenly turns red on every patch and that’s how we get to find out. We’re
> working on adding many more projects under Windows CI tests, so at some
> point all the relevant Oslo ones will be covered and we’ll be able to
> prevent those situations before the code gets merged.
>
> What about preparing some basic integration tests for oslo.privsep that we
> could add to our Windows CI infrastructure? This would give you peace of
> mind during development on Linux without needing to test manually all your
> patches on Windows, knowing that the Windows CI will give an error if the
> patch contains non portable code (or code that doesn’t behave as expected).
>
> Alessandro
>
>
>
> From: Angus Lees <g...@inodes.org>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Thursday 26 November 2015 at 01:56
> To: Claudiu Belu <cb...@cloudbasesolutions.com>, "OpenStack Development
> Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org
> >
> Subject: Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows
>
> So I spent a day yesterday trying to get to the point where I could just
> run a no-change "tox" successfully on windows.  Unfortunately I gave up
> when I realised I still had several days of downloading+building things
> ahead of me and clearly I was doing it the hard way :(
>
> Could you point me to the "dev environment how-to" doc for hyper-v work?
> I'm thinking of something like
> http://docs.openstack.org/developer/nova/development.environment.html#setup 
> with
> simple cut+paste instructions for the totally-windows-clueless like me ;)
>  Or perhaps a pre-prepared disk image, if the Microsoft license allows such
> a thing.  In particular, the details on
> https://wiki.openstack.org/wiki/Hyper-V look out of date (Grizzly, and
> several docs.openstack.org links are broken), and the links to things
> like https://cloudbase.it/hyper-v-nova-compute-installer-unattended-setup/ 
> look
> like deployment rather than development environments (?).
>
> In particular, I think(?) I should be careful *not* to install too much of
> a cygwin-style environment, since I think(?) that might no longer be
> representative of the environment in which hyper-v is expected to operate.
> If I'm wrong, and cygwin/msys is ok, then it looks like there are several
> free-software-on-windows "distributions" that would make life a whole lot
> simpler (by frankly being less like Windows).  Some guidance from people
> who understand the issues here would be appreciated.
>
>  - Gus
>
> On Tue, 24 Nov 2015 at 22:01 Claudiu Belu <cb...@cloudbasesolutions.com>
> wrote:
>
>> Hello,
>>
>> Thanks Dims for raising the concern and Angus for reaching out. :)
>>
>> Most of the time, python development on Windows is not too far off from
>> Linux. But the two systems are quite different, which imply different
>> modules (fcntl, pwd, grp modules do not exist in Windows) or different
>> implementations of some modules (multiprocessing uses Popen instead of
>> os.fork, os module is quite different) or some socket options and signals
>> are different in Windows.
>>
>> 1.a. As I've said earlier, some modules do not exist in Windows

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Angus Lees]

On Thu, 26 Nov 2015 at 14:19 Robert Collins 
wrote:

> On 26 November 2015 at 15:54, Alessandro Pilotti
>  wrote:
> > When done, open a PowerShell or command prompt and set your PATH and
> > PYTHONPATH e.g.:
> >
> > $ENV:PATH += ";C:\Python27;C:\Python27\Scripts"
> > $ENV:PYTHONPATH = "."
>

Wow, this syntax is completely foreign to me.  You have no idea how much of
a newbie I feel, thanks ;)


> > For running tests you can use for example nose since tox / testr don’t
> > really work:
> >
> > pip install mock
> > pip install nose
> > nosetests .
>
> i haven't seen any bug reports on testrepository vis-a-vis windows;
> please do file them, otherwise I'll presume it works.
>

>From my brief day or two of experience, I think the tox executable itself
works fine - it's more the typical tox.ini:
- "python3.5.exe" doesn't typically exist - the regular python 3.5 windows
install seems to just install an unversioned python.exe and breaks the
usual tox basepython values.  Creating a "python3.5" symlink doesn't seem
to be a thing ;)
- Just about every tox.ini we have is full of unixisms - see for example
nova/tox.ini's liberal use of bash scripts and find commands.
- Virtualenv installs things differently on windows.  In particular the
venv has all the executables in $venv\Scripts\ rather than $venv/bin/.

I quickly gave up on using tox, and was running the virtualenv and "pip
install -r requirements.txt" steps by hand.  The wheels (pun unintended)
well and truly fell off once I hit my first C library dependencies (for me,
yaml and netifaces pulled in by oslo.utils), and I was looking at having to
set up a full C toolchain from scratch too.

I didn't get far enough to try testr.


Alessandro: I think the main blocker I've hit so far (other than "how do I
cut+paste" ;) is installing the non-python library dependencies.  Is there
something I can do to steal your pre-built wheels (and dlls?), or use a
pre-built distro like cygwin/msys2[1], or perhaps we should build a conda
repository for openstack/windows like I did once for openstack/linux[2], or
...?

[1] I didn't get an answer to my earlier question, I presume installing
cygwin _is_ going to invalidate my hyper-v-suitability tests, right?  I
presume other "unix-like distros" like msys2 (mingw-based afaict) are
similarly bad?

[2] https://conda.anaconda.org/gus  - It was a (mostly positive) experiment
in trying something other than pip.  I haven't maintained the repo, but it
didn't take that long to package the transitive dependencies of nova.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [hyper-v] oslo.privsep vs Windows

[Angus Lees]

Dims has just raised[1] the excellent concern that oslo.privsep will need
to at least survive on Windows, because hyper-v.  I have no real experience
coding on windows (I wrote a windows C program once, but I only ever ran it
under wine ;) and certainly none within an OpenStack/python context:

1) How can I test whatever I'm working on to see if I have mistakenly
introduced something Linux-specific?  Surely this is a challenge common
across every project in the nova/oslo/hyper-v stack.

2) What predicate should I use to guard the inevitable Linux-specific or
Windows-specific code branches?

and I guess:
3) What does a typical OpenStack/hyper-v install even look like? Do we run
rootwrap with some sudo-like-thing, or just run everything as the superuser?
What _should_ oslo.privsep do for this environment?

[1] https://review.openstack.org/#/c/244984


 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [oslo] Please stop removing usedevelop from tox.ini (at least for now)

[Angus Lees]

Ah, I could have been clearer.  A workaround for (2) in the original post:
If you're starting from an older pip version it can be hard to bootstrap
the right order of new things to correctly survive the new "; predicate"
requirements.txt syntax.

On Mon, 23 Nov 2015 at 18:10 Robert Collins <robe...@robertcollins.net>
wrote:

> Workaround for what?
>
> -Rob
>
> On 23 November 2015 at 19:39, Angus Lees <g...@inodes.org> wrote:
> > The workaround I found is to set this in tox.ini:
> >
> > install_command =
> >  sh -c 'pip install -U "pip>=6" && pip install -U "$@"' pip {opts}
> > {packages}
> >
> > It's pretty ugly, but works fine and was unfortunately the only way I
> could
> > find to update a dependency before parsing requirements.txt.
> >
> >  - Gus
> >
> > On Tue, 17 Nov 2015 at 06:25 Doug Hellmann <d...@doughellmann.com>
> wrote:
> >>
> >> Excerpts from gord chung's message of 2015-11-16 12:04:27 -0500:
> >> >
> >> > On 16/11/2015 11:13 AM, Doug Hellmann wrote:
> >> > >   does anyone want to sign up to remove that last
> >> > > namespace package?
> >> > already in action: https://review.openstack.org/#/c/243255/
> >> >
> >>
> >> Thanks, Gordon!
> >>
> >>
> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
>
> --
> Robert Collins <rbtcoll...@hp.com>
> Distinguished Technologist
> HP Converged Cloud
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [oslo] Please stop removing usedevelop from tox.ini (at least for now)

[Angus Lees]

The workaround I found is to set this in tox.ini:

install_command =
 sh -c 'pip install -U "pip>=6" && pip install -U "$@"' pip {opts}
{packages}

It's pretty ugly, but works fine and was unfortunately the only way I could
find to update a dependency before parsing requirements.txt.

 - Gus

On Tue, 17 Nov 2015 at 06:25 Doug Hellmann  wrote:

> Excerpts from gord chung's message of 2015-11-16 12:04:27 -0500:
> >
> > On 16/11/2015 11:13 AM, Doug Hellmann wrote:
> > >   does anyone want to sign up to remove that last
> > > namespace package?
> > already in action: https://review.openstack.org/#/c/243255/
> >
>
> Thanks, Gordon!
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [oslo.privsep] Any progress on privsep?

[Angus Lees]

On Fri, 18 Sep 2015 at 14:13 Li Ma  wrote:

> Hi stackers,
>
> Currently we are discussing the possibility of using a pure python
> library to configure network in neutron [1]. We find out that it is
> impossible to do it without privsep, because we run external commands
> which cannot be replaced by python calls via rootwrap.
>
> Privsep has been merged in the Liberty cycle. I just wonder how it is
> going on.
>
> [1] https://bugs.launchpad.net/neutron/+bug/1492714


Thanks for your interest :)  This entire cycle has been spent on the spec.
It looks like it might be approved very soon (got the first +2 overnight),
which will then unblock a string of "create new oslo project" changes.

During the spec discussion, the API was changed (for the better).  Now it
looks like the discussion has settled down, I'm getting to work rewriting
it following the new API.  It took me about 2 weeks to write it the first
time around (almost all on testing framework), so I'd expect something of
similar magnitude this time.

I don't make predictions about timelines that rely on the OpenStack review
process, but if you forced me I'd _guess_ it will be ready for projects to
try out early in M.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo.serialization] Security or convenience?

[Angus Lees]

I'm working on a draft spec[1] for a new privilege separation mechanism
(oslo.privsep) and one of the reviewers mentioned oslo.serialization.  Yay.

My question is: From a quick glance over the current objects, it looks fine
atm - but is the intention that this library remain suitable for
security-sensitive purposes?

I guess I'm mostly concerned about things like PyYaml's !!python/object
feature or pickle's ability to serialise arbitrary objects - super useful
in normal use, just not in a security context.

 - Gus

[1] https://review.openstack.org/#/c/204073
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap discussions at OSSG mid-cycle

[Angus Lees]

On Wed, 13 May 2015 at 02:16 Thierry Carrez thie...@openstack.org wrote:

 Lucas Fisher wrote:
  We spent some time at the OSSG mid-cycle meet-up this week discussing
 root wrap, looking at the existing code, and considering some of the
 mailing list discussions.
 
  Summary of our discussions:
 https://github.com/hyakuhei/OSSG-Security-Practices/blob/master/ossg_rootwrap.md
 
  The one line summary is we like the idea of a privileged daemon with
 higher level interfaces to the commands being run. It has a number of
 advantages such as easier to audit, enables better input sanitization,
 cleaner interfaces, and easier to take advantage of Linux capabilities,
 SELinux, AppArmour, etc. The write-up has some more details.

 For those interested in that topic and willing to work on the next
 stage, we'll have a work session on the future of rootwrap in the Oslo
 track at the Design Summit in Vancouver:

 http://sched.co/3B2B


Fwiw, I've continued work on my privsep proposal(*) and how it interacts
with existing rootwrap.  I look forward to discussing it and alternatives
at the session.

(*) https://review.openstack.org/#/c/155631

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

[Angus Lees]

On Tue, 12 May 2015 at 05:08 Mike Bayer mba...@redhat.com wrote:

 On 5/11/15 2:02 PM, Attila Fazekas wrote:
  The scary part of a blocking I/O call is when you have two
  python thread (or green thread) and one of them is holding a DB lock the
 other
  is waiting for the same lock in a native blocking I/O syscall.
 that's a database deadlock and whether you use eventlet, threads,
 asycnio or even just two transactions in a single-threaded script, that
 can happen regardless.  if your two eventlet non blocking greenlets
 are waiting forever for a deadlock,  you're just as deadlocked as if you
 have OS threads.


Not true (if I understand the situation Attila is referring to).

 If you do a read(2) in native code, the python itself might not be able
 to preempt it
  Your transaction might be finished with `DB Lock wait timeout`,
  with 30 sec of doing nothing, instead of scheduling to the another
 python thread,
  which would be able to release the lock.


 Here's the you're losing me part because Python threads are OS
 threads, so Python isn't directly involved trying to preempt anything,
 unless you're referring to the effect of the GIL locking up the
 program.   However, it's pretty easy to make two threads in Python hit a
 database and do a deadlock against each other, and the rest of the
 program's threads continue to run just fine; in a DB deadlock situation
 you are blocked on IO and IO releases the GIL.

 If you can illustrate a test script that demonstrates the actual failing
 of OS threads that does not occur greenlets here, that would make it
 immediately apparent what it is you're getting at here.


1. Thread A does something that takes a lock on the DB side
2. Thread B does something that blocks waiting for that same DB lock
3. Depends on the threading model - see below

In a true preemptive threading system (eg: regular python threads), (3)
is:

3.  Eventually A finishes its transaction/whatever, commits and releases
the DB lock
4. B then takes the lock and proceeds
5. Profit

However, in a system where B's DB client can't be preempted (eg: eventlet
or asyncio calling into a C-based mysql library, and A and B are running on
the same underlying kernel thread), (3) is:

3. B will never be preempted, A will never be rescheduled, and thus A will
never complete whatever it was doing.
4. Deadlock (in mysql-python's case, until a deadlock timer raises an
exception and kills B 30s later)
5. Sadness.  More specifically, we add a @retry to paper over the
particular observed occurrence and then repeat this discussion on os-dev
when the topic comes up again 6 months later.

Note that this is not the usual database transaction deadlock caused by A
and B each taking a lock and then trying to take the other's lock - this is
a deadlock purely in the client-side code caused entirely by the lack of
preemption during an otherwise safe series of DB operations.

See my oslo.db unittest in Ib35c95defea8ace5b456af28801659f2ba67eb96 that
reproduces the above with eventlet and allows you to test the behaviour of
various DB drivers.

(zzzeek: I know you've already seen all of the above in previous
discussions, so sorry for repeating).

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Need guidance - functional tests and rootwrap

[Angus Lees]

The tox.ini entry for dsvm-fullstack sets OS_ROOTWRAP_CMD=sudo
{envbindir}/neutron-rootwrap {envdir}/etc/neutron/rootwrap.conf (and
something similar for rootwrap-daemon).

Is this the answer you were looking for, or are you saying OS_ROOTWRAP_CMD
doesn't appear to be honoured in your case?

 - Gus

On Sat, 25 Apr 2015 at 00:45 Ihar Hrachyshka ihrac...@redhat.com wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 On 04/24/2015 04:02 PM, Ihar Hrachyshka wrote:
  On 04/24/2015 03:48 PM, Paul Michali wrote:
  Hi, I'm floundering a bit, and could use some guidance on
  this...
 
  For the neutron-vpnaas repo, I am trying to modify the
  functional jobs (dsvm-functional and dsvm-functional-sswan) to
  act in a similar manner to neutron, where devstack is configured,
  but no stacking is performed.
 
  I'm trying to do the same thing and have the jobs doing the
  configuration only. Side note: there are two jobs, because there
  are currently two reference implementations of VPN drivers, each
  of which require a different IPSec package installed.
 
  As part of this setup, in tox.ini, the neutron
  deploy_rootwrap.sh script is called which places the rootwrap
  filters and config file in the repo's
  .tox/dsvm-functional/etc/neutron/ area (or
  ./tox/dsvm-functional-sswan/etc/neutron/).
 
  Now, the issue I see is that tests trying to run ip commands,
  are failing saying that the config file is invalid:
 
  ERROR neutron_vpnaas.services.vpn.common.netns_wrapper [-]
  Incorrect configuration file: /etc/neutron/rootwrap.conf
 
  As you can see, this is trying to access the rootwrap.conf in
  /etc/neutron and not the one in
  /opt/stack/new/neutron-vpnaas/.tox/dsvm-functioanl-sswan/etc/neutron/
 .
 
   For Neutron, how is the dsvm-functional job directing the
  rootwrap daemon to use the files in the repo's .tox area?
 
  It may be the case that oslo_config.cfg.find_config_files is
  involved, doing its dirty config file autodiscovery job. May I ask
  you to try out [1] that is designed to avoid it, and report back
  with the result?
 
  [1]:
  https://review.openstack.org/#/c/172354/1/neutron/tests/base.py
 

 Nah, that won't help for rootwrap. It does not even rely on
 oslo.config, and the config file is passed with CLI args. I recommend
 checking what's cfg.CONF.AGENT.root_helper_daemon value inside your
 failing test cases to see whether tox properly passed
 OS_ROOTWRAP_DAEMON_CMD, with {envdir} properly substituted.

 Ihar
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2

 iQEcBAEBCAAGBQJVOlbTAAoJEC5aWaUY1u57zkgH+wa5yvVYqglN+B7qpkIfR5QB
 5X+6fh9O2KNV8qkDkSKwfRgqs8UouNGOO39zYcgG/QOlqfRKv9ROGkLyNzRihaRg
 ynmDSiXVSiW/wnW+R8ymBSFiU30O88jtlBxlwYYUlz1pdbdQxpVUWPspvYrYU95O
 zdBkifNEvDpYhb/DySq6dlOJB+VQ2IlnCsBhkZeiKQz/T2fnYDoTNZ05beLZez2s
 kntPkYXG11dYRDYQxF76A3fFSboiy2TkX7wl8wK29WQI350gk3Fc/ob0QlMYR0Kf
 IcvEHh+g7cvkZkcX3vn3dDTnI9WUorDUjvnvfw8PGvJaB/edniUBjSC6HHmhBv8=
 =Pg+J
 -END PGP SIGNATURE-

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

[Angus Lees]

On Thu, 23 Apr 2015 at 17:11 Victor Stinner vstin...@redhat.com wrote:

 As announced, changes are boring, just obvious Python2/Python3 issues:

 - strip L from long integer literals: 123L = 123
 - replace dict.iteritems() with six.iteritems(dict)
 - replace list.sort(cmp_func) with list.sort(key=key_func)
 - replace raise exc_info[0], exc_info[1], exc_info[2] with
 six.reraise(*exc_info)
 - moved functions / modules

   * get http.client, urllib.request and other stdlib modules from
 six.moves since they moved between Python 2 and Python 3
   * get itertools.izip/zip_longest from six.moves
   * replace unichr() with six.unichr()

 - replace filter(func, data) with [item for item in data if func(item)]
 - replace unicode() with six.text_type
 - replace (int, long) with six.integer_types
 - replace file() with open()
 - replace StringIO() with six.StringIO()

 These changes are not enough to port nova to Python 3. But they are
 required to be able to find next Python 3 bugs.


Is there already a static analysis tool that helps find these things?
 (Would a pylint check for the above be useful?  Some of them would be hard
to find reliably, but a bunch of the above would be trivial)

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Fwd: [Neutron][DVR]Neutron distributed SNAT

[Angus Lees]

On Mon Feb 16 2015 at 9:37:22 PM Kevin Benton blak...@gmail.com wrote:

 It's basically very much like floating IPs, only you're handing out a
 sub-slice of a floating-IP to each machine - if you like.

 This requires participation of the upstream router (L4 policy routing
 pointing to next hops that distinguish each L3 agent) or intervention on
 the switches between the router an L3 agents (a few OpenFlow rules would
 make this simple). Both approaches need to adapt to L3 agent changes so
 static configuration is not adequate. Unfortunately, both of these are
 outside of the control of Neutron so I don't see an easy way to push this
 state in a generic fashion.


(Just to continue this thought experiment)

The L3 agents that would need to forward ingress traffic to the right
hypervisor only need to know which [IP+port range] has been assigned to
which hypervisor.  This information is fairly static, so these forwarders
are effectively stateless and can be trivially replicated to deal with the
desired ingress volume and reliability.

When I've built similar systems in the past, the easy way to interface with
the rest of the provider network was to use whatever dynamic routing
protocol was already in use, and just advertise multiple ECMP routes for
the SNAT source IPs from the forwarders (ideally advertising from the
forwarders themselves, so they stop if there's a connectivity issue).  All
the cleverness then happens on the forwarding hosts (we could call them
L3 agents).  It's simple and works well, but I agree we have no precedent
in neutron at present.

On Mon, Feb 16, 2015 at 12:33 AM, Robert Collins robe...@robertcollins.net
 wrote:

 Or a pool of SNAT addresses ~= to the size of the hypervisor count.


Oh yeah. If we can afford to assign a unique SNAT address per hypervisor
then we're done - at that point it really is just like floating-ips.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Fwd: [Neutron][DVR]Neutron distributed SNAT

[Angus Lees]

Conntrack synchronisation gets us HA on the SNAT node, but that's a long
way from distributed SNAT.

Distributed SNAT (in at least one implementation) needs a way to allocate
unique [IP + ephemeral port ranges] to hypervisors, and then some sort of
layer4 loadbalancer capable of forwarding the ingress traffic to that IP
back to the right hypervisor/guest based on the ephemeral port range.  It's
basically very much like floating IPs, only you're handing out a sub-slice
of a floating-IP to each machine - if you like.

On Mon Feb 16 2015 at 6:12:33 PM Kevin Benton blak...@gmail.com wrote:

 Has there been any work to use conntrack synchronization similar to L3 HA
 in DVR so failover is fast on the SNAT node?

 On Sat, Feb 14, 2015 at 1:31 PM, Carl Baldwin c...@ecbaldwin.net wrote:


 On Feb 10, 2015 2:36 AM, Wilence Yao wilence@gmail.com wrote:
 
 
  Hi all,
After OpenStack Juno, floating ip is handled by dvr, but SNAT is
 still handled by l3agent on network node. The distributed SNAT is in future
 plans for DVR. In my opinion, SNAT can move to DVR as well as floating ip.
 I have searched in blueprint, there is little  about distributed SNAT. Is
 there any different between distributed floating ip and distributed SNAT?

 The difference is that a shared snat address is shared among instances on
 multiple compute nodes.  A floating ip is exclusive to a single instance on
 one compute node.  I'm interested to hear your ideas for distributing it.

 Carl

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Kevin Benton
  
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][security][rootwrap] Proposal to replace rootwrap/sudo with privsep helper process (for neutron, but others too)

[Angus Lees]

On Fri Feb 13 2015 at 10:35:49 PM Miguel Ángel Ajo majop...@redhat.com
wrote:

  We have an ongoing effort in neutron to move to rootwrap-daemon.


 https://review.openstack.org/#/q/status:open+project:openstack/neutron+branch:master+topic:bp/rootwrap-daemon-mode,n,z

 Thanks for replying. I should have mentioned rootwrap-daemon in my
original post:

The main difference between this privsep proposal and rootwrap (in
daemon-mode or rootwrap classic) is that rootwrap is oriented around
command lines, and privsep is oriented around python function calls.

Using functions rather than command lines means we can move to using native
libraries more easily (as I've done here with pyroute2), which allows a
more closely integrated, performant and safer interface.  It also means we
can describe more customised behaviour in the exposed function API, leading
to fewer cross-process calls and less scope for exposing unintended
behaviour.

To pick one example, starting up a long-running daemon (assuming we wanted
to do this) from the privsep process just works without any need to work
around the privsep mechanism.

To speed up multiple system calls, and be able to spawn daemons inside
 namespaces.


Yep, rootwrap-daemon certainly removes the python rootwrap startup time;
privsep *also* removes the subprocess exec startup time for cases where we
have moved to native libraries (again, see the pyroute2 example).

I have to read a bit what are the good  bad points of privsep.

 The advantage of rootwrap-daemon, is that we don’t need to change all our
 networking libraries across neutron,
 and we kill the sudo/rootwrap spawn for every call, yet keeping
 the rootwrap permission granularity.


The good news is that privsep and rootwrap (in either mode) can coexist
just fine.  A staged migration to privsep might involve spawning the
privsep daemon via sudo on the first call to something that needs it.  This
approach means we wouldn't require adding the privsep_daemon.start() call
early in any relevant main() - and the only downside is that we retain a
(single) dependency on sudo/sudoers.

 - Gus

Miguel Ángel Ajo

 On Friday, 13 de February de 2015 at 10:54, Angus Lees wrote:

 On Fri Feb 13 2015 at 5:45:36 PM Eric Windisch e...@windisch.us wrote:

 ᐧ


 from neutron.agent.privileged.commands import ip_lib as priv_ip
 def foo():
 # Need to create a new veth interface pair - that usually requires
 root/NET_ADMIN
 priv_ip.CreateLink('veth', 'veth0', peer='veth1')

 Because we now have elevated privileges directly (on the privileged daemon
 side) without having to shell out through sudo, we can do all sorts of
 nicer things like just using netlink directly to configure networking.
 This avoids the overhead of executing subcommands, the ugliness (and
 danger) of generating command lines and regex parsing output, and make us
 less reliant on specific versions of command line tools (since the kernel
 API should be very stable).


 One of the advantages of spawning a new process is being able to use flags
 to clone(2) and to set capabilities. This basically means to create
 containers, by some definition. Anything you have in a privileged daemon
 or privileged process ideally should reduce its privilege set for any
 operation it performs. That might mean it clones itself and executes
 Python, or it may execvp an executable, but either way, the new process
 would have less-than-full-privilege.

 For instance, writing a file might require root access, but does not need
 the ability to load kernel modules. Changing network interfaces does not
 need access to the filesystem, no more than changes to the filesystem needs
 access to the network. The capabilities and namespaces mechanisms resolve
 these security conundrums and simplify principle of least privilege.


 Agreed wholeheartedly, and I'd appreciate your thoughts on how I'm using
 capabilities in this change.  The privsep daemon limits itself to a
 particular set of capabilities (and drops root). The assumption is that
 most OpenStack services commonly need the same small set of capabilities to
 perform their duties (neutron - net_admin+sys_admin for example), so it
 makes sense to reuse the same privileged process.

 If we have a single service that somehow needs to frequently use a broad
 swathe of capabilities then we might want to break it up further somehow
 between the different internal aspects (multiple privsep helpers?) - but is
 there such a case?   There's also no problems with mixing privsep for
 frequent operations with the existing sudo/rootwrap approach for
 rare/awkward cases.

  - Gus

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][security][rootwrap] Proposal to replace rootwrap/sudo with privsep helper process (for neutron, but others too)

[Angus Lees]

On Fri Feb 13 2015 at 5:45:36 PM Eric Windisch e...@windisch.us wrote:

 ᐧ


 from neutron.agent.privileged.commands import ip_lib as priv_ip
 def foo():
 # Need to create a new veth interface pair - that usually
 requires root/NET_ADMIN
 priv_ip.CreateLink('veth', 'veth0', peer='veth1')

 Because we now have elevated privileges directly (on the privileged
 daemon side) without having to shell out through sudo, we can do all sorts
 of nicer things like just using netlink directly to configure networking.
 This avoids the overhead of executing subcommands, the ugliness (and
 danger) of generating command lines and regex parsing output, and make us
 less reliant on specific versions of command line tools (since the kernel
 API should be very stable).


 One of the advantages of spawning a new process is being able to use flags
 to clone(2) and to set capabilities. This basically means to create
 containers, by some definition. Anything you have in a privileged daemon
 or privileged process ideally should reduce its privilege set for any
 operation it performs. That might mean it clones itself and executes
 Python, or it may execvp an executable, but either way, the new process
 would have less-than-full-privilege.

 For instance, writing a file might require root access, but does not need
 the ability to load kernel modules. Changing network interfaces does not
 need access to the filesystem, no more than changes to the filesystem needs
 access to the network. The capabilities and namespaces mechanisms resolve
 these security conundrums and simplify principle of least privilege.


Agreed wholeheartedly, and I'd appreciate your thoughts on how I'm using
capabilities in this change.  The privsep daemon limits itself to a
particular set of capabilities (and drops root). The assumption is that
most OpenStack services commonly need the same small set of capabilities to
perform their duties (neutron - net_admin+sys_admin for example), so it
makes sense to reuse the same privileged process.

If we have a single service that somehow needs to frequently use a broad
swathe of capabilities then we might want to break it up further somehow
between the different internal aspects (multiple privsep helpers?) - but is
there such a case?   There's also no problems with mixing privsep for
frequent operations with the existing sudo/rootwrap approach for
rare/awkward cases.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][security][rootwrap] Proposal to replace rootwrap/sudo with privsep helper process (for neutron, but others too)

[Angus Lees]

So inspired by the Rootwrap on root-intensive nodes thread, I went and
wrote a proof-of-concept privsep daemon for neutron:
https://review.openstack.org/#/c/155631
There's nothing neutron-specific in the core mechanism and it could easily
be moved out into a common (oslo) library and reused across other projects.


The basic principles guiding a few design choices were:
- it had to be obvious what code would run with elevated privileges
- the interface between that and the rest of the system had to be easy to
understand and audit
- it had to be about as easy as just implementing a function to add new
functionality

The current code does the predictable things to get a privileged buddy
process: assumes you started as root, creates a socketpair, forks, keeps
limited perms and setuids the privileged process, then reads commands over
the socket.  When the socket closes (for whatever reason), the daemon exits.

Currently it scrobbles around below the neutron.agent.privileged._commands
namespace and allows you to invoke any normal function that doesn't start
start with an underscore.  I think I will change this to use some sort of
more explicit decorator, but you get the idea.

On the client side, it generates client stubs of all the same functions
below neutron.agent.privileged.commands at import-time.  Using the daemon
is then as simple as just calling the client stub:

from neutron.agent.privileged import daemon as privsep_daemon
def main():
privsep_daemon.start()
...

from neutron.agent.privileged.commands import ip_lib as priv_ip
def foo():
# Need to create a new veth interface pair - that usually requires
root/NET_ADMIN
priv_ip.CreateLink('veth', 'veth0', peer='veth1')

Because we now have elevated privileges directly (on the privileged daemon
side) without having to shell out through sudo, we can do all sorts of
nicer things like just using netlink directly to configure networking.
This avoids the overhead of executing subcommands, the ugliness (and
danger) of generating command lines and regex parsing output, and make us
less reliant on specific versions of command line tools (since the kernel
API should be very stable).
I demonstrate some of that in the above change by a set of privileged ipset
functions that still call out to commands but don't use sh -c or sudo
anywhere (so are immune to shell metacharacters in arguments), and ip_lib
functions that use pyroute2 to just call netlink directly for network
interface create/delete/update.

Please discuss.  I could have done this in a spec, but I felt the basic
concept and motivation was obvious and the code specifics were of such
importance that this was better explored in a poc change.  I can post-facto
write a spec if it turns out folks would prefer that.

 - Gus

(if you're curious, it took about a day to write the code, and then about 3
long days of debugging eventlet-related conflicts with the 3rd party
libraries I'd just pulled in.  +1 to removing eventlet, particularly in
low-concurrent-queries agent processes where we can presumably just remove
it and use system threads without any further thought)

On Fri Feb 06 2015 at 5:52:44 PM Steven Dake (stdake) std...@cisco.com
wrote:



 On 2/4/15, 10:24 AM, Daniel P. Berrange berra...@redhat.com wrote:

 On Wed, Feb 04, 2015 at 09:10:06AM -0800, James E. Blair wrote:
  Thierry Carrez thie...@openstack.org writes:
 
   You make a good point when you mention traditional distro here. I
   would argue that containers are slightly changing the rules of the
   don't-run-as-root game.
  
   Solution (2) aligns pretty well with container-powered OpenStack
   deployments -- running compute nodes as root in a container (and
   embracing abovementioned simplicity/performance gains) sounds like a
   pretty strong combo.
 
  This sounds at least a little like a suggestion that containers are a
  substitute for the security provided by running non-root.  The security
  landscape around containers is complex, and while there are a lot of
  benefits, I believe the general consensus is that uid 0 processes should
  not be seen as fully isolated.
 
  From https://docs.docker.com/articles/security/ :
 
Docker containers are, by default, quite secure; especially if you
take care of running your processes inside the containers as
non-privileged users (i.e., non-root).
 
  Which is not to say that using containers is not a good idea, but
  rather, if one does, one should avoid running as root (perhaps with
  capabilities), and use selinux (or similar).
 
 Yep, I've seen attempts by some folks to run nova-compute and libvirtd
 and QEMU inside a docker container. Because of the inherantly privileged
 nature of what Nova/libvirt/qemu need to do, you end up having to share
 all the host namespaces with the docker container, except for the
 filesystem
 namespace and even that you need to bind mount a bunch of stuff over. As
 a result the container isn't really offerring 

Re: [openstack-dev] [neutron][security][rootwrap] Proposal to replace rootwrap/sudo with privsep helper process (for neutron, but others too)

[Angus Lees]

On Fri Feb 13 2015 at 4:05:33 PM Robert Collins robe...@robertcollins.net
wrote:


 On 13 Feb 2015 17:42, Angus Lees g...@inodes.org wrote:
 
  So inspired by the Rootwrap on root-intensive nodes thread, I went and
 wrote a proof-of-concept privsep daemon for neutron:
 https://review.openstack.org/#/c/155631
  There's nothing neutron-specific in the core mechanism and it could
 easily be moved out into a common (oslo) library and reused across other
 projects.

 Bravo. More conceptual than a code review my questions are. msgpack rather
 than protobuf ? Given your previous experience there I'm just curious.

I have no educated preference between the two, and I didn't know of any
high-performance precedent within openstack.  msgpack was just the first
thing I came across that seemed well supported, fast, and only handled dumb
types (no object auto-vivifying features that might backfire on us).

We could use json too if we wanted to avoid a new dependency, or presumably
numerous other choices.


  Are you concerned that commands might call into less trusted areas of
 code? Would it make sense to have the privileged commands be separate
 somehow to avoid this?

Hrm, not particularly, although we should explore any implications.  If a
standalone chunk of python imported other python libraries, then they may
have a path that ends up with them able to be called - which I figure is
similar to the current situation that also requires an explicit python
import (or some other chain of object references).  If there's a bug that
lets you escape the python level and run arbitrary C code, then it won't
matter what's already loaded and we only have the linux
capabilities/permissions mechanisms to save us.

In addition, the current simple fork/no-exec is also good for sharing most
of the pages in memory - making the overhead extremely minimal.

Oh, if you mean separate just in a filesystem/code organisation sense,
rather than a Linux process sense, then yes I do think they should be in a
separate place for ease of auditing.  in my change above I have them all
below a particular neutron.agent.privileged._commands prefix, and the
communication assumes/restricts it to this.  We can of course pick another
namespace prefix, but I agree that even with some different decorator-based
method, I don't think we should just have privileged commands scattered
anywhere throughout our regular codebase.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][oslo.db][nova] TL; DR Things everybody should know about Galera

[Angus Lees]

Thanks for the additional details Peter.  This confirms the parts I'd
deduced from the docs I could find, and is useful knowledge.

On Sat Feb 07 2015 at 2:24:23 AM Peter Boros peter.bo...@percona.com
wrote:

 - Like many others said it before me, consistent reads can be achieved
 with wsrep_causal_reads set on in the session.


So the example was two dependent command-line invocations (write followed
by read) that have no way to re-use the same DB session (without
introducing lots of affinity issues that we'd also like to avoid).

Enabling wsrep_casual_reads makes sure the latter read sees the effects of
the earlier write, but comes at the cost of delaying all reads by some
amount depending on the write-load of the galera cluster (if I understand
correctly).  This additional delay was raised as a concern severe enough
not to just go down this path.

Really we don't care about other writes that may have occurred (we always
need to deal with races against other actors), we just want to ensure our
earlier write has taken effect on the galera server where we sent the
second read request.  If we had some way to say wsrep_delay_until
$first_txid then we we could be sure of read-after-write from a different
DB session and also (in the vast majority of cases) suffer no additional
delay.  An opaque sequencer is a generic concept across many of the
distributed consensus stores I'm familiar with, so this needn't be exposed
as a Galera-only quirk.


Meh, I gather people are bored with the topic at this point.  As I
suggested much earlier, I'd just enable wsrep_casual_reads on the first
request for the session and then move on to some other problem ;)

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][tripleo] Update on Kolla

[Angus Lees]

Without taking anything away from this fine and sensible direction, where
_would_ be a good place to discuss more radical container-based deployments?
(I'm fine if the answer is use ad-hoc communication channels for now)

On Fri Feb 06 2015 at 7:24:12 AM Steven Dake (stdake) std...@cisco.com
wrote:


 On 2/5/15, 10:41 AM, Paul Bourke paul.bou...@oracle.com wrote:

 Hi Steve,
 
 Thanks for the update.  For those interested in Kolla development and
 discussion, where is the best place to go?
 
 Regards,
 -Paul

 Paul,

 We hang out in the irc channel #tripleo on freenode.

 We use launchpad at http://launchpad.net/kolla

 We use the openstack-dev mailing list with the tag [kolla]

 Regards,
 -steve



 
 On 05/02/15 17:25, Steven Dake (stdake) wrote:
  Hey folks,
 
  I wanted to provide a brief update on where we are headed with Kolla.
Initially Kolla began as a POC to show that containers could be used
  to deploy OpenStack with the long term plan of integrating that
  functionality into TripleO.  That goal has not changed.
 
  The tripleo community wants baby steps not outright replacement of the
  infrastructure as was proposed by using Kubernetes in Kolla as a
  dependency in this review [1].   Our first step in this process will be
  to create three new container types.  We will create a controller node,
  a storage node, and a compute node.  We have proven that compute can be
  used.[2]  Further, one of our community members is working on a DIB
  elements-container tool here [3] and we plan to use this along with our
  other RD in the area to integrate container tech into tripleo.
 
  I¹m not quite sure how it will all come together, but after numerous
  discussions with folks from the tripleo and heat teams, I believe we can
  find a path forward.
 
  Regards
  -steve
 
 
  [1] https://review.openstack.org/#/c/144199/
  [2]
 
 http://sdake.io/2015/01/28/an-atomic-upgrade-process-for-
 openstack-comput
 e-nodes/
  _[3] _https://review.openstack.org/#/c/152017/
 
 
 
 __
 ___
 _
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 ___
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
 unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][oslo.db][nova] TL; DR Things everybody should know about Galera

[Angus Lees]

On Fri Feb 06 2015 at 4:25:43 AM Clint Byrum cl...@fewbar.com wrote:

 In a single thread, using a single database session, then a read after
 successful commit is guaranteed to read a version of the database
 that existed after that commit.


Ah, I'm relieved to hear this clarification - thanks.

I'd like to see actual examples where that will matter. Meanwhile making
 all selects wait for the cluster will basically just ruin responsiveness
 and waste tons of time, so we should be careful to think this through
 before making any blanket policy.


Matthew's example earlier in the thread is simply a user issuing two
related commands in succession:

$ nova aggregate-create
$ nova aggregate-details

Once that fails a few times, the user will put a poorly commented sleep 2
in between the two statements, and this will fix the problem most of the
time.  A better fix would repeat the aggregate-details query multiple
times until it looks like it has found the previous create.

Now, that sleep or poll is of course a poor version of something you could
do at a lower level, by waiting for reads+writes to propagate to a majority
quorum.

I'd also like to see consideration given to systems that handle
 distributed consistency in a more active manner. etcd and Zookeeper are
 both such systems, and might serve as efficient guards for critical
 sections without raising latency.


+1 for moving to such systems.  Then we can have a repeat of the above
conversation without the added complications of SQL semantics ;)

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][oslo.db][nova] TL; DR Things everybody should know about Galera

[Angus Lees]

On Fri Feb 06 2015 at 12:59:13 PM Gregory Haynes g...@greghaynes.net
wrote:

 Excerpts from Joshua Harlow's message of 2015-02-06 01:26:25 +:
  Angus Lees wrote:
   On Fri Feb 06 2015 at 4:25:43 AM Clint Byrum cl...@fewbar.com
   mailto:cl...@fewbar.com wrote:
   I'd also like to see consideration given to systems that handle
   distributed consistency in a more active manner. etcd and
 Zookeeper are
   both such systems, and might serve as efficient guards for critical
   sections without raising latency.
  
  
   +1 for moving to such systems.  Then we can have a repeat of the above
   conversation without the added complications of SQL semantics ;)
  
 
  So just an fyi:
 
  http://docs.openstack.org/developer/tooz/ exists.
 
  Specifically:
 
  http://docs.openstack.org/developer/tooz/developers.
 html#tooz.coordination.CoordinationDriver.get_lock
 
  It has a locking api that it provides (that plugs into the various
  backends); there is also a WIP https://review.openstack.org/#/c/151463/
  driver that is being worked for etc.d.
 

 An interesting note about the etcd implementation is that you can
 select per-request whether you want to wait for quorum on a read or not.
 This means that in theory you could obtain higher throughput for most
 operations which do not require this and then only gain quorum for
 operations which require it (e.g. locks).


Along those lines and in an effort to be a bit less doom-and-gloom, I spent
my lunch break trying to find non-marketing documentation on the Galera
replication protocol and how it is exposed. (It was surprisingly difficult
to find such information *)

It's easy to get the transaction ID of the last commit
(wsrep_last_committed), but I can't find a way to wait until at least a
particular transaction ID has been synced.  If we can find that latter
functionality, then we can expose that sequencer all the way through (HTTP
header?) and then any follow-on commands can mention the sequencer of the
previous write command that they really need to see the effects of.

In practice, this should lead to zero additional wait time, since the
Galera replication has almost certainly already caught up by the time the
second command comes in - and we can just read from the local server with
no additional delay.

See the various *Index variables in the etcd API, for how the same idea
gets used there.

 - Gus

(*) In case you're also curious, the only doc I found with any details was
http://galeracluster.com/documentation-webpages/certificationbasedreplication.html
and its sibling pages.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] high dhcp lease times in neutron deployments considered harmful (or not???)

[Angus Lees]

On Wed Feb 04 2015 at 8:02:04 PM Kevin Benton blak...@gmail.com wrote:

 I proposed an alternative to adjusting the lease time early on the in the
 thread. By specifying the renewal time (DHCP option 58), we can have the
 benefits of a long lease time (resiliency to long DHCP server outages)
 while having a frequent renewal interval to check for IP changes. I favored
 this approach because it only required a patch to dnsmasq to allow that
 option to be set and patch to our agent to set that option, both of which
 are pretty straight-forward.


 Yep, I should have said +1 to this in my other post.  Simple coding change
that is strictly better than the current situation (other than a slight
increase in DHCP request traffic).

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][oslo.db][nova] TL; DR Things everybody should know about Galera

[Angus Lees]

On Thu Feb 05 2015 at 9:02:49 AM Robert Collins robe...@robertcollins.net
wrote:

 On 5 February 2015 at 10:24, Joshua Harlow harlo...@outlook.com wrote:
  How interesting,
 
  Why are people using galera if it behaves like this? :-/

 Because its actually fairly normal. In fact its an instance of point 7
 on https://wiki.openstack.org/wiki/BasicDesignTenets - one of our
 oldest wiki pages :).

 In more detail, consider what happens in full isolation when you have
 the A and B example given, but B starts its transaction before A.

 B BEGIN
 A BEGIN
 A INSERT foo
 A COMMIT
 B SELECT foo - NULL


Note that this still makes sense from each of A and B's individual view of
the world.

If I understood correctly, the big change with Galera that Matthew is
highlighting is that read-after-write may not be consistent from the pov of
a single thread.

Not have read-after-write is *really* hard to code to (see for example x86
SMP cache coherency, C++ threading semantics, etc which all provide
read-after-write for this reason).  This is particularly true when the
affected operations are hidden behind an ORM - it isn't clear what might
involve a database call and sequencers (or logical clocks, etc) aren't made
explicit in the API.

I strongly suggest just enabling wsrep_casual_reads on all galera sessions,
unless you can guarantee that the high-level task is purely read-only, and
then moving on to something else ;)  If we choose performance over
correctness here then we're just signing up for lots of debugging of hard
to reproduce race conditions, and the fixes are going to look like what
wsrep_casual_reads does anyway.

(Mind you, exposing sequencers at every API interaction would be awesome,
and I look forward to a future framework and toolchain that makes that easy
to do correctly)

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] high dhcp lease times in neutron deployments considered harmful (or not???)

[Angus Lees]

There's clearly not going to be any amount of time that satisfies both
concerns here.

Just to get some other options on the table, here's some things that would
allow a non-zero dhcp lease timeout _and_ address Kevin's original bug
report:

- Just don't allow users to change their IPs without a reboot.

- Bounce the link under the VM when the IP is changed, to force the guest
to re-request a DHCP lease immediately.

- Remove the IP spoofing firewall feature  (- my favourite, for what it's
worth. I've never liked presenting a layer2 abstraction but then forcing
specific layer3 addressing choices by default)

- Make the IP spoofing firewall allow an overlap of both old and new
addresses until the DHCP lease time is up (or the instance reboots).  Adds
some additional async tasks, but this is clearly the required solution if
we want to keep all our existing features.

On Wed Feb 04 2015 at 4:28:11 PM Aaron Rosen aaronoro...@gmail.com wrote:

 I believe I was the one who changed the default value of this. When we
 upgraded our internal cloud ~6k networks back then from folsom to grizzly
 we didn't account that if the dhcp-agents went offline that instances would
 give up their lease and unconfigure themselves causing an outage. Setting a
 larger value for this helps to avoid this downtime (as Brian pointed out as
 well). Personally, I wouldn't really expect my instance to automatically
 change it's ip  - I think requiring the user to reboot the instance or use
 the console to correct the ip should be good enough. Especially since this
 will help buy you shorter down time if an agent fails for a little while
 which is probably more important than having the instance change it's ip.

 Aaron

 On Tue, Feb 3, 2015 at 5:25 PM, Kevin Benton blak...@gmail.com wrote:

 I definitely understand the use-case of having updatable stuff and I
 don't intend to support any proposals to strip away that functionality.
 Brian was suggesting was to block port IP changes since it depended on DHCP
 to deliver that information to the hosts. I was just pointing out that we
 would need to block any API operations that resulted in different
 information being delivered via DHCP for that approach to make sense.

 On Tue, Feb 3, 2015 at 5:01 PM, Robert Collins robe...@robertcollins.net
  wrote:

 On 3 February 2015 at 00:48, Kevin Benton blak...@gmail.com wrote:
 The only thing this discussion has convinced me of is that allowing
 users
  to change the fixed IP address on a neutron port leads to a bad
  user-experience.
 ...

 Documenting a VM reboot is necessary, or even deprecating this (you
 won't
  like that) are sounding better to me by the minute.
 
  If this is an approach you really want to go with, then we should at
 least
  be consistent and deprecate the extra dhcp options extension (or at
 least
  the ability to update ports' dhcp options). Updating subnet attributes
 like
  gateway_ip, dns_nameserves, and host_routes should be thrown out as
 well.
  All of these things depend on the DHCP server to deliver updated
 information
  and are hindered by renewal times. Why discriminate against IP updates
 on a
  port? A failure to receive many of those other types of changes could
 result
  in just as severe of a connection disruption.

 So the reason we added the extra dhcp options extension was to support
 PXE booting physical machines for Nova baremetal, and then Ironic. It
 wasn't added for end users to use on the port, but as a generic way of
 supporting the specific PXE options needed - and that was done that
 way after discussing w/Neutron devs.

 We update ports for two reasons. Primarily, Ironic is HA and will move
 the TFTPd that boots are happening from if an Ironic node has failed.
 Secondly, because a non uncommon operation on physical machines is to
 replace broken NICs, and forcing a redeploy seemed unreasonable. The
 former case doesn't affect running nodes since its only consulted on
 reboot. The second case is by definition only possible when the NIC in
 question is offline (whether hotplug hardware or not).

 -Rob


 --
 Robert Collins rbtcoll...@hp.com
 Distinguished Technologist
 HP Converged Cloud


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Kevin Benton

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 

Re: [openstack-dev] [oslo.db] PyMySQL review

[Angus Lees]

The other factor I haven't seen mentioned here is that our usual
eventlet+mysqldb setup can deadlock rather easily(*), resulting in the
entire python process sitting idle for 30s until the mysqldb deadlock timer
goes off and raises an exception.  At this point (in nova at least), the
request is usually retried and almost certainly succeeds.  In neutron (as
another example), the requests weren't so easy to retry and code examples
where this was noticed had to be explicitly restructured to defer work that
might yield the greenlet.
ie: it's a big cost on coding, and deployments, as well as just being plain
incorrect code.

(*) eg: send simultaneous requests that modify the same resource.
Eventually you'll have two greenlets within the same api server executing
conflicting database operations.  Entire python process freezes.   (Note
this is not a regular db transaction block, neither greenlet will ever exit
their transactions without the deadlock timer).

Bug here: https://bugs.launchpad.net/oslo.db/+bug/1350149
Test that reproduces the issue here:
https://review.openstack.org/#/c/104436/

I'm dismayed that the conversation always skews towards discussing
performance when the current choice isn't even correct yet, and that as a
community we seem to be unwilling/unable to get behind what should be quite
an obvious technical issue with straightforward solutions.

If we absolutely can't switch to another mysql driver, another option that
was suggested recently (and passes the above test) is using
eventlet.monkey_patch(MySQLdb=True).  I haven't done the investigation to
find out why that isn't the default, or what the downsides are.  This
obviously doesn't help us with other factors, like python3-ness either.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] Linux capabilities vs sudo/rootwrap?

[Angus Lees]

[I tried to find any previous discussion of this and failed - I'd
appreciate a pointer to any email threads / specs where this has already
been discussed.]

Currently neutron is given the ability to do just about anything to
networking via rootwrap, sudo, and the IpFilter check (allow anything
except 'netns exec').  This is completely in line with the role a typical
neutron agent is expected to play on the local system.

There are also recurring discussions/issues around the overhead of
rootwrap, costs of sudo calls, etc - and projects such as rootwrap daemon
underway to improve this.

How crazy would it be to just give neutron CAP_NET_ADMIN (where required),
and allow it to make network changes via ip (netlink) calls directly?
We will still need rootwrap/sudo for other cases, but this should remove a
lot of the separate process overhead for common operations, make us
independent of iproute cli versions, and allow us to use a direct
programmatic API (rtnetlink and other syscalls) rather than generating
command lines and regex parsing output everywhere.

For what it's worth, CAP_NET_ADMIN is not sufficient to allow 'netns exec'
(requires CAP_SYS_ADMIN), so it preserves the IpFilter semantics. On the
downside, many of the frequent rootwrap calls _do_ involve
creating/modifying network namespaces so we wouldn't see advantages for
these cases.  I need to experiment further before having a proposal for
that part (just granting CAP_SYS_ADMIN too is too broad; user namespaces
help a lot, but they're very new and scary so not available everywhere).

Thoughts?
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Linux capabilities vs sudo/rootwrap?

[Angus Lees]

Afaik ovs-vsctl just talks to ovsdb-server via some regular IPC mechanism -
usually a unix socket, see --db.  So yes, provided the file permissions on
the unix socket work out, then no root powers (or any kernel capability) is
necessary.

Having said that, there will be plenty of examples where sudo/rootwrap is
still required.  I'm not suggesting we can (or should!) get rid of it
entirely - just for basic network configuration.

On Wed Dec 10 2014 at 12:42:46 PM George Shuklin george.shuk...@gmail.com
wrote:

  Is ovs-vsctl gonna be happy with CAP_NET_ADMIN?

 On 12/10/2014 02:43 AM, Angus Lees wrote:

 [I tried to find any previous discussion of this and failed - I'd
 appreciate a pointer to any email threads / specs where this has already
 been discussed.]

  Currently neutron is given the ability to do just about anything to
 networking via rootwrap, sudo, and the IpFilter check (allow anything
 except 'netns exec').  This is completely in line with the role a typical
 neutron agent is expected to play on the local system.

  There are also recurring discussions/issues around the overhead of
 rootwrap, costs of sudo calls, etc - and projects such as rootwrap daemon
 underway to improve this.

  How crazy would it be to just give neutron CAP_NET_ADMIN (where
 required), and allow it to make network changes via ip (netlink) calls
 directly?
 We will still need rootwrap/sudo for other cases, but this should remove a
 lot of the separate process overhead for common operations, make us
 independent of iproute cli versions, and allow us to use a direct
 programmatic API (rtnetlink and other syscalls) rather than generating
 command lines and regex parsing output everywhere.

  For what it's worth, CAP_NET_ADMIN is not sufficient to allow 'netns
 exec' (requires CAP_SYS_ADMIN), so it preserves the IpFilter semantics. On
 the downside, many of the frequent rootwrap calls _do_ involve
 creating/modifying network namespaces so we wouldn't see advantages for
 these cases.  I need to experiment further before having a proposal for
 that part (just granting CAP_SYS_ADMIN too is too broad; user namespaces
 help a lot, but they're very new and scary so not available everywhere).

  Thoughts?


 ___
 OpenStack-dev mailing 
 listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


  ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][neutron] Proposal to split Neutron into separate repositories

[Angus Lees]

On Mon Dec 01 2014 at 2:06:18 PM henry hly henry4...@gmail.com wrote:

 My suggestion is that starting with LB and VPN as a trial, which can
 never be distributed.


.. Sure they can!   Loadbalancing in particular _should_ be distributed if
both the clients and backends are in the same cluster...

(I agree with your suggestion to start with LB+VPN btw, just not your
reasoning ;)

 - Gus
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] Re: [neutron] the hostname regex pattern fix also changed behaviour :(

[Angus Lees]

On Fri Nov 28 2014 at 10:49:21 PM Ihar Hrachyshka ihrac...@redhat.com
wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA512

 On 28/11/14 01:26, Angus Lees wrote:
  Context: https://review.openstack.org/#/c/135616
 
  If we're happy disabling the check for components being all-digits, then
  a minimal change to the existing regex that could be backported might be
  something like
r'(?=^.{1,254}$)(^(?:[a-zA-Z0-9_](?:[a-zA-Z0-9_-]{,61}[a-zA-
 Z0-9])\.)*(?:[a-zA-Z]{2,})$)'
 
  Alternatively (and clearly preferable for Kilo), Kevin has a replacement
  underway that rewrites this entirely to conform to modern RFCs in
  I003cf14d95070707e43e40d55da62e11a28dfa4e

 With the change, will existing instances work as before?


Yes, this cuts straight to the heart of the matter:  What's the purpose of
these validation checks?  Specifically, if someone is using an invalid
hostname that passed the previous check but doesn't pass an
improved/updated check, should we continue to allow it?
I figure our role here is either to allow exactly what the relevant
standards say, and deliberately reject/break anything that falls outside
that - or be liberal, restrict only to some sort of 'safe' input and then
let the underlying system perform the final validation.  I can see plenty
of reasons for either approach, but somewhere in the middle doesn't seem to
make much sense - and I think the approach chosen also dictates any
migration path.

As they currently stand, I think both Kevin's and my alternative above
_should_ be more liberal than the original (before the fix) regex.
Specifically, they now allow all-digits hostname components - in line with
newer RFCs.

However, TLD handling is a little different between the variants:
- Kevin's continues to reject an all-digits TLD (also following RFC
guidance)
- mine and the original force TLDs to be all alpha (a-z; no digits or
dash/underscore)

The TLD handling is more interesting because an unqualified hostname (with
no '.' characters) hits the TLD logic in all variants, but the original has
a \.? quirk that means an unqualified hostname is forced to end with at
least 2 alpha-only chars.  As written above, mine is probably too
restrictive for unqualified names, and this would need to be fixed.

As the above shows, describing regex patterns in prose is long, boring and
inaccurate.  Someone who is going to have to approve the change should just
dictate what they want here and then we'll go do that :P
I suggest they also consider the DoS-fix-backport and the Kilo-and-forwards
cases separately.

 - Gus
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] the hostname regex pattern fix also changed behaviour :(

[Angus Lees]

Context: https://review.openstack.org/#/c/135616

As far as I can make out, the fix for CVE-2014-7821 removed a backslash
that effectively disables the negative look-ahead assertion that verifies
that hostname can't be all-digits. Worse, the new version now rejects
hostnames where a component starts with a digit.

This certainly addressed the immediate issue of that regex was expensive,
but the change in behaviour looks like it was unintended.  Given that we
backported this DoS fix to released versions of neutron, what do we want to
do about it now?

In general this regex is crazy complex for what it verifies.  I can't see
any discussion of where it came from nor precisely what it was intended to
accept/reject when it was introduced in patch 16 of
https://review.openstack.org/#/c/14219.

If we're happy disabling the check for components being all-digits, then a
minimal change to the existing regex that could be backported might be
something like
  
r'(?=^.{1,254}$)(^(?:[a-zA-Z0-9_](?:[a-zA-Z0-9_-]{,61}[a-zA-Z0-9])\.)*(?:[a-zA-Z]{2,})$)'

Alternatively (and clearly preferable for Kilo), Kevin has a replacement
underway that rewrites this entirely to conform to modern RFCs in
I003cf14d95070707e43e40d55da62e11a28dfa4e
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][nova] New specs on routed networking

[Angus Lees]

On Tue, 28 Oct 2014 09:07:03 PM Rohit Agarwalla wrote:
 Agreed. The way I'm thinking about this is that tenants shouldn't care what
 the underlying implementation is - L2 or L3. As long as the connectivity
 requirements are met using the model/API, end users should be fine. The
 data center network design should be an administrators decision based on
 the implementation mechanism that has been configured for OpenStack.

I don't know anything about Project Calico, but I have been involved with 
running a large cloud network previously that made heavy use of L3 overlays.  

Just because these points weren't raised earlier in this thread:  In my 
experience, a move to L3 involves losing:

- broadcast/multicast.  It's possible to do L3 multicast/IGMP/etc, but that's 
a whole can of worms - so perhaps best to just say up front that this is a 
non-broadcast network.

- support for other IP protocols.

- various L2 games like virtual MAC addresses, etc that NFV/etc people like.


We gain:

- the ability to have proper hierarchical addressing underneath (which is a 
big one for scaling a single network).  This itself is a tradeoff however - 
an efficient/strict hierarchical addressing scheme means VMs can't choose their 
own IP addresses, and VM migration is messy/limited/impossible.

- hardware support for dynamic L3 routing is generally universal, through a 
small set of mostly-standard protocols (BGP, ISIS, etc).

- can play various L3 games like BGP/anycast, which is super useful for 
geographically diverse services.


It's certainly a useful tradeoff for many use cases.  Users lose some 
generality in return for more powerful cooperation with the provider around 
particular features, so I sort of think of it like a step halfway up the IaaS-
PaaS stack - except for networking.

 - Gus

 Thanks
 Rohit
 
 From: Kevin Benton blak...@gmail.commailto:blak...@gmail.com
 Reply-To: OpenStack Development Mailing List (not for usage questions)
 openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
  Date: Tuesday, October 28, 2014 1:01 PM
 To: OpenStack Development Mailing List (not for usage questions)
 openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
  Subject: Re: [openstack-dev] [neutron][nova] New specs on routed
 networking
 1. Every packet L3 FIB Lookup : Radix Tree Search, instead of current L2
 Hash/Index Lookup ? 2. Will there be Hierarchical network ?  How much
 of the Routes will be imported from external world ? 3. Will there be 
 Separate routing domain for overlay network  ? Or it will be mixed with
 external/underlay network ?
 These are all implementation specific details. Different deployments and
 network backends can implement them however they want. What we need to
 discuss now is how this model will look to the end-user and API.
 4. What will be the basic use case of this ? Thinking of L3 switching to
 support BGP-MPLS L3 VPN Scenario right from compute node ?
 I think the simplest use case is just that a provider doesn't want to deal
 with extending L2 domains all over their datacenter.
 
 On Tue, Oct 28, 2014 at 12:39 PM, A, Keshava
 keshav...@hp.commailto:keshav...@hp.com wrote: Hi Cory,
 
 Yes that is the basic question I have.
 
 OpenStack cloud  is ready to move away from Flat L2 network ?
 
 1. Every packet L3 FIB Lookup : Radix Tree Search, instead of current L2
 Hash/Index Lookup ? 2. Will there be Hierarchical network ?  How much
 of the Routes will be imported from external world ? 3. Will there be 
 Separate routing domain for overlay network  ? Or it will be mixed with
 external/underlay network ? 4. What will be the basic use case of this ?
 Thinking of L3 switching to support BGP-MPLS L3 VPN Scenario right from
 compute node ?
 
 Others can give their opinion also.
 
 Thanks  Regards,
 keshava
 
 -Original Message-
 From: Cory Benfield
 [mailto:cory.benfi...@metaswitch.commailto:cory.benfi...@metaswitch.com]
 Sent: Tuesday, October 28, 2014 10:35 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [neutron][nova] New specs on routed networking
 
 On Tue, Oct 28, 2014 at 07:44:48, A, Keshava wrote:
  Hi,
  
  Current Open-stack was built as flat network.
  
  With the introduction of the L3 lookup (by inserting the routing table
  in forwarding path) and separate 'VIF Route Type' interface:
  
  At what point of time in the packet processing  decision will be made
  to lookup FIB  during ? For each packet there will additional  FIB
  lookup ?
  
  How about the  impact on  'inter compute traffic', processed by  DVR  ?
  Here thinking  OpenStack cloud as hierarchical network instead of Flat
  network ?
 
 Keshava,
 
 It's difficult for me to answer in general terms: the proposed specs are
 general enough to allow multiple approaches to building purely-routed
 networks in OpenStack, and they may all have slightly different answers to
 some of these questions. I can, however, 

Re: [openstack-dev] [Neutron][LBaaS][Octavia] Usage Requirements

[Angus Lees]

On Tue, 28 Oct 2014 04:42:27 PM Jorge Miramontes wrote:
 Thanks for the reply Angus,
 
 DDoS attacks are definitely a concern we are trying to address here. My
 assumptions are based on a solution that is engineered for this type of
 thing. Are you more concerned with network I/O during a DoS attack or
 storing the logs? Under the idea I had, I wanted to make the amount of
 time logs are stored for configurable so that the operator can choose
 whether they want the logs after processing or not. The network I/O of
 pumping logs out is a concern of mine, however.

My primary concern was the generated network I/O, and the write bandwidth to 
storage media implied by that (not so much the accumulated volume of data).

We're in an era where 10Gb/s networking is now common for serving/loadbalancer 
infrastructure and as far as I can see the trend for networking is climbing 
more steeply that storage I/O, so it's only going to get worse.   10Gb/s of 
short-lived connections is a *lot* to try to write to reliable storage 
somewhere and later analyse.
It's a useful option for some users, but it would be a shame to have to limit 
loadbalancer throughput by the logging infrastructure just because we didn't 
have an alternative available.

I think you're right, that we don't have an obviously-correct choice here.  I 
think we need to expose both cheap sampling/polling of counters and more 
detailed logging of connections matching patterns (and indeed actual packet 
capture would be nice too).  Someone could then choose to base their billing 
on either datasource depending on their own accuracy-vs-cost-of-collection 
tradeoffs.  I don't see that either approach is going to be sufficiently 
universal to obsolete the other :(

Also: UDP.   Most providers are all about HTTP now, but there are still some 
people that need to bill for UDP, SIP, VPN, etc traffic.

 - Gus

 Sampling seems like the go-to solution for gathering usage but I was
 looking for something different as sampling can get messy and can be
 inaccurate for certain metrics. Depending on the sampling rate, this
 solution has the potential to miss spikes in traffic if you are gathering
 gauge metrics such as active connections/sessions. Using logs would be
 100% accurate in this case. Also, I'm assuming LBaaS will have events so
 combining sampling with events (CREATE, UPDATE, SUSPEND, DELETE, etc.)
 gets complicated. Combining logs with events is arguably less complicated
 as the granularity of logs is high. Due to this granularity, one can split
 the logs based on the event times cleanly. Since sampling will have a
 fixed cadence you will have to perform a manual sample at the time of
 the event (i.e. add complexity).
 
 At the end of the day there is no free lunch so more insight is
 appreciated. Thanks for the feedback.
 
 Cheers,
 --Jorge
 
 On 10/27/14 6:55 PM, Angus Lees g...@inodes.org wrote:
 On Wed, 22 Oct 2014 11:29:27 AM Robert van Leeuwen wrote:
   I,d like to start a conversation on usage requirements and have a few
   suggestions. I advocate that, since we will be using TCP and
 
 HTTP/HTTPS
 
   based protocols, we inherently enable connection logging for load
  
   balancers for several reasons:
  Just request from the operator side of things:
  Please think about the scalability when storing all logs.
  
  e.g. we are currently logging http requests to one load balanced
 
 application
 
  (that would be a fit for LBAAS) It is about 500 requests per second,
 
 which
 
  adds up to 40GB per day (in elasticsearch.) Please make sure whatever
  solution is chosen it can cope with machines doing 1000s of requests per
  second...
 
 And to take this further, what happens during DoS attack (either syn
 flood or
 full connections)?  How do we ensure that we don't lose our logging
 system
 and/or amplify the DoS attack?
 
 One solution is sampling, with a tunable knob for the sampling rate -
 perhaps
 tunable per-vip.  This still increases linearly with attack traffic,
 unless you
 use time-based sampling (1-every-N-seconds rather than 1-every-N-packets).
 
 One of the advantages of (eg) polling the number of current sessions is
 that
 the cost of that monitoring is essentially fixed regardless of the number
 of
 connections passing through.  Numerous other metrics (rate of new
 connections,
 etc) also have this property and could presumably be used for accurate
 billing
 - without amplifying attacks.
 
 I think we should be careful about whether we want logging or metrics for
 more
 accurate billing.  Both are useful, but full logging is only really
 required
 for ad-hoc debugging (important! but different).
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo

Re: [openstack-dev] [neutron][nova] New specs on routed networking

[Angus Lees]

On Wed, 29 Oct 2014 12:21:10 AM Fred Baker wrote:
 On Oct 28, 2014, at 4:59 PM, Angus Lees g...@inodes.org wrote:
  On Tue, 28 Oct 2014 09:07:03 PM Rohit Agarwalla wrote:
  Agreed. The way I'm thinking about this is that tenants shouldn't care
  what
  the underlying implementation is - L2 or L3. As long as the connectivity
  requirements are met using the model/API, end users should be fine. The
  data center network design should be an administrators decision based on
  the implementation mechanism that has been configured for OpenStack.
  
  I don't know anything about Project Calico, but I have been involved with
  running a large cloud network previously that made heavy use of L3
  overlays.
  
  Just because these points weren't raised earlier in this thread:  In my
  experience, a move to L3 involves losing:
  
  - broadcast/multicast.  It's possible to do L3 multicast/IGMP/etc, but
  that's a whole can of worms - so perhaps best to just say up front that
  this is a non-broadcast network.
  
  - support for other IP protocols.
  
  - various L2 games like virtual MAC addresses, etc that NFV/etc people
  like.
 I’m a little confused. IP supports multicast. It requires a routing
 protocol, and you have to “join” the multicast group, but it’s not out of
 the picture.

Agreed, you absolutely can do multicast and broadcast on an L3 overlay 
network.  I was just saying that IGMP support tends to be a lot more 
inconsistent and flaky across vendors compared with L2 multicast (which pretty 
much always works).

Further, if the goal of moving to routed L3 is to allow a network to span 
more geographically diverse underlying networks, then we might want to 
administratively prohibit broadcast due to its increased cost and it no longer 
being a hard requirement for basic functionality (no need for ARP/DHCP 
anymore!).

If we're foregoing an L2 abstraction and moving to L3, I was merely suggesting 
it might also be reasonable to say that broadcast/multicast are not supported 
and thus the requirements on the underlying infrastructure can be drastically 
reduced.  Non-broadcast L3 overlay networks are common and prove to be useful 
for just about every task except mDNS/WINS discovery, which everyone is rather 
happy to leave behind ;)

 What other “IP” protocols do you have in mind? Are you thinking about
 IPX/CLNP/etc? Or are you thinking about new network layers?

eg: If the underlying L3 network only supported IPv4, then it would be 
impossible to run IPv6 (without yet another overlay network).  With a L2 
abstraction, theoretically any IP protocol can be used.

 I’m afraid the L2 games leave me a little cold. We have been there, such as
 with DECNET IV. I’d need to understand what you were trying to achieve
 before I would consider that a loss.

Sure, just listing it as one of the changes for completeness.

Traditional network devices often use VRRP or similar for HA failover, and 
so NFV-on-L3 would need to use some alternative (failover via overlapping BGP 
advertisements, for example, is easy and works well, so this isn't hard - just 
different).

  We gain:
  
  - the ability to have proper hierarchical addressing underneath (which is
  a
  big one for scaling a single network).  This itself is a tradeoff
  however - an efficient/strict hierarchical addressing scheme means VMs
  can't choose their own IP addresses, and VM migration is
  messy/limited/impossible.
 
 It does require some variation on a host route, and it leads us to ask about
 renumbering. The hard part of VM migration is at the application layer, not
 the network, and is therefore pretty much the same.
  - hardware support for dynamic L3 routing is generally universal, through
  a
  small set of mostly-standard protocols (BGP, ISIS, etc).
  
  - can play various L3 games like BGP/anycast, which is super useful for
  geographically diverse services.
  
  
  It's certainly a useful tradeoff for many use cases.  Users lose some
  generality in return for more powerful cooperation with the provider
  around
  particular features, so I sort of think of it like a step halfway up the
  IaaS- 
  PaaS stack - except for networking.
  
  - Gus
  
  Thanks
  Rohit
  
  From: Kevin Benton blak...@gmail.commailto:blak...@gmail.com
  Reply-To: OpenStack Development Mailing List (not for usage questions)
  openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.o
  rg
  
  Date: Tuesday, October 28, 2014 1:01 PM
  
  To: OpenStack Development Mailing List (not for usage questions)
  openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.o
  rg
  
  Subject: Re: [openstack-dev] [neutron][nova] New specs on routed
  
  networking
  
  1. Every packet L3 FIB Lookup : Radix Tree Search, instead of current L2
  Hash/Index Lookup ? 2. Will there be Hierarchical network ?  How
  much
  of the Routes will be imported from external world ? 3. Will there be
  Separate routing domain for overlay network  ? Or it will be mixed

Re: [openstack-dev] [all] Key signing at the summit?

[Angus Lees]

On Mon, 27 Oct 2014 02:45:27 PM Jeremy Stanley wrote:
 If there is interest in doing another Sassaman-Projected Method
 exercise at future events, a USB document camera would be useful to
 procure in advance

Is this what the young kids these days are calling a phone?

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

[Angus Lees]

On Wed, 15 Oct 2014 08:19:03 PM Clint Byrum wrote:
   I think it would be a good idea for containers' filesystem contents to
   be a whole distro. What's at question in this thread is what should be
   running. If we can just chroot into the container's FS and run
   apt-get/yum
   install our tools, and then nsenter and attach to the running process,
   then huzzah: I think we have best of both worlds.
 
  
 
  Erm, yes that's exactly what you can do with containers (docker, lxc, and 
  presumably any other use of containers with a private/ephemeral
  filesystem).
  
 
 The point I was trying to make is that this case was not being addressed
 by the don't run init in a container crowd. I am in that crowd, and
 thus, wanted to make the point: this is how I think it will work when
 I do finally get around to trying containers for a real workload.

So you don't need init in a container in order to do the above (chroot into a 
container and run apt-get/yum and attach gdb/whatever to a running process).
(Oh wait, perhaps you're already agreeing with that?  I'm confused, so I'm 
going to explain how in case others are curious anyway.)


You just need to find the pid of a process in the container (perhaps using 
docker inspect to go from container name - pid) and then:
 nsenter -t $pid -m -u -i -n -p -w
will give you a shell (by default) running in the already existing container.  
From there you can install whatever additional packages you want and poke at 
the existing processes from their own (inside the container) pov.  A handy 
script (posted to os-dev previously) is:

 #!/bin/sh
 # name or id
 container=$1; shift 
 pid=$(docker inspect --format '{{ .State.Pid }}' $container)
 if [ $pid -eq 0 ]; then
echo No pid found for $container - dead? 2
exit 1
 fi
 exec nsenter -t $pid -m -u -i -n -p -w $@

If the docker container is destroyed/recreated at any point then your 
modifications to the filesystem (installing additional packages) are lost, 
which 
is probably what you wanted.


An interesting variant of this approach is CoreOS's toolbox.  CoreOS is a 
very minimal OS that is designed to be just enough to run docker containers.  
Consequently it doesn't have many debugging tools available (no tcpdump, eg).  
But it has a toolbox command that is basically a simple shell script that 
pulls down and runs a full-featured generic distro (redhat by default iirc) 
and runs that in privileged mode.
Inside that generic distro container you get access to the real network 
devices and processes so you can install additional tools and tcpdump, etc as 
you wish.  When you exit, it gets cleaned up.  Works quite nicely.

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

[Angus Lees]

On Wed, 15 Oct 2014 09:51:03 AM Clint Byrum wrote:
 Excerpts from Vishvananda Ishaya's message of 2014-10-15 07:52:34 -0700:
  On Oct 14, 2014, at 1:12 PM, Clint Byrum cl...@fewbar.com wrote:
   Excerpts from Lars Kellogg-Stedman's message of 2014-10-14 12:50:48 
-0700:
   On Tue, Oct 14, 2014 at 03:25:56PM -0400, Jay Pipes wrote:
   I think the above strategy is spot on. Unfortunately, that's not how
   the
   Docker ecosystem works.
   
   I'm not sure I agree here, but again nobody is forcing you to use this
   tool.
   
   operating system that the image is built for. I see you didn't respond
   to my point that in your openstack-containers environment, you end up
   with Debian *and* Fedora images, since you use the official MySQL
   dockerhub image. And therefore you will end up needing to know
   sysadmin specifics (such as how network interfaces are set up) on
   multiple operating system distributions.  
   I missed that part, but ideally you don't *care* about the
   distribution in use.  All you care about is the application.  Your
   container environment (docker itself, or maybe a higher level
   abstraction) sets up networking for you, and away you go.
   
   If you have to perform system administration tasks inside your
   containers, my general feeling is that something is wrong.
   
   Speaking as a curmudgeon ops guy from back in the day.. the reason
   I choose the OS I do is precisely because it helps me _when something
   is wrong_. And the best way an OS can help me is to provide excellent
   debugging tools, and otherwise move out of the way.
   
   When something _is_ wrong and I want to attach GDB to mysqld in said
   container, I could build a new container with debugging tools installed,
   but that may lose the very system state that I'm debugging. So I need to
   run things inside the container like apt-get or yum to install GDB.. and
   at some point you start to realize that having a whole OS is actually a
   good thing even if it means needing to think about a few more things up
   front, such as which OS will I use? and what tools do I need
   installed
   in my containers?
   
   What I mean to say is, just grabbing off the shelf has unstated
   consequences.
  
  If this is how people are going to use and think about containers, I would
  submit they are a huge waste of time. The performance value they offer is
  dramatically outweighed by the flexibilty and existing tooling that exists
  for virtual machines. As I state in my blog post[1] if we really want to
  get value from containers, we must convert to the single application per
  container view. This means having standard ways of doing the above either
  on the host machine or in a debugging container that is as easy (or
  easier)
  than the workflow you mention. There are not good ways to do this yet, and
  the community hand-waves it away, saying things like, well you could …”.
  You could isn’t good enough. The result is that a lot of people that are
  using containers today are doing fat containers with a full os.
 
 I think we really agree.
 
 What the container universe hasn't worked out is all the stuff that the
 distros have worked out for a long time now: consistency.
 
 I think it would be a good idea for containers' filesystem contents to
 be a whole distro. What's at question in this thread is what should be
 running. If we can just chroot into the container's FS and run apt-get/yum
 install our tools, and then nsenter and attach to the running process,
 then huzzah: I think we have best of both worlds.

Erm, yes that's exactly what you can do with containers (docker, lxc, and 
presumably any other use of containers with a private/ephemeral filesystem).

To others here:

There's a lot of strongly-held opinions being expressed on this thread, and a 
number of them appear to be based on misinformation or a lack of understanding 
of the technology and goals.  I'm happy to talk over IRC/VC/whatever to anyone 
about why I think this sort of stuff is worth pursuing (and I assume there are 
plenty of others too).  I'd also suggest reading docs and/or in-person at your 
local docker/devops meetup would be a more efficient method of learning rather 
than this to-and-fro on the os-dev mailing list...

 To the container makers: consider that things can and will go wrong,
 and the answer may already exist as a traditional tool, and not be
 restart the container.
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

[Angus Lees]

On Wed, 15 Oct 2014 12:40:16 AM Fox, Kevin M wrote:
 Systemd has invested a lot of time/effort to be able to relaunch failed
 services, support spawning and maintaining unix sockets and services across
 them, etc, that you'd have to push out of and across docker containers. All
 of that can be done, but why reinvent the wheel? Like you said, pacemaker
 can be made to make it all work, but I have yet to see a way to deploy
 pacemaker services anywhere near as easy as systemd+yum makes it. (Thanks
 be to redhat. :)

You should also consider fleet, if you want a systemd-approach to containers. 
 
It's basically a cluster-wide systemd that often (but doesn't have to) 
start/restart docker containers on various hosts.

I tried it for a short while and it isn't bad.  The verbosity and 
repetitiveness of the systemd files was a bit annoying, but that would be easy 
to script away.  I did like how simple it was, and the ability to express 
dependencies between systemd entities.

Note that fleet is essentially systemd managing containers from the outside - 
not running systemd inside the container.  So in many ways it's a 
repeat/reinforcement of the same conversation we're already having.

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] on Dockerfile patterns

[Angus Lees]

On Tue, 14 Oct 2014 07:51:54 AM Steven Dake wrote:
 Angus,
 
 On 10/13/2014 08:51 PM, Angus Lees wrote:
  I've been reading a bunch of the existing Dockerfiles, and I have two
  humble requests:
  
  
  1. It would be good if the interesting code came from python
  sdist/bdists
  rather than rpms.
  
  This will make it possible to rebuild the containers using code from a
  private branch or even unsubmitted code, without having to go through a
  redhat/rpm release process first.
  
  I care much less about where the python dependencies come from. Pulling
  them from rpms rather than pip/pypi seems like a very good idea, given
  the relative difficulty of caching pypi content and we also pull in the
  required C, etc libraries for free.
  
  
  With this in place, I think I could drop my own containers and switch to
  reusing kolla's for building virtual testing environments.  This would
  make me happy.
 
 I've captured this requirement here:
 https://blueprints.launchpad.net/kolla/+spec/run-from-master
 
 I also believe it would be interesting to run from master or a stable
 branch for CD.  Unfortunately I'm still working on the nova-compute
 docker code, but if someone comes along and picks up that blueprint, i
 expect it will get implemented :)  Maybe that could be you.

Yeah I've already got a bunch of working containers that pull from master[1], 
but I've been thinking I should change that to use an externally supplied 
bdist.  The downside is you quickly end up wanting a docker container to build 
your deployment docker container.  I gather this is quite a common thing to 
do, but I haven't found the time to script it up yet.

[1] https://github.com/anguslees/kube-openstack/tree/master/docker

I could indeed work on this, and I guess I was gauging the level of enthusiasm 
within kolla for such a change.  I don't want to take time away from the 
alternative I have that already does what I need only to push uphill to get it 
integrated :/

  2. I think we should separate out run the server from do once-off
  setup.
  
  Currently the containers run a start.sh that typically sets up the
  database, runs the servers, creates keystone users and sets up the
  keystone catalog.  In something like k8s, the container will almost
  certainly be run multiple times in parallel and restarted numerous times,
  so all those other steps go against the service-oriented k8s ideal and
  are at-best wasted.
  
  I suggest making the container contain the deployed code and offer a few
  thin scripts/commands for entrypoints.  The main
  replicationController/pod _just_ starts the server, and then we have
  separate pods (or perhaps even non-k8s container invocations) that do
  initial database setup/migrate, and post- install keystone setup.
 
 The server may not start before the configuration of the server is
 complete.  I guess I don't quite understand what you indicate here when
 you say we have separate pods that do initial database setup/migrate.
 Do you mean have dependencies in some way, or for eg:
 
 glance-registry-setup-pod.yaml - the glance registry pod descriptor
 which sets up the db and keystone
 glance-registry-pod.yaml - the glance registry pod descriptor which
 starts the application and waits for db/keystone setup
 
 and start these two pods as part of the same selector (glance-registry)?
 
 That idea sounds pretty appealing although probably won't be ready to go
 for milestone #1.

So the way I do it now, I have a replicationController that starts/manages 
(eg) nova-api pods[2].  I separately have a nova-db-sync pod[3] that basically 
just runs nova-manage db sync.

I then have a simple shell script[4] that starts them all at the same time.  
The nova-api pods crash and get restarted a few times until the database has 
been appropriately configured by the nova-db-sync pod, and then they're fine 
and 
start serving.

When nova-db-sync exits successfully, the pod just sits in state terminated 
thanks to restartPolicy: onFailure.  Sometime later I can delete the 
terminated nova-db-sync pod, but it's also harmless if I just leave it or even 
if it gets occasionally re-run as part of some sort of update.


[2] 
https://github.com/anguslees/kube-openstack/blob/master/kubernetes-in/nova-api-repcon.yaml
[3] 
https://github.com/anguslees/kube-openstack/blob/master/kubernetes-in/nova-db-sync-pod.yaml
[4] https://github.com/anguslees/kube-openstack/blob/master/kubecfg-create.sh


  I'm open to whether we want to make these as lightweight/independent as
  possible (every daemon in an individual container), or limit it to one per
  project (eg: run nova-api, nova-conductor, nova-scheduler, etc all in one
  container).   I think the differences are run-time scalability and
  resource- attribution vs upfront coding effort and are not hugely
  significant either way.
  
  Post-install catalog setup we can combine into one cross-service setup
  like
  tripleO does[1].  Although k8s doesn't have explicit support

Re: [openstack-dev] [neutron][all] Naming convention for unused variables

[Angus Lees]

On Tue, 14 Oct 2014 12:28:29 PM Angus Lees wrote:
 (Context: https://review.openstack.org/#/c/117418/)
 
 I'm looking for some rough consensus on what naming conventions we want for
 unused variables in Neutron, and across the larger OpenStack python codebase
 since there's no reason for Neutron to innovate here.

So after carefully collecting and summarising all one opinion[1], we're going 
with:

 __ (double-underscore) and
 _foo (leading underscore)


[1] Next time I'll be sure to mention docker in the subject line ;)

 - Gus

 As far as I can see, there are two cases:
 
 
 1.  The I just don't care variable
 
 Eg:_, _, filename = path.rpartition('/')
 
 In python this is very commonly '_', but this conflicts with the gettext
 builtin so we should avoid it in OpenStack.
 
 Possible candidates include:
 
 a.  'x'
 b. '__'  (double-underscore)
 c. No convention
 
 
 2.  I know it is unused, but the name still serves as documentation
 
 Note this turns up as two cases: as a local, and as a function parameter.
 
 Eg:   out, _err = execute('df', path)
 
 Eg:   def makefile(self, _mode, _other):
 return self._buffer
 
 I deliberately chose that second example to highlight that the leading-
 underscore convention collides with its use for private properties.
 
 Possible candidates include:
 
 a. _foo   (leading-underscore, note collides with private properties)
 b. unused_foo   (suggested in the Google python styleguide)
 c. NOQA_foo   (as suggested in c/117418)
 d. No convention  (including not indicating that variables are known-unused)
 
 
 As with all style discussions, everyone feels irrationally attached to their
 favourite, but the important bit is to be consistent to aid readability 
 (and in this case, also to help the mechanical code checkers).
 
 Vote / Discuss / Suggest additional alternatives.

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron][all] Naming convention for unused variables

[Angus Lees]

(Context: https://review.openstack.org/#/c/117418/)

I'm looking for some rough consensus on what naming conventions we want for 
unused variables in Neutron, and across the larger OpenStack python codebase 
since there's no reason for Neutron to innovate here.

As far as I can see, there are two cases:


1.  The I just don't care variable

Eg:_, _, filename = path.rpartition('/')

In python this is very commonly '_', but this conflicts with the gettext 
builtin so we should avoid it in OpenStack.

Possible candidates include:

a.  'x'
b. '__'  (double-underscore) 
c. No convention


2.  I know it is unused, but the name still serves as documentation

Note this turns up as two cases: as a local, and as a function parameter.

Eg:   out, _err = execute('df', path)

Eg:   def makefile(self, _mode, _other):
return self._buffer

I deliberately chose that second example to highlight that the leading-
underscore convention collides with its use for private properties.

Possible candidates include:

a. _foo   (leading-underscore, note collides with private properties)
b. unused_foo   (suggested in the Google python styleguide)
c. NOQA_foo   (as suggested in c/117418)
d. No convention  (including not indicating that variables are known-unused)


As with all style discussions, everyone feels irrationally attached to their 
favourite, but the important bit is to be consistent to aid readability  (and 
in this case, also to help the mechanical code checkers).

Vote / Discuss / Suggest additional alternatives.

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [kolla] on Dockerfile patterns

[Angus Lees]

I've been reading a bunch of the existing Dockerfiles, and I have two humble 
requests:


1. It would be good if the interesting code came from python sdist/bdists 
rather than rpms.

This will make it possible to rebuild the containers using code from a private 
branch or even unsubmitted code, without having to go through a redhat/rpm 
release process first.

I care much less about where the python dependencies come from. Pulling them 
from rpms rather than pip/pypi seems like a very good idea, given the relative 
difficulty of caching pypi content and we also pull in the required C, etc 
libraries for free.


With this in place, I think I could drop my own containers and switch to 
reusing kolla's for building virtual testing environments.  This would make me 
happy.


2. I think we should separate out run the server from do once-off setup.

Currently the containers run a start.sh that typically sets up the database, 
runs the servers, creates keystone users and sets up the keystone catalog.  In 
something like k8s, the container will almost certainly be run multiple times 
in parallel and restarted numerous times, so all those other steps go against 
the service-oriented k8s ideal and are at-best wasted.

I suggest making the container contain the deployed code and offer a few thin 
scripts/commands for entrypoints.  The main replicationController/pod _just_ 
starts the server, and then we have separate pods (or perhaps even non-k8s 
container invocations) that do initial database setup/migrate, and post-
install keystone setup.

I'm open to whether we want to make these as lightweight/independent as 
possible (every daemon in an individual container), or limit it to one per 
project (eg: run nova-api, nova-conductor, nova-scheduler, etc all in one 
container).   I think the differences are run-time scalability and resource-
attribution vs upfront coding effort and are not hugely significant either way.

Post-install catalog setup we can combine into one cross-service setup like 
tripleO does[1].  Although k8s doesn't have explicit support for batch tasks 
currently, I'm doing the pre-install setup in restartPolicy: onFailure pods 
currently and it seems to work quite well[2].

(I'm saying post install catalog setup, but really keystone catalog can 
happen at any point pre/post aiui.)

[1] 
https://github.com/openstack/tripleo-incubator/blob/master/scripts/setup-endpoints
[2] 
https://github.com/anguslees/kube-openstack/blob/master/kubernetes-in/nova-db-sync-pod.yaml

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] kolla, keystone, and endpoints (oh my!)

[Angus Lees]

On Mon, 6 Oct 2014 10:33:21 AM Lars Kellogg-Stedman wrote:
 Hello all,
 
 I wanted to expand on a discussion we had in #tripleo on Friday
 regarding Kubernetes and the Keystone service catalog.
 
 ## The problem
 
 The essential problem is that Kubernetes and Keystone both provide
 service discovery mechanisms, and they are not aware of each other.
 
 Kubernetes provides information about available services through
 Docker environment variables, whereas Keystone maintains a list of API
 endpoints in a database.

What you haven't stated here is whether the catalog endpoints should be 
reachable outside the kubernetes minions or not.

Perhaps we could even use this mysterious(*) keystone publicURL/internalURL 
division to publish different external and kubernetes-only versions, since we 
can presumably always do more efficient communication pod-pod.


(*) Is the publicURL/internalURL/adminURL split documented anywhere?  Does 
anything other than keystone-manage use anything other than publicURL?

 When you configure the keystone endpoints, it is tempting to simply
 use the service environment variables provided by Kubernetes, but this
 is problematic: the variables point at the host-local kube-proxy
 instance.  This is a problem right now because if that host were to
 go down, that endpoint would no longer be available.  That will be a
 problem in the near future because the proxy address will soon be
 pod-local, and thus inaccessible from other pods (even on the same
 host).
 
 One could instrument container start scripts to replace endpoints in
 keystone when the container boots, but if a service has cached
 information from the catalog it may not update in a timely fashion.
 
 ## The (?) solution
 
 I spent some time this weekend experimenting with setting up a
 pod-local proxy that takes all the service information provided by
 Kubernetes and generates an haproxy configuration (and starts haproxy
 with that configuration):

This sounds good.

Some alternatives off the top of my head, just to feed the discussion:

1.  Fixed hostname

Add something like this to the start.sh wrapper script:
 echo $SERVICE_HOST proxy  /etc/hosts
and then use http://proxy:$port/... etc as the endpoint in keystone catalog.

2. Fixed IP addresses

Create a regular OpenStack loadbalancer and configure this (possibly publicly 
available) IP in keystone catalog.

I _think_ this could even be a loadbalancer controlled by the neutron we just 
set up, assuming the the loadbalancer HA works without help and the nova-
neutron bootstrap layer was setup using regular k8s service env vars and 
not the loadbalancer IPs.

Variant a)

Create an external proxy kubernetes job that adds itself to a regular 
OpenStack loadbalancer and forwards traffic to the local kubernetes proxy (and 
thence to the real backend).  We can just walk the standard kubernetes 
environment variables to find configured k8s services.

Variant b)

Have an out-of-band job that periodically finds the current location of 
suitable endpoints using a list pods with label matching and then 
(re)configures a loadbalancer with the hostports of whatever it finds.
Avoids the double proxy inherent in (a) (and (c) aiui), at the cost of more 
frequent loadbalancer config churn.

Variant c)

Teach kubernetes how to configure OpenStack loadbalancers directly by adding a 
kubernetes/pkg/cloudprovider/openstack.  Then we add 
CreateExternalLoadBalancer: true to the appropriate service definitions and 
kubernetes basically does variant (a) itself.


Fwiw, I think we should do (2.c) anyway, just to keep up with the other 
providers that already exist in kubernetes.  I haven't written any go in a 
year or so, but I might even do this myself.


In case it needs to be said, I think we should watch discussions like 
https://github.com/GoogleCloudPlatform/kubernetes/issues/1161 and try to 
follow the standard kubernetes approaches as they emerge.

 - Gus

 - https://github.com/larsks/kolla/tree/larsks/hautoproxy/docker/hautoproxy
 
 This greatly simplifies the configuration of openstack service
 containers: in all cases, the remote address of another service will
 be at http://127.0.0.1/, so you can simply configure that address into
 the keystone catalog.
 
 It requires minimal configuration: you simply add the hautproxy
 container to your pod.
 
 This seems to do the right thing in all situations: if a pod is
 rescheduled on another host, the haproxy configuration will pick up
 the appropriate service environment variables for that host, and
 services inside the pod will contain to use 127.0.0.1 as the remote
 address.
 
 If you use the .json files from
 https://github.com/larsks/kolla/tree/larsks/hautoproxy, you can see
 this in action.  Specifically, if you start the services for mariadb,
 keystone, and glance, and then start the corresponding ponds, you will
 end up with functional keystone and glance services.
 
 Here's a short script that will do just that:
 
 #!/bin/sh
 
 for 

Re: [openstack-dev] [Ceilometer] Adding pylint checking of new ceilometer patches

[Angus Lees]

You can turn off lots of the refactor recommendation checks.  I've been
running pylint across neutron and it's uncovered half a dozen legitimate
bugs so far - and that's with many tests still disabled.

I agree that the defaults are too noisy, but its about the only tool that
does linting across files - pep8 for example only looks at the current file
(and not even the parse tree).
On 4 Oct 2014 03:22, Doug Hellmann d...@doughellmann.com wrote:


 On Oct 3, 2014, at 1:09 PM, Neal, Phil phil.n...@hp.com wrote:

  From: Dina Belova [mailto:dbel...@mirantis.com]
  On Friday, October 03, 2014 2:53 AM
 
  Igor,
 
  Personally this idea looks really nice to me, as this will help to avoid
  strange code being merged and not found via reviewing process.
 
  Cheers,
  Dina
 
  On Fri, Oct 3, 2014 at 12:40 PM, Igor Degtiarov
  idegtia...@mirantis.com wrote:
  Hi folks!
 
  I try too guess do we need in ceilometer checking new patches for
  critical errors with pylint?
 
  As far as I know Nova and Sahara and others have such check. Actually
  it is not checking of all project but comparing of the number of
  errors without new patch and with it, and if diff is more then 0 then
  patch are not taken.
 
  Looking a bit deeper it seems that Nova struggled with false positives
 and resorted to https://review.openstack.org/#/c/28754/ , which layers
 some historical checking of git on top of pylint's tendency to check only
 the latest commit. I can't say I'm too deeply versed in the code,  but it's
 enough to make me wonder if we want to go that direction and avoid the
 issues altogether?

 I haven’t looked at it in a while, but I’ve never been particularly
 excited by pylint. It’s extremely picky, encourages enforcing some
 questionable rules (arbitrary limits on variable name length?), and repots
 a lot of false positives. That combination tends to result in making
 writing software annoying without helping with quality in any real way.

 Doug

 
 
  I have taken as pattern Sahara's solution and proposed a patch for
  ceilometer:
  https://review.openstack.org/#/c/125906/
 
  Cheers,
  Igor Degtiarov
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
  --
  Best regards,
  Dina Belova
  Software Engineer
  Mirantis Inc.
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [kolla] What I already have for openstack/kubernetes/docker

[Angus Lees]

Sorry I couldn't make the IRC meeting.  sdake quite rightly suggested I send 
this to the broader list for dissection.

I spent yesterday templatising my k8s configs so I could publish them without 
revealing all my passwords ;)

  https://github.com/anguslees/kube-openstack


Please take a look and let me know if any of this is useful.  I think the good 
bits are:

- A simpler method of handling k8s pod routes by just using etcd and two shell 
loops to setup a poor-mans dynamic routing protocol.  For all its simplicity, 
this should scale to hundreds of nodes just fine, and a sharding hierarchy 
would be easy enough to add at that point  (see the networking portions in 
heat-kube-coreos-rax.yaml)

- Dockerfiles for nova + keystone, and a start on glance.  The structure should 
be similar for all the other control jobs that don't need to mess with 
hardware directly.  In particular, I'm experimenting with what it would be 
like if environment variables were supported directly in oslo.config files, and 
so far it looks good.

I chose to build these from git master.  I'm not sure if that's a good idea or 
not, but it's what I need to use this for dev work.   A possible improvement 
would be to base these on something like dockerfile/python-runtime.

- k8s config for keystone + nova + a start on glance.  Again, these should be a 
good model for other control jobs.

- I use heat to setup the initial deployment environment and generate all 
the passwords, and then stamp the generated values into kubernetes template 
files.  This assumes an already active undercloud, but it also removes easily 
isolated tasks like set up a mysql server and provide its address here from 
our list of problems to tackle.


I'm trying to run servers independently wherever possible, rather than 
bundling them into the same pod or container.  This gives maximum freedom with 
very little overhead (thanks to docker).  This also means my containers are 
basically dumb software distribution, without a complicated start.sh.

I don't have anything that configures keystone users or catalog yet - I was 
going to do that in a single pass that just added all the service ports some 
time after keystone was configured but not as part of each individual service.

-- 
 - Gus


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] Kolla Blueprints

[Angus Lees]

On Wed, 1 Oct 2014 09:05:23 PM Fox, Kevin M wrote:
 Has anyone figured out a way of having a floating ip like feature with
 docker so that you can have rabbitmq, mysql, or ceph mon's at fixed ip's
 and be able to migrate them around from physical host to physical host and
 still have them at fixed locations that you can easily put in static config
 files?

This is part of the additional functionality kubernetes adds on top of docker.

kubernetes uses a proxy on every host which knows about all the published 
services.  The services share a port space (ie: every service has to have a 
unique port assigned), and the proxies know where to forward requests to find 
one of the backends for that service.

docker communicates parameters via environment variables and has a few 
standard environment variables that are used for links to other containers.  
Kubernetes also uses these link env variables but points them at the proxies 
instead of directly to the other containers.  Since oslo.config can't look up 
environment variables directly (that's something I'd like to add), I have a 
simple shell one-liner that expands environment variables in the relevant 
config files before starting the openstack server.

As a concrete example: I configure a keystone service in my kubernetes config 
and in my static config files I use values like:

   identity_uri = http://$ENV[KEYSTONE_PORT_5000_TCP_ADDR]:
$ENV[KEYSTONE_PORT_5000_TCP_PORT]/v2.0

docker/kubernetes sets those env variables to refer to the proxy on the local 
host and the port number from my service config - this information is static 
for the lifetime of that docker instance.  The proxy will reroute the requests 
dynamically to wherever the actual instances are running right now.

I hope that's enough detail - I encourage you to read the kubernetes docs 
since they have diagrams, etc that will make it much clearer than the above.

 - Gus

 Maybe iptables rules? Maybe adding another bridge? Maybe just disabling the
 docker network stack all together and binding the service to a fixed,
 static address on the host?
 
 Also, I ran across:
 http://jperrin.github.io/centos/2014/09/25/centos-docker-and-systemd/ and
 it does seem to work. I was able to get openssh-server and keystone to work
 in the same container without needing to write custom start/stop scripts.
 This kind of setup would make a nova compute container much, much easier.
 
 Thanks,
 Kevin
 
 From: Steven Dake [sd...@redhat.com]
 Sent: Wednesday, October 01, 2014 8:04 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [kolla] Kolla Blueprints
 
 On 09/30/2014 09:55 AM, Chmouel Boudjnah wrote:
 
 On Tue, Sep 30, 2014 at 6:41 PM, Steven Dake
 sd...@redhat.commailto:sd...@redhat.com wrote:
 
 I've done a first round of prioritization.  I think key things we need
 people to step up for are nova and rabbitmq containers.
 
 For the developers, please take a moment to pick a specific blueprint to
 work on.  If your already working on something, this hsould help to prevent
 duplicate work :)
 
 
 As I understand in the current implementations[1]  the containers are
 configured with a mix of shell scripts using crudini and other shell
 command. Is it the way to configure the containers? and is a deployment
 tool like Ansible (or others) is something that is planned to be used in
 the future?
 
 Chmouel
 
 Chmouel,
 
 I am not really sure what the best solution to configure the containers.  It
 is clear to me the current shell scripts are fragile in nature and do not
 handle container restart properly.  The idea of using Puppet or Ansible as
 a CM tool has been discussed with no resolution.  At the moment, I'm
 satisified with a somewhat hacky solution if we can get the containers
 operational.
 
 Regards,
 -steve
 
 
 
 
 [1] from https://github.com/jlabocki/superhappyfunshow/
 
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker

[Angus Lees]

On Thu, 25 Sep 2014 04:01:38 PM Fox, Kevin M wrote:
 Doesn't nova with a docker driver and heat autoscaling handle case 2 and 3
 for control jobs? Has anyone tried yet?

For reference, the cases were:

 - Something to deploy the code (docker / distro packages / pip install /
 etc)
 - Something to choose where to deploy
 - Something to respond to machine outages / autoscaling and re-deploy as
 necessary


I tried for a while, yes.  The problems I ran into (and I'd be interested to 
know if there are solutions to these):

- I'm trying to deploy into VMs on rackspace public cloud (just because that's 
what I have).  This means I can't use the nova docker driver, without 
constructing an entire self-contained openstack undercloud first.

- heat+cloud-init (afaics) can't deal with circular dependencies (like nova-
neutron) since the machines need to exist first before you can refer to their 
IPs.
From what I can see, TripleO gets around this by always scheduling them on the 
same machine and just using the known local IP.  Other installs declare fixed 
IPs up front - on rackspace I can't do that (easily).
I can't use loadbalancers via heat for this because the loadbalancers need to 
know the backend node addresses, which means the nodes have to exist first and 
you're back to a circular dependency.

For comparision, with kubernetes you declare the loadbalancer-equivalents 
(services) up front with a search expression for the backends.  In a second 
pass you create the backends (pods) which can refer to any of the loadbalanced 
endpoints.  The loadbalancers then reconfigure themselves on the fly to find 
the 
new backends.  You _can_ do a similar lazy-loadbalancer-reconfig thing with 
openstack too, but not with heat and not just out of the box.

- My experiences using heat for anything complex have been extremely 
frustrating.  The version on rackspace public cloud is ancient and limited, 
and quite easy to get into a state where the only fix is to destroy the entire 
stack and recreate it.  I'm sure these are fixed in newer versions of heat, but 
last time I tried I was unable to run it standalone against an arms-length 
keystone because some of the recursive heat callbacks became confused about 
which auth token to use.

(I'm sure this can be fixed, if it wasn't already just me using it wrong in the 
first place.)

- As far as I know, nothing in a heat/loadbalancer/nova stack will actually 
reschedule jobs away from a failed machine.  There's also no lazy 
discovery/nameservice mechanism, so updating IP address declarations in cloud-
configs tend to ripple through the heat config and cause all sorts of 
VMs/containers to be reinstalled without any sort of throttling or rolling 
update.


So: I think there's some things to learn from the kubernetes approach, which 
is why I'm trying to gain more experience with it.  I know I'm learning more 
about the various OpenStack components along the way too ;)

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and Manage OpenStack using Kubernetes and Docker

[Angus Lees]

On Wed, 24 Sep 2014 10:31:19 PM Alan Kavanagh wrote:
 Steven
 I have to ask what is the motivation and benefits we get from integrating
 Kubernetes into Openstack? Would be really useful if you can elaborate and
 outline some use cases and benefits Openstack and Kubernetes can gain.

I've no idea what Steven's motivation is, but here's my reasoning for going 
down a similar path:

OpenStack deployment is basically two types of software:
1. Control jobs, various API servers, etc that are basically just regular 
python wsgi apps.
2. Compute/network node agents that run under hypervisors, configure host 
networking, etc.

The 2nd group probably wants to run on baremetal and is mostly identical on 
all such machines, but the 1st group wants higher level PaaS type things.

In particular, for the control jobs you want:

- Something to deploy the code (docker / distro packages / pip install / etc)
- Something to choose where to deploy
- Something to respond to machine outages / autoscaling and re-deploy as 
necessary

These last few don't have strong existing options within OpenStack yet (as far 
as I'm aware).  Having explored a few different approaches recently, kubernetes 
is certainly not the only option - but is a reasonable contender here.


So: I certainly don't see kubernetes as competing with anything in OpenStack - 
but as filling a gap in job management with something that has a fairly 
lightweight config syntax and is relatively simple to deploy on VMs or 
baremetal.  I also think the phrase integrating kubernetes into OpenStack is 
overstating the task at hand.

The primary downside I've discovered so far seems to be that kubernetes is 
very young and still has an awkward cli, a few easy to encounter bugs, etc.

 - Gus

 From: Steven Dake [mailto:sd...@redhat.com]
 Sent: September-24-14 7:41 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [all][tripleo] New Project - Kolla: Deploy and
 Manage OpenStack using Kubernetes and Docker
 
 On 09/24/2014 10:12 AM, Joshua Harlow wrote:
 Sounds like an interesting project/goal and will be interesting to see where
 this goes.
 
 A few questions/comments:
 
 How much golang will people be exposed to with this addition?
 
 Joshua,
 
 I expect very little.  We intend to use Kubernetes as an upstream project,
 rather then something we contribute to directly.
 
 
 Seeing that this could be the first 'go' using project it will be
 interesting to see where this goes (since afaik none of the infra support
 exists, and people aren't likely to familiar with go vs python in the
 openstack community overall).
 
 What's your thoughts on how this will affect the existing openstack
 container effort?
 
 I don't think it will have any impact on the existing Magnum project.  At
 some point if Magnum implements scheduling of docker containers, we may add
 support for Magnum in addition to Kubernetes, but it is impossible to tell
 at this point.  I don't want to derail either project by trying to force
 them together unnaturally so early.
 
 
 I see that kubernetes isn't exactly a small project either (~90k LOC, for
 those who use these types of metrics), so I wonder how that will affect
 people getting involved here, aka, who has the resources/operators/other...
 available to actually setup/deploy/run kubernetes, when operators are
 likely still just struggling to run openstack itself (at least operators
 are getting used to the openstack warts, a new set of kubernetes warts
 could not be so helpful).
 
 Yup it is fairly large in size.  Time will tell if this approach will work.
 
 This is an experiment as Robert and others on the thread have pointed out
 :).
 
 Regards
 -steve
 
 
 On Sep 23, 2014, at 3:40 PM, Steven Dake
 sd...@redhat.commailto:sd...@redhat.com wrote:
 
 
 Hi folks,
 
 I'm pleased to announce the development of a new project Kolla which is
 Greek for glue :). Kolla has a goal of providing an implementation that
 deploys OpenStack using Kubernetes and Docker. This project will begin as a
 StackForge project separate from the TripleO/Deployment program code base.
 Our long term goal is to merge into the TripleO/Deployment program rather
 then create a new program.
 
 
 
 Docker is a container technology for delivering hermetically sealed
 applications and has about 620 technical contributors [1]. We intend to
 produce docker images for a variety of platforms beginning with Fedora 20.
 We are completely open to any distro support, so if folks want to add new
 Linux distribution to Kolla please feel free to submit patches :)
 
 
 
 Kubernetes at the most basic level is a Docker scheduler produced by and
 used within Google [2]. Kubernetes has in excess of 100 technical
 contributors. Kubernetes is more then just a scheduler, it provides
 additional functionality such as load balancing and scaling and has a
 significant roadmap.
 
 
 The #tripleo channel on Freenode will be used for Kolla developer and user
 

Re: [openstack-dev] [all] [clients] [keystone] lack of retrying tokens leads to overall OpenStack fragility

[Angus Lees]

On Wed, 17 Sep 2014 04:53:28 PM Duncan Thomas wrote:
 On 16 September 2014 01:28, Nathan Kinder nkin...@redhat.com wrote:
  The idea would be to leave normal tokens with a smaller validity period
  (like the current default of an hour), but also allow one-time use
  tokens to be requested.
 
 Cinder backup makes many requests to swift during a backup, one per
 chunk to be uploaded plus one or more for the metadata file.

Right, and what if the HTTP connection times out and needs to be retried. Can 
I reuse my single use token?

Also: single-use tokens scale badly since they need a strongly consistent 
validation point that in normal use requires frequent writes.

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] OpenStack bootstrapping hour - Friday Sept 19th - 3pm EST

[Angus Lees]

On Tue, 16 Sep 2014 12:04:53 AM Rochelle.RochelleGrober wrote:
 +1000
 This is *great*.  Not only for newbies, but refreshers, learning different
 approaches, putting faces to the signatures, etc.  And Mock best practices
 is a brilliant starting place for developers.
 
 I'd like to vote for a few others:
 - Development environment (different ones: PyCharms, Eclipse, IDE for Docs,
 etc) - Tracking down a bug: log searching, back tracing, etc.
 - Fixing a bug:  From assigning in Launchpad through clone, fix, git review,
 etc. - Writing an integrated test: setup, data recording/collection/clean
 tear down.

Really looking forward to this series, thanks to everyone who has a hand in 
putting them together.

In addition to the cross-project meta-topics mentioned above, I'd also love to 
see a technical overview for each of the major projects of OpenStack (in 
separate talks).  They would need to be refreshed periodically as architecture 
evolves, but that's ok.

Existing docs and presentations spend time talking about user-visible 
features, but very few talk about the implementation and internals.
It took me an embarrassingly long time and a read of devstack to even work out 
which code ran on the hosts vs on controller nodes (for example) ;)

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [clients] [keystone] lack of retrying tokens leads to overall OpenStack fragility

[Angus Lees]

On Thu, 11 Sep 2014 03:00:02 PM Duncan Thomas wrote:
 On 11 September 2014 03:17, Angus Lees g...@inodes.org wrote:
  (As inspired by eg kerberos)
  2. Ensure at some environmental/top layer that the advertised token
  lifetime exceeds the timeout set on the request, before making the
  request.  This implies (since there's no special handling in place)
  failing if the token was expired earlier than expected.
 
 We've a related problem in cinder (cinder-backup uses the user's token
 to talk to swift, and the backup can easily take longer than the token
 expiry time) which could not be solved by this, since the time the
 backup takes is unknown (compression, service and resource contention,
 etc alter the time by multiple orders of magnitude)

Yes, this sounds like another example of the cross-service problem I was 
describing with refreshing the token at the bottom layer - but I disagree that 
this is handled any better by refreshing tokens on-demand at the bottom layer.

In order to have cinder refresh the token while talking to swift, it needs to 
know the user's password (ouch - why even have the token) or have magic token 
creating powers (in which case none of this matters, because cinder can just 
create tokens any time it wants).

As far as I can see, we either need to be able to 1) generate tokens that _do_ 
last long enough, 2) pass user+password to cinder so it is capable of 
creating new tokens as necessary, or 3) only perform token-based auth once at 
the start of a long cinder-glance workflow like this, and then use some sort 
of limited-scope-but-unlimited-time session token for follow-on requests.

I think I'm advocating for (1) or (3), and (2) as a distant third.


... Unless there's some other option here?  Your dismissal above sounded like 
there was already a solution for this - what's the current solution?

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [clients] [keystone] lack of retrying tokens leads to overall OpenStack fragility

[Angus Lees]

On Thu, 11 Sep 2014 03:21:52 PM Steven Hardy wrote:
 On Wed, Sep 10, 2014 at 08:46:45PM -0400, Jamie Lennox wrote:
  For service to service communication there are two types.
  1) using the user's token like nova-cinder. If this token expires there
  is really nothing that nova can do except raise 401 and make the client
  do it again. 2) using a service user like nova-neutron. This should
  allow automatic reauthentication and will be fixed/standardied by
  sessions.
 (1) is the problem I'm trying to solve in bug #1306294, and (for Heat at
 least) there seems to be two solutions, neither of which I particularly
 like:
 
 - Require username/password to be passed into the service (something we've
   been trying to banish via migrating to trusts for deferred
   authentication)
 - Create a trust, and impersonate the user for the duration of the request,
   or after the token expires until it is completed, using the service user
   credentials and the trust_id.
 
 It's the second one which I'm deliberating over - technically it will work,
 and we create the trust anyway (e.g for later use to do autoscaling etc),
 but can anyone from the keystone team comment on the legitimacy of the
 approach?
 
 Intuitively it seems wrong, but I can't see any other way if we want to
 support token-only auth and cope with folks doing stuff which takes 2 hours
 with a 1 hour token expiry?

A possible 3rd option is some sort of longer lived, but limited scope 
capability token.

The user would create a capability token that represents anyone possessing 
this token is (eg) allowed to write to swift as $user.  The token could be 
created by keystone as a trusted 3rd party or by swift (doesn't matter which), 
in response to a request authenticated as $user.  The client then includes 
that token in the request *to cinder*, so cinder can pass it back to swift 
when doing the writes.
This capability token would be of much longer duration (long enough to 
complete the cinder-swift task), which is ok because it is of a much more 
limited scope (ideally as fine grained as we can bother implementing).

(I like this option)


A 4th option is to have much longer lived tokens everywhere (long enough for 
this backup), but the user is able to expire it early via keystone whenever 
they feel it might be compromised (aiui this is exactly how things work now - 
we just need to increase the timeout).  Greater exposure to replay attacks, 
but if detected they can still be invalidated quickly.

(This is the easiest option, it's basically just formalising what the 
operators are already doing)


A 5th option (wow) is to have the end user/client repeatedly push in fresh 
tokens during long-running operations (and heat is the uber-example since it 
basically wants to impersonate the user forever).  Those tokens would then 
need to be refreshed all the way down the stack for any outstanding operations 
that might need the new token.

(This or the 4th option seems ugly but unavoidable for forever services like 
heat.  There has to be some way to invalidate their access if they go rogue, 
either by time (and thus needs a refresh mechanism) or by invalidation-via-
keystone (which implies the token lasts forever unless invalidated))


However we do it:  the permission to do the action should come from the 
original user - and this is expressed as tokens coming from the original 
client/user in some form.   By allowing services to create something without 
the original client/user being involved, we're really just bypassing the token 
authentication mechanism (and there are easier ways to ignore the token ;)

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] i need some help on this bug Bug #1365892

[Angus Lees]

On Wed, 10 Sep 2014 01:04:02 PM Mike Bayer wrote:
 In this case it appears to be a “safety” in case someone uses the
 ConnectionContext object outside of being a context manager.  I’d fix that
 and require that it be used as a context manager only.

Oh look, I have a new pylint hammer that is designed for exactly this nail:

https://review.openstack.org/#/c/120320/

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Angus Lees]

On Fri, 12 Sep 2014 01:08:04 PM Doug Hellmann wrote:
 On Sep 12, 2014, at 1:03 PM, Ihar Hrachyshka ihrac...@redhat.com wrote:
  Signed PGP part
  
  On 12/09/14 17:30, Mike Bayer wrote:
   On Sep 12, 2014, at 10:40 AM, Ihar Hrachyshka ihrac...@redhat.com
   
   wrote:
   Signed PGP part On 12/09/14 16:33, Mike Bayer wrote:
   I agree with this, changing the MySQL driver now is not an
   option.
   
   That was not the proposal. The proposal was to introduce support
   to run against something different from MySQLdb + a gate job for
   that alternative. The next cycle was supposed to do thorough
   regression testing, benchmarking, etc. to decide whether we're ok
   to recommend that alternative to users.
   
   ah, well that is a great idea.  But we can have that throughout
   Kilo anyway, why not ?
  
  Sure, it's not the end of the world. We'll just need to postpone work
  till RC1 (=opening of master for new stuff), pass spec bureauracy
  (reapplying for kilo)... That's some burden, but not tragedy.
  
  The only thing that I'm really sad about is that Juno users won't be
  able to try out that driver on their setup just to see how it works,
  so it narrows testing base to gate while we could get some valuable
  deployment feedback in Juno already.
 
 It’s all experimental, right? And implemented in libraries? So those users
 could update oslo.db and sqlalchemy-migrate and test the results under
 Juno.

Note that it's also theoretically possible to run the migrate with mysqldb and 
the regular production service (post-migrate) with an alternate mysql driver 
that won't deadlock eventlet...  (ie: there's no reason the mysql driver 
choice needs to be universal and simultaneous)


I'm sad that we (as a project) still haven't been able to make this 
technically trivial fix - or even make it an option for testing - after the 
original problem was identified and the fix proposed 2.5+ months ago.  I'm 
encouraged to see various meta-threads popping up discussing issues with our 
development model and hopefully we can do better in future :(

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [clients] [keystone] lack of retrying tokens leads to overall OpenStack fragility

[Angus Lees]

On Wed, 10 Sep 2014 10:14:32 AM Sean Dague wrote:
 Going through the untriaged Nova bugs, and there are a few on a similar
 pattern:
 
 Nova operation in progress takes a while
 Crosses keystone token expiration time
 Timeout thrown
 Operation fails
 Terrible 500 error sent back to user
 
 It seems like we should have a standard pattern that on token expiration
 the underlying code at least gives one retry to try to establish a new
 token to complete the flow, however as far as I can tell *no* clients do
 this.

Just because this came up in conversation a few weeks ago in the context of 
the ironic client.  I've read some docs and written a keystone client, but I'm 
not super-familiar with keystone internals - apologies if I miss something 
fundamental.


There are two broadly different approaches to dealing with this:

(As described by Sean, and implemented in a few clients)
1. At the bottom layer, try to refresh the token and immediately retry 
whenever a server response indicates the token has expired.

(As inspired by eg kerberos)
2. Ensure at some environmental/top layer that the advertised token lifetime 
exceeds the timeout set on the request, before making the request.  This 
implies (since there's no special handling in place) failing if the token was 
expired earlier than expected.


The primary distinction being that in (2) the client is ignorant of how to 
create tokens, and just assumes they're valid.

(2) is particularly easy to code for simple one shot command line clients.  
For a persistent client, the easiest approach is probably to have an 
asynchronous loop that just keeps refreshing the stored token whenever it 
approaches expiry - max_single_request_timeout.

My concern with (1) is that it involves passing username/password all the way 
down to the bottom layers - see the heat example where this means crossing 
into another program/service.  Moreover, if the token was expired earlier than 
advertised it probably means the admin has deliberately rejected the user or 
something and the intent is that they _should_ be locked out - it would be 
unfortunate to have a synchronised retry attack on keystone from all the 
rejected clients at that point :/

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Kilo Cycle Goals Exercise

[Angus Lees]

So easy/obvious it probably isn't even worth mentioning:

Drop support for python2.6

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [nova] non-deterministic gate failures due to unclosed eventlet Timeouts

[Angus Lees]

On Mon, 8 Sep 2014 05:25:22 PM Jay Pipes wrote:
 On 09/07/2014 10:43 AM, Matt Riedemann wrote:
  On 9/7/2014 8:39 AM, John Schwarz wrote:
  Hi,
  
  Long story short: for future reference, if you initialize an eventlet
  Timeout, make sure you close it (either with a context manager or simply
  timeout.close()), and be extra-careful when writing tests using
  eventlet Timeouts, because these timeouts don't implicitly expire and
  will cause unexpected behaviours (see [1]) like gate failures. In our
  case this caused non-deterministic failures on the dsvm-functional test
  suite.
  
  
  Late last week, a bug was found ([2]) in which an eventlet Timeout
  object was initialized but not closed. This instance was left inside
  eventlet's inner-workings and triggered non-deterministic Timeout: 10
  seconds errors and failures in dsvm-functional tests.
  
  As mentioned earlier, initializing a new eventlet.timeout.Timeout
  instance also registers it to inner mechanisms that exist within the
  library, and the reference remains there until it is explicitly removed
  (and not until the scope leaves the function block, as some would have
  thought). Thus, the old code (simply creating an instance without
  assigning it to a variable) left no way to close the timeout object.
  This reference remains throughout the life of a worker, so this can
  (and did) effect other tests and procedures using eventlet under the
  same process. Obviously this could easily effect production-grade
  systems with very high load.
  
  For future reference:
1) If you run into a Timeout: %d seconds exception whose traceback
  
  includes hub.switch() and self.greenlet.switch() calls, there might
  be a latent Timeout somewhere in the code, and a search for all
  eventlet.timeout.Timeout instances will probably produce the culprit.
  
2) The setup used to reproduce this error for debugging purposes is a
  
  baremetal machine running a VM with devstack. In the baremetal machine I
  used some 6 dd if=/dev/zero of=/dev/null to simulate high CPU load
  (full command can be found at [3]), and in the VM I ran the
  dsvm-functional suite. Using only a VM with similar high CPU simulation
  fails to produce the result.
  
  [1]
  http://eventlet.net/doc/modules/timeout.html#eventlet.timeout.eventlet.ti
  meout.Timeout.Timeout.cancel
  
  [2] https://review.openstack.org/#/c/119001/
  [3]
  http://stackoverflow.com/questions/2925606/how-to-create-a-cpu-spike-with
  -a-bash-command
  
  
  
  --
  John Schwarz,
  Software Engineer, Red Hat.
  
  
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
  
  Thanks, that might be what's causing this timeout/gate failure in the
  nova unit tests. [1]
  
  [1] https://bugs.launchpad.net/nova/+bug/1357578
 
 Indeed, there are a couple places where eventlet.timeout.Timeout() seems
 to be used in the test suite without a context manager or calling
 close() explicitly:
 
 tests/virt/libvirt/test_driver.py
 8925:raise eventlet.timeout.Timeout()
 
 tests/virt/hyperv/test_vmops.py
 196:mock_with_timeout.side_effect = etimeout.Timeout()

If it's useful for anyone, I wrote a quick pylint test that will catch all the 
above cases of misused context managers.

(Indeed, it will currently trigger on the raise Timeout() case, which is 
probably too eager but can be disabled in the usual #pylint meta-comment way)

Here: https://review.openstack.org/#/c/120320/ 

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Angus Lees]

On Wed, 20 Aug 2014 05:03:51 PM Clark Boylan wrote:
 On Mon, Aug 18, 2014, at 01:59 AM, Ihar Hrachyshka wrote:
  -BEGIN PGP SIGNED MESSAGE-
  Hash: SHA512
  
  On 17/08/14 02:09, Angus Lees wrote:
   On 16 Aug 2014 06:09, Doug Hellmann d...@doughellmann.com
   
   mailto:d...@doughellmann.com wrote:
   On Aug 15, 2014, at 9:29 AM, Ihar Hrachyshka
   ihrac...@redhat.com
   
   mailto:ihrac...@redhat.com wrote:
   Signed PGP part Some updates on the matter:
   
   - oslo-spec was approved with narrowed scope which is now
   'enabled mysqlconnector as an alternative in gate' instead of
   'switch the default db driver to mysqlconnector'. We'll revisit
   the switch part the next cycle once we have the new driver
   running in gate and real benchmarking is heavy-lifted.
   
   - there are several patches that are needed to make devstack
   and tempest passing deployment and testing. Those are collected
   under the hood of: https://review.openstack.org/#/c/114207/ Not
   much of them.
   
   - we'll need a new oslo.db release to bump versions (this is
   needed to set raise_on_warnings=False for the new driver, which
   was incorrectly set to True in sqlalchemy till very recently).
   This is expected to be released this month (as per Roman
   Podoliaka).
   
   This release is currently blocked on landing some changes in
   projects
   
   using the library so they don?t break when the new version starts
   using different exception classes. We?re tracking that work in
   https://etherpad.openstack.org/p/sqla_exceptions_caught
   
   It looks like we?re down to 2 patches, one for cinder
   
   (https://review.openstack.org/#/c/111760/) and one for glance
   (https://review.openstack.org/#/c/109655). Roman, can you verify
   that those are the only two projects that need changes for the
   exception issue?
   
   - once the corresponding patch for sqlalchemy-migrate is
   merged, we'll also need a new version released for this.
   
   So we're going for a new version of sqlalchemy?  (We have a
   separate workaround for raise_on_warnings that doesn't require the
   new sqlalchemy release if this brings too many other issues)
  
  Wrong. We're going for a new version of *sqlalchemy-migrate*. Which is
  the code that we inherited from Mike and currently track in stackforge.
  
   - on PyPI side, no news for now. The last time I've heard from
   Geert (the maintainer of MySQL Connector for Python), he was
   working on this. I suspect there are some legal considerations
   running inside Oracle. I'll update once I know more about
   that.
   
   If we don?t have the new package on PyPI, how do we plan to
   include it
   
   in the gate? Are there options to allow an exception, or to make
   the mirroring software download it anyway?
   
   We can test via devstack without waiting for pypi, since devstack
   will install via rpms/debs.
  
  I expect that it will be settled. I have no indication that the issue
  is unsolvable, it will just take a bit more time than we're accustomed
  to. :)
  
  At the moment, we install MySQLdb from distro packages for devstack.
  Same applies to new driver. It will be still great to see the package
  published on PyPI so that we can track its version requirements
  instead of relying on distros to package it properly. But I don't see
  it as a blocker.
  
  Also, we will probably be able to run with other drivers supported by
  SQLAlchemy once all the work is done.
 
 So I got bored last night and decided to take a stab at making PyMySQL
 work since I was a proponent of it earlier. Thankfully it did just
 mostly work like I thought it would.
 https://review.openstack.org/#/c/115495/ is the WIP devstack change to
 test this out.

Thanks!

 Postgres tests fail because it was applying the pymysql driver to the
 postgres connection string. We can clean this up later in devstack
 and/or devstack-gate depending on how we need to apply this stuff.
 Bashate failed because I had to monkeypatch in a fix for a ceilometer
 issue loading sqlalchemy drivers. The tempest neutron full job fails on
 one test occasionally. Not sure yet if that is normal neutron full
 failure mode or if a new thing from PyMySQL. The regular tempest job
 passes just fine.
 
 There are also some DB related errors in the logs that will need to be
 cleaned up but overall it just works. So I would like to repropose that
 we stop focusing all of this effort on the hard thing and use the easy
 thing if it meets our needs. We can continue to make alternatives work,
 but that is a different problem that we can solve at a different pace. I
 am not sure how to test the neutron thing that Gus was running into
 though so we should also check that really quickly.

TL;DR: pymysql passes my test case.
I'm perfectly happy to move towards using mysql+pymysql in gate tests.  (The 
various changes I've been submitting are to support _any_ non-default driver).

If anyone cares, my test case is in https://review.openstack.org

Re: [openstack-dev] [oslo] Issues with POSIX semaphores and other locks in lockutils

[Angus Lees]

On Mon, 18 Aug 2014 10:05:28 PM Pádraig Brady wrote:
 On 08/18/2014 03:38 PM, Julien Danjou wrote:
  On Thu, Aug 14 2014, Yuriy Taraday wrote:
  
  Hi Yuriy,
  
  […]
  
  Looking forward to your opinions.
  
  This looks like a good summary of the situation.
  
  I've added a solution E based on pthread, but didn't get very far about
  it for now.
 
 In my experience I would just go with the fcntl locks.
 They're auto unlocked and well supported, and importantly,
 supported for distributed processes.
 
 I'm not sure how problematic the lock_path config is TBH.
 That is adjusted automatically in certain cases where needed anyway.
 
 Pádraig.

I added a C2 (fcntl locks on byte ranges) option to the etherpad that I 
believe addresses all the issues raised.I must be missing something 
because it seems too obvious to have not been considered before :/

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Angus Lees]

On 16 Aug 2014 06:09, Doug Hellmann d...@doughellmann.com wrote:


 On Aug 15, 2014, at 9:29 AM, Ihar Hrachyshka ihrac...@redhat.com wrote:

  Signed PGP part
  Some updates on the matter:
 
  - oslo-spec was approved with narrowed scope which is now 'enabled
  mysqlconnector as an alternative in gate' instead of 'switch the
  default db driver to mysqlconnector'. We'll revisit the switch part
  the next cycle once we have the new driver running in gate and real
  benchmarking is heavy-lifted.
 
  - there are several patches that are needed to make devstack and
  tempest passing deployment and testing. Those are collected under the
  hood of: https://review.openstack.org/#/c/114207/ Not much of them.
 
  - we'll need a new oslo.db release to bump versions (this is needed to
  set raise_on_warnings=False for the new driver, which was incorrectly
  set to True in sqlalchemy till very recently). This is expected to be
  released this month (as per Roman Podoliaka).

 This release is currently blocked on landing some changes in projects
using the library so they don’t break when the new version starts using
different exception classes. We’re tracking that work in
https://etherpad.openstack.org/p/sqla_exceptions_caught

 It looks like we’re down to 2 patches, one for cinder (
https://review.openstack.org/#/c/111760/) and one for glance (
https://review.openstack.org/#/c/109655). Roman, can you verify that those
are the only two projects that need changes for the exception issue?

 
  - once the corresponding patch for sqlalchemy-migrate is merged, we'll
  also need a new version released for this.

So we're going for a new version of sqlalchemy?  (We have a separate
workaround for raise_on_warnings that doesn't require the new sqlalchemy
release if this brings too many other issues)

  - on PyPI side, no news for now. The last time I've heard from Geert
  (the maintainer of MySQL Connector for Python), he was working on
  this. I suspect there are some legal considerations running inside
  Oracle. I'll update once I know more about that.

 If we don’t have the new package on PyPI, how do we plan to include it in
the gate? Are there options to allow an exception, or to make the mirroring
software download it anyway?

We can test via devstack without waiting for pypi, since devstack will
install via rpms/debs.

- Gus
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] Which changes need accompanying bugs?

[Angus Lees]

I'm doing various small cleanup changes as I explore the neutron codebase.  
Some of these cleanups are to fix actual bugs discovered in the code.  Almost 
all of them are tiny and obviously correct.

A recurring reviewer comment is that the change should have had an 
accompanying bug report and that they would rather that change was not 
submitted without one (or at least, they've -1'ed my change).

I often didn't discover these issues by encountering an actual production 
issue so I'm unsure what to include in the bug report other than basically a 
copy of the change description.  I also haven't worked out the pattern yet of 
which changes should have a bug and which don't need one.

There's a section describing blueprints in NeutronDevelopment but nothing on 
bugs.  It would be great if someone who understands the nuances here could add 
some words on when to file bugs:
Which type of changes should have accompanying bug reports?
What is the purpose of that bug, and what should it contain?

-- 
Thanks,
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Which changes need accompanying bugs?

[Angus Lees]

On Wed, 13 Aug 2014 11:11:51 AM Kevin Benton wrote:
 Is the pylint static analysis that caught that error prone to false
 positives? If not, I agree that it would be really nice if that were made
 part of the tox check so these don't have to be fixed after the fact.

At the moment pylint on neutron is *very* noisy, and I've been looking through 
the reported issues by hand to get a feel for what's involved.  Enabling 
pylint is a separate discussion that I'd like to have - in some other thread.

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Which changes need accompanying bugs?

[Angus Lees]

On Wed, 13 Aug 2014 12:46:03 AM Kevin Benton wrote:
 I'm not sure what the guideline is, but I would like to point out a good
 reason to have the bug report even for obvious fixes.
 When users encounters bugs, they go to launchpad to report them. They don't
 first scan the commits of the master branch to see what was fixed. Having
 the bug in launchpad provides a way to track the status (fixed, backported,
 impact, etc) of the bug and reduces the chances of duplicated bugs.

Aha, I didn't realise there were people interested in the state of the code 
but following only bugs and not review/commit logs.  In particular release 
backporting.  Good to know!

 Can you provide an example of a patch that you felt was trivial that a
 reviewer requested a bug for so we have something concrete to discuss and
 establish guidelines around?

Here's one that fixes unittests in the presence of HTTP_PROXY:
https://review.openstack.org/#/c/110853/
(I see I haven't filed the bug report for that yet, I'll do that now)

.. and another that is removing an unused argument to a function:
https://review.openstack.org/#/c/80/
I _think_ the intention here was to have a discussion of implementation impact 
on the bug rather than in the review(?) but that discussion hasn't moved 
(yet).


To be clear: I'm perfectly happy to file an accompanying bug for each change 
(_every_ change if desired!).
I want to make sure my accompanying bug reports are as useful as they can be, 
so I'm trying to learn how and when they're used.  At the moment I'm basically 
waiting until a reviewer asks for one and then copying the change description 
and a prose version of the initial patch into a new bug report.

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] The gate: a failure analysis

[Angus Lees]

On Mon, 21 Jul 2014 04:39:49 PM David Kranz wrote:
 On 07/21/2014 04:13 PM, Jay Pipes wrote:
  On 07/21/2014 02:03 PM, Clint Byrum wrote:
  Thanks Matthew for the analysis.
  
  I think you missed something though.
  
  Right now the frustration is that unrelated intermittent bugs stop your
  presumably good change from getting in.
  
  Without gating, the result would be that even more bugs, many of them
  not
  intermittent at all, would get in. Right now, the one random developer
  who has to hunt down the rechecks and do them is inconvenienced. But
  without a gate, _every single_ developer will be inconvenienced until
  the fix is merged.
  
  The false negative rate is _way_ too high. Nobody would disagree there.
  However, adding more false negatives and allowing more people to ignore
  the ones we already have, seems like it would have the opposite effect:
  Now instead of annoying the people who hit the random intermittent bugs,
  we'll be annoying _everybody_ as they hit the non-intermittent ones.
  
  +10
 
 Right, but perhaps there is a middle ground. We must not allow changes
 in that can't pass through the gate, but we can separate the problems
 of constant rechecks using too many resources, and of constant rechecks
 causing developer pain. If failures were deterministic we would skip the
 failing tests until they were fixed. Unfortunately many of the common
 failures can blow up any test, or even the whole process. Following on
 what Sam said, what if we automatically reran jobs that failed in a
 known way, and disallowed recheck/reverify no bug? Developers would
 then have to track down what bug caused a failure or file a new one. But
 they would have to do so much less frequently, and as more common
 failures were catalogued it would become less and less frequent.
 
 Some might (reasonably) argue that this would be a bad thing because it
 would reduce the incentive for people to fix bugs if there were less
 pain being inflicted. But given how hard it is to track down these race
 bugs, and that we as a community have no way to force time to be spent
 on them, and that it does not appear that these bugs are causing real
 systems to fall down (only our gating process), perhaps something
 different should be considered?

So to pick an example dear to my heart, I've been working on removing these 
gate failures:
http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkxvY2sgd2FpdCB0aW1lb3V0IGV4Y2VlZGVkOyB0cnkgcmVzdGFydGluZyB0cmFuc2FjdGlvblwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDA2NTI3OTA3NzkzfQ==

.. caused by a bad interaction between eventlet and our default choice of 
mysql driver.  It would also affect any real world deployment using mysql.

The problem has been identified and the fix proposed for almost a month now, 
but 
actually fixing the gate jobs is still no-where in sight.  The fix is (pretty 
much) as easy as a pip install and a slightly modified database connection 
string.
I look forward to a discussion of the meta-issues surrounding this, but it is 
not because no-one tracked down or fixed the bug :(

 - Gus

   -David
 
  Best,
  -jay
  
  Excerpts from Matthew Booth's message of 2014-07-21 03:38:07 -0700:
  On Friday evening I had a dependent series of 5 changes all with
  approval waiting to be merged. These were all refactor changes in the
  VMware driver. The changes were:
  
  * VMware: DatastorePath join() and __eq__()
  https://review.openstack.org/#/c/103949/
  
  * VMware: use datastore classes get_allowed_datastores/_sub_folder
  https://review.openstack.org/#/c/103950/
  
  * VMware: use datastore classes in file_move/delete/exists, mkdir
  https://review.openstack.org/#/c/103951/
  
  * VMware: Trivial indentation cleanups in vmops
  https://review.openstack.org/#/c/104149/
  
  * VMware: Convert vmops to use instance as an object
  https://review.openstack.org/#/c/104144/
  
  The last change merged this morning.
  
  In order to merge these changes, over the weekend I manually submitted:
  
  * 35 rechecks due to false negatives, an average of 7 per change
  * 19 resubmissions after a change passed, but its dependency did not
  
  Other interesting numbers:
  
  * 16 unique bugs
  * An 87% false negative rate
  * 0 bugs found in the change under test
  
  Because we don't fail fast, that is an average of at least 7.3 hours in
  the gate. Much more in fact, because some runs fail on the second pass,
  not the first. Because we don't resubmit automatically, that is only if
  a developer is actively monitoring the process continuously, and
  resubmits immediately on failure. In practise this is much longer,
  because sometimes we have to sleep.
  
  All of the above numbers are counted from the change receiving an
  approval +2 until final merging. There were far more failures than this
  during the approval process.
  
  Why do we test individual changes in the gate? The 

Re: [openstack-dev] [gate] The gate: a failure analysis

[Angus Lees]

On Mon, 28 Jul 2014 10:22:07 AM Doug Hellmann wrote:
 On Jul 28, 2014, at 2:52 AM, Angus Lees g...@inodes.org wrote:
  On Mon, 21 Jul 2014 04:39:49 PM David Kranz wrote:
  On 07/21/2014 04:13 PM, Jay Pipes wrote:
  On 07/21/2014 02:03 PM, Clint Byrum wrote:
  Thanks Matthew for the analysis.
  
  I think you missed something though.
  
  Right now the frustration is that unrelated intermittent bugs stop your
  presumably good change from getting in.
  
  Without gating, the result would be that even more bugs, many of them
  not
  intermittent at all, would get in. Right now, the one random developer
  who has to hunt down the rechecks and do them is inconvenienced. But
  without a gate, _every single_ developer will be inconvenienced until
  the fix is merged.
  
  The false negative rate is _way_ too high. Nobody would disagree there.
  However, adding more false negatives and allowing more people to ignore
  the ones we already have, seems like it would have the opposite effect:
  Now instead of annoying the people who hit the random intermittent
  bugs,
  we'll be annoying _everybody_ as they hit the non-intermittent ones.
  
  +10
  
  Right, but perhaps there is a middle ground. We must not allow changes
  in that can't pass through the gate, but we can separate the problems
  of constant rechecks using too many resources, and of constant rechecks
  causing developer pain. If failures were deterministic we would skip the
  failing tests until they were fixed. Unfortunately many of the common
  failures can blow up any test, or even the whole process. Following on
  what Sam said, what if we automatically reran jobs that failed in a
  known way, and disallowed recheck/reverify no bug? Developers would
  then have to track down what bug caused a failure or file a new one. But
  they would have to do so much less frequently, and as more common
  failures were catalogued it would become less and less frequent.
  
  Some might (reasonably) argue that this would be a bad thing because it
  would reduce the incentive for people to fix bugs if there were less
  pain being inflicted. But given how hard it is to track down these race
  bugs, and that we as a community have no way to force time to be spent
  on them, and that it does not appear that these bugs are causing real
  systems to fall down (only our gating process), perhaps something
  different should be considered?
  
  So to pick an example dear to my heart, I've been working on removing
  these
  gate failures:
  http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkxvY2sgd2FpdCB0aW1lb3V0IGV
  4Y2VlZGVkOyB0cnkgcmVzdGFydGluZyB0cmFuc2FjdGlvblwiIiwiZmllbGRzIjpbXSwib2Zmc
  2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOns
  idXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDA2NTI3OTA3NzkzfQ==
  
  .. caused by a bad interaction between eventlet and our default choice of
  mysql driver.  It would also affect any real world deployment using mysql.
  
  The problem has been identified and the fix proposed for almost a month
  now, but actually fixing the gate jobs is still no-where in sight.  The
  fix is (pretty much) as easy as a pip install and a slightly modified
  database connection string.
  I look forward to a discussion of the meta-issues surrounding this, but it
  is not because no-one tracked down or fixed the bug :(
 
 I believe the main blocking issue right now is that Oracle doesn’t upload
 that library to PyPI, and so our build-chain won’t be able to download it
 as it is currently configured. I think the last I saw someone was going to
 talk to Oracle about uploading the source. Have we heard back?

Yes, positive conversations are underway and we'll get there eventually.  My 
point was also about apparent priorities, however.  If addressing gate 
failures was *urgent*, we wouldn't wait for such a conversation to complete 
before making our own workarounds(*).  I don't feel we (as a group) are 
sufficiently terrified of false negatives.

(*) Indeed, the affected devstack gate tests install mysqlconnector via 
debs/rpms.  I think only the oslo.db opportunistic tests talk to mysql via 
pip-installed packages, and these don't also use eventlet.

 - Gus

 
 Doug
 
  - Gus
  
   -David
   
  Best,
  -jay
  
  Excerpts from Matthew Booth's message of 2014-07-21 03:38:07 -0700:
  On Friday evening I had a dependent series of 5 changes all with
  approval waiting to be merged. These were all refactor changes in the
  VMware driver. The changes were:
  
  * VMware: DatastorePath join() and __eq__()
  https://review.openstack.org/#/c/103949/
  
  * VMware: use datastore classes get_allowed_datastores/_sub_folder
  https://review.openstack.org/#/c/103950/
  
  * VMware: use datastore classes in file_move/delete/exists, mkdir
  https://review.openstack.org/#/c/103951/
  
  * VMware: Trivial indentation cleanups in vmops
  https://review.openstack.org/#/c/104149/
  
  * VMware: Convert vmops to use instance as an object
  https

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Angus Lees]

Status, as I understand it:

* oslo.db changes to support other mysql drivers:

https://review.openstack.org/#/c/104425/  (merged)
https://review.openstack.org/#/c/106928/  (awaiting oslo.db review)
https://review.openstack.org/#/c/107221/  (awaiting oslo.db review)

(These are sufficient to allow operators to switch connection strings and
use mysqlconnector.  The rest is all for our testing environment)

* oslo.db change to allow testing with other mysql drivers:

https://review.openstack.org/#/c/104428/  (awaiting oslo.db review)
https://review.openstack.org/#/c/104447/  (awaiting oslo.db review.
 Ongoing discussion towards a larger rewrite of oslo.db testing instead)

* Integration into jenkins environment:

Blocked on getting Oracle to distribute mysql-connector via pypi.
Ihar and others are having conversations with the upstream author.

* Devstack change to switch to mysqlconnector for neutron:

https://review.openstack.org/#/c/105209/  (marked wip)
Ihar: do you want me to pick this up, or are you going to continue it once
some of the above has settled?

* oslo.db gate test that reproduces the deadlock with eventlet:

https://review.openstack.org/#/c/104436/  (In review.  Can't be submitted
until gate environment is switched to mysqlconnector)


Overall I'm not happy with the rate of change - but we're getting there.  I
look forward to getting this fixed :/


On 18 July 2014 21:45, Ihar Hrachyshka ihrac...@redhat.com wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA512

 On 14/07/14 17:03, Ihar Hrachyshka wrote:
  On 14/07/14 15:54, Clark Boylan wrote:
  On Sun, Jul 13, 2014 at 9:20 AM, Ihar Hrachyshka
  ihrac...@redhat.com wrote: On 11/07/14 19:20, Clark Boylan
  wrote:
  Before we get too far ahead of ourselves mysql-connector
  is not hosted on pypi. Instead it is an external package
  link. We recently managed to remove all packages that are
  hosted as external package links from openstack and will
  not add new ones in. Before we can use mysql-connector in
  the gate oracle will need to publish mysql-connector on
  pypi properly.
 
  There is misunderstanding in our community on how we deploy db
  client modules. No project actually depends on any of them. We
  assume deployers will install the proper one and configure
  'connection' string to use it. In case of devstack, we install
  the appropriate package from distribution packages, not pip.
 
  Correct, but for all of the other test suites (unittests) we
  install the db clients via pip because tox runs them and
  virtualenvs allowing site packages cause too many problems. See
 
 
 https://git.openstack.org/cgit/openstack/nova/tree/test-requirements.txt#n8
 .
 
 
 
 
 So we do actually depend on these things being pip installable.
  Basically this allows devs to run `tox` and it works.
 
  Roger that, and thanks for clarification. I'm trying to reach the
  author and the maintainer of mysqlconnector-python to see whether
  I'll be able to convince him to publish the packages on
  pypi.python.org.
 

 I've reached the maintainer of the module, he told me he is currently
 working on uploading releases directly to pypi.python.org.

 
  I would argue that we should have devstack install via pip too
  for consistency, but that is a different issue (it is already
  installing all of the other python dependencies this way so
  why special case?).
 
  What we do is recommending a module for our users in our
  documentation.
 
  That said, I assume the gate is a non-issue. Correct?
 
 
  That said there is at least one other pure python
  alternative, PyMySQL. PyMySQL supports py3k and pypy. We
  should look at using PyMySQL instead if we want to start
  with a reasonable path to getting this in the gate.
 
  MySQL Connector supports py3k too (not sure about pypy though).
 
 
  Clark
 
  On Fri, Jul 11, 2014 at 10:07 AM, Miguel Angel Ajo Pelayo
  mangel...@redhat.com wrote:
  +1 here too,
 
  Amazed with the performance gains, x2.4 seems a lot, and
  we'd get rid of deadlocks.
 
 
 
  - Original Message -
  +1
 
  I'm pretty excited about the possibilities here.  I've
  had this mysqldb/eventlet contention in the back of my
  mind for some time now. I'm glad to see some work
  being done in this area.
 
  Carl
 
  On Fri, Jul 11, 2014 at 7:04 AM, Ihar Hrachyshka
  ihrac...@redhat.com wrote:
  On 09/07/14 13:17, Ihar Hrachyshka wrote:
  Hi all,
 
  Multiple projects are suffering from db lock
  timeouts due to deadlocks deep in mysqldb
  library that we use to interact with mysql
  servers. In essence, the problem is due to
  missing eventlet support in mysqldb module,
  meaning when a db lock is encountered, the
  library does not yield to the next green thread,
  allowing other threads to eventually unlock the
  grabbed lock, and instead it just blocks the main
  thread, that eventually raises timeout exception
  (OperationalError).
 
  The failed operation is not retried, leaving
  failing request not served. In Nova, there is a
  

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

[Angus Lees]

On 10 July 2014 00:59, Roman Podoliaka rpodoly...@mirantis.com wrote:

 Not sure what issues you are talking about, but I just replaced
 mysql with mysql+mysqlconnector in my db connection string  in
 neutron.conf and neutron-db-manage upgrade head worked like a charm
 for an empty schema.


Yep, I don't think we're far away from it being that simple.  Most of the
changes/work I've seen discussed is in shifting the test environments and
suitably addressing everyone's performance/uncertainty concerns.

Ihar, could please elaborate on what changes to oslo.db are needed?
 (as an oslo.db developer I'm very interested in this part :) )


The changes I've been working on are (and most of these need oslo.db
reviews):

https://review.openstack.org/#/c/104436/
Test that concurrent sqlalchemy transactions don't block
(This test reproduces the core issue.  There's an open question where it
belongs)

https://review.openstack.org/#/c/104425/
Add DBDuplicateEntry detection for mysqlconnector driver

https://review.openstack.org/#/c/104428/
Allow tox tests with complex OS_TEST_DBAPI_CONNECTION URLs

https://review.openstack.org/#/c/104447/
Support OS_TEST_DBAPI_ADMIN_CONNECTION override

https://review.openstack.org/#/c/104430/
Don't drop pre-existing database before tests

https://github.com/zzzeek/sqlalchemy/pull/102
sqlalchemy mysql unicode improvement, that has various workarounds
available to us in the meantime

 - Gus
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Where should a test for eventlet and oslo.db interaction go?

[Angus Lees]

We have an issue with neutron (and presumably elsewhere), where mysqldb and
eventlet may deadlock, until the mysqldb deadlock timer fires.
I believe it's responsible for ~all of these failures:
http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkxvY2sgd2FpdCB0aW1lb3V0IGV4Y2VlZGVkOyB0cnkgcmVzdGFydGluZyB0cmFuc2FjdGlvblwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDA0OTcwMzgwMjc0fQ==

Now, the fix is one thing and is underway (the current favourite option is
just switching to a different mysql client library) - my question here is
instead about this test:

https://review.openstack.org/#/c/104436/

This test (as written) is against oslo.db and drives eventlet + sqlalchemy
to confirm that the current sqlalchemy driver does _not_ have the above
deadlock observed with mysqldb.  I think it (or some version of it) is an
important test, but the oslo.db guys don't want it in their testsuite since
they've purged every explicit mention of eventlet.  I'm sympathetic to this
pov.

I think we should have something like this test *somewhere*, at least as
long as we're using eventlet frequently.

I'm a bit new to openstack, so I'm lost in a maze of testing options.
 Could some kind member of the TC point to where this test *should* go?

-- 
 - Gus
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][nova] nova needs a new release of neutronclient for OverQuotaClient exception

[Angus Lees]

On Tue, 24 Jun 2014 02:46:33 PM Kyle Mestery wrote:
 On Mon, Jun 23, 2014 at 11:08 AM, Kyle Mestery
 
 mest...@noironetworks.com wrote:
  On Mon, Jun 23, 2014 at 8:54 AM, Matt Riedemann
  
  mrie...@linux.vnet.ibm.com wrote:
  There are at least two changes [1][2] proposed to Nova that use the new
  OverQuotaClient exception in python-neutronclient, but the unit test jobs
  no longer test against trunk-level code of the client packages so they
  fail. So I'm here to lobby for a new release of python-neutronclient if
  possible so we can keep these fixes moving.  Are there any issues with
  that? 
  Thanks for bringing this up Matt. I've put this on the agenda for the
  Neutron meeting today, I'll reply on this thread with what comes out
  of that discussion.
  
  Kyle
 
 As discussed in the meeting, we're going to work on making a new
 release of the client Matt. Ping me in channel later this week, we're
 working the details out on that release at the moment.

fyi, it would also make sense to include this neutronclient fix too:
 https://review.openstack.org/#/c/98318/
(assuming it gets sufficient reviews+submitted)

 Thanks,
 Kyle
 
  [1]
  https://wiki.openstack.org/wiki/Network/Meetings#Team_Discussion_Topics
  
  [1] https://review.openstack.org/#/c/62581/
  [2] https://review.openstack.org/#/c/101462/
  --
  
  Thanks,
  
  Matt Riedemann
  
  
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Distributed locking

[Angus Lees]

On Fri, 13 Jun 2014 09:40:30 AM Matthew Booth wrote:
 On 12/06/14 21:38, Joshua Harlow wrote:
  So just a few thoughts before going to far down this path,
  
  Can we make sure we really really understand the use-case where we think
  this is needed. I think it's fine that this use-case exists, but I just
  want to make it very clear to others why its needed and why distributing
  locking is the only *correct* way.
 
 An example use of this would be side-loading an image from another
 node's image cache rather than fetching it from glance, which would have
 very significant performance benefits in the VMware driver, and possibly
 other places. The copier must take a read lock on the image to prevent
 the owner from ageing it during the copy. Holding a read lock would also
 assure the copier that the image it is copying is complete.

For this particular example, taking a lock every time seems expensive.  An 
alternative would be to just try to read from another node, and if the result 
wasn't complete+valid for whatever reason then fallback to reading from 
glance.

  * What happens when a node goes down that owns the lock, how does the
  software react to this?
 
 This can be well defined according to the behaviour of the backend. For
 example, it is well defined in zookeeper when a node's session expires.
 If the lock holder is no longer a valid node, it would be fenced before
 deleting its lock, allowing other nodes to continue.
 
 Without fencing it would not be possible to safely continue in this case.

So I'm sorry for explaining myself poorly in my earlier post.  I think you've 
just described waiting for the lock to expire before another node can take it, 
which is just a regular lock behaviour.  What additional steps do you want 
Fence() to perform at this point?

(I can see if the resource provider had some form of fencing, then it could do 
all sorts of additional things - but I gather your original use case is 
exactly where that *isn't* an option)


If the lock was allowed to go stale and not released cleanly, then we should 
forcibly reboot the stale instance before allowing the lock to be held again 
shouldn't be too hard to add.

- Is this just rebooting the instance sufficient for similar situations or 
would 
we need configurable actions?
- Which bot do we trust to issue the reboot command?

From the locking service pov, I can think of several ways to implement this, 
so we probably want to export a high-level operation and allow the details to 
vary to suit the underlying locking implementation.

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Distributed locking

[Angus Lees]

On Thu, 12 Jun 2014 05:06:38 PM Julien Danjou wrote:
 On Thu, Jun 12 2014, Matthew Booth wrote:
  This looks interesting. It doesn't have hooks for fencing, though.
  
  What's the status of tooz? Would you be interested in adding fencing
  hooks?
 
 It's maintained and developer, we have plan to use it in Ceilometer and
 others projects. Joshua also wants to use it for Taskflow.
 
 We are blocked for now by https://review.openstack.org/#/c/93443/ and by
 the lack of resource to complete that request obviously, so help
 appreciated. :)
 
 As for fencing hooks, it sounds like a good idea.

As far as I understand these things, in distributed-locking-speak fencing 
just means breaking someone else's lock.

I think your options here are (and apologies if I'm repeating things that are 
obvious):

1. Have a force unlock protocol (numerous alternatives exist).  Assume the 
lock holder implements it properly and stops accessing the shared resource 
when asked.

2. Kill the lock holder using some method unrelated to the locking service and 
wait for the locking protocol to realise ex-holder is dead through usual 
liveness tests.  Assume not being able to hold the lock implies no longer able 
to access the shared resource.
The liveness test usually involves the holder pinging the lock service 
periodically, and everyone has to wait for some agreed timeout before assuming 
a client is dead.

(1) involves a lot of trust - and seems particularly bad if the reason you are 
breaking the lock is because the holder is misbehaving.
Assuming (2) is the only reasonable choice, I don't think the lock service 
needs explicit support for fencing, since the exact method for killing the 
holder is unrelated, and relatively uninteresting (probably always going to be 
an instance delete in OS).


Perhaps more interesting is exactly what conditions you require before 
attempting to kill the lock holder - you wouldn't want just any job deciding 
it was warranted, or else a misbehaving client would cause mayhem.  Again, I 
suggest your options here are:

1. Require human judgement.
ie: provide monitoring for whatever is misbehaving and make it obvious that 
one mitigation is to nuke the apparent holder.

2. Require the lock breaker to be able to reach a majority of nodes as some 
proof of I'm working, my opinion must be right.
In a paxos system, reaching a majority of nodes basically becomes hold a 
lock, we end back up with my liveness test is better than yours somehow, 
and I'm not sure how to resolve that without human judgement (but I'm not 
familiar with existing approaches).  Again, I don't think this needs 
additional support from the lock service, beyond a liveness test (which 
zookeeper, for example, has).

tl;dr: I'm interested in what sort of automated fencing behaviour you'd like.

-- 
 - Gus

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev