Re: [openstack-dev] [nova][keystone] keystoneauth1 and keystonemiddle setting

[Brant Knudson]

On Mon, Aug 14, 2017 at 2:48 AM, Chen CH Ji  wrote:

> In fixing bug 1704798, there's a proposed patch
> https://review.openstack.org/#/c/485121/7
> but we stuck at http_connection_timeout and timeout value in keystoneauth1
> and keystonemiddle repo
>
> basically we want to reuse the keystone_auth section in nova.conf to avoid
> create another section so we can
> use following to create a session
>
> sess = ks_loading.load_session_from_conf_options(CONF,
> 'keystone_authtoken', auth=context.get_auth_plugin())
>
> any comments or we have to create another section and configure it anyway?
> thanks
>
>
> Best Regards!
>
> Kevin (Chen) Ji 纪 晨
>
> Engineer, zVM Development, CSTL
> Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com
> Phone: +86-10-82451493 <+86%2010%208245%201493>
> Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
> Beijing 100193, PRC
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
I think reusing the keystone_authtoken config is a bad idea.
keystone_authtoken contains the configuration for the auth_token middleware
so this is what we keystone developers expect it to be used for. A
deployment may have different security needs for the auth_token middleware
vs checking quotas in which case they'll need different users or project
for the auth_token middleware and quota checking. And even if we don't need
it now we might need it in the future, and it's going to create a lot of
work going forward to rearchitect.

If a deployer wants to use the same authentication for both auth_token
middleware and the proxy, they can create a new section with the config and
point both keystone_authtoken and quota checking to it (by setting the
auth_section).

-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] [tc] Nominatimg Eric Fried for service-types-authority core

[Brant Knudson]

On Tue, Aug 8, 2017 at 12:35 PM, Doug Hellmann 
wrote:

> Excerpts from Chris Dent's message of 2017-08-08 16:35:18 +0100:
> >
> > I don't believe we have an established procedure for core
> > nominations in the service-types-authority group, so I'll just go
> > ahead and take the initiative. I think we should make Eric Fried
> > (efried on IRC) a core in that group. He's been doing a great deal
> > of work related to service types and service catalog, is around all the
> > time, and would be a worthy addition.
> >
> > If you don't like this idea but would like to say so privately,
> > please contact me. Otherwise I'll give a few days and make it so.
>
> +1 for adding Eric.
>
> >
> > The [tc] tag is on here because the repo is considered "owned by the
> > technical committee".
> >
> > We may also wish to consider removing Anne Gentle and Brant Knudson
> > who are less available these days.
> >
>
> I don't have a problem with removing them, but it would be good to
> confirm whether they expect to have more time in queens before we do.
>
> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

I don't even have time to keep up with emails to -dev lately, so go ahead
and remove me as a core reviewer here. I likely wouldn't work on this going
forward since I'm not even sure why we have a service catalog. Shouldn't we
be using something used throughout the industry (consul, for example)?

 - Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][nova] Persistent application credentials

[Brant Knudson]

On Thu, Jul 20, 2017 at 8:02 PM, Zane Bitter  wrote:

>
> * If Keystone supported either a public-key or a Kerberos-style
> authentication mechanism to get a token


Keystone (via support for accepting authentication from the web server
hosting it) can be configured to accept X.509 and kerberos, see
http://www.jamielennox.net/blog/2015/02/12/step-by-step-kerberized-keystone/

-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone][Mistral][Devstack] Confusion between auth_url and auth_uri in keystone middleware

[Brant Knudson]

On Thu, Jun 15, 2017 at 1:12 PM, Harry Rybacki  wrote:

> On Thu, Jun 15, 2017 at 1:57 PM, Brant Knudson  wrote:
> >
> >
> > On Thu, Jun 15, 2017 at 5:14 AM, Mikhail Fedosin 
> wrote:
> >>
> >> Recently I decided to remove deprecated parameters from
> keystone_authtoken
> >> mistral config and replace them with recommended function of devstack
> [1].
> >> In doing so, I discovered a strange behavior of configuration
> mechanism, and
> >> specifically parameters auth_uri and auth_url.
> >>
> >> 1. The parameter auth_url is not included in the list of the middleware
> >> parameters, there is auth_uri only [2]. Nevertheless, it must be
> present,
> >> because it's required by identity plugin [3]. Attempts to remove or
> replace
> >> it with the recommended auth_uri result with these stacktraces [4]
> >>
> >> 2. Even if auth_url is set, it can't be used later, because it is not
> >> registered in oslo_config [5]
> >>
> >> So I would like to get an advise from keystone team and understand what
> I
> >> should do in such cases. Official documentation doesn't add clarity on
> the
> >> matter because it recommends to use auth_uri in some cases and auth_url
> in
> >> others.
> >
> >
> > While to a human auth_uri and auth_url might look very similar they're
> > treated completely differently by auth_token / keystoneauth. One doesn't
> > replace the other in any way. So it shouldn't be surprising that
> > documentation would say to use auth_uri for one thing and auth_url for
> > something else.
> >
> In this case it's probably worth filing a docs bug against Keystone.
> If one person is confused by this, others likely are or will be.
>
> - Harry
>
>
I created a bug against keystonemiddleware:
https://bugs.launchpad.net/keystonemiddleware/+bug/1698401 . HTH.

- Brant


> >  - Brant
> >
> >
> >>
> >> My suggestion is to add auth_url in the list of keystone authtoken
> >> middleware config options, so that the parameter can be used by the
> others.
> >>
> >> Best,
> >> Mike
> >>
> >> [1] https://review.openstack.org/#/c/473796/
> >> [2]
> >> https://github.com/openstack/keystonemiddleware/blob/
> master/keystonemiddleware/auth_token/_opts.py#L31
> >> [3]
> >> https://github.com/openstack/keystoneauth/blob/master/
> keystoneauth1/loading/identity.py#L37
> >> [4] http://paste.openstack.org/show/612662/
> >> [5] http://paste.openstack.org/show/612664/
> >>
> >> 
> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> >
> > 
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone][Mistral][Devstack] Confusion between auth_url and auth_uri in keystone middleware

[Brant Knudson]

On Thu, Jun 15, 2017 at 5:14 AM, Mikhail Fedosin  wrote:

> Recently I decided to remove deprecated parameters from keystone_authtoken
> mistral config and replace them with recommended function of devstack [1].
> In doing so, I discovered a strange behavior of configuration mechanism,
> and specifically parameters auth_uri and auth_url.
>
> 1. The parameter auth_url is not included in the list of the middleware
> parameters, there is auth_uri only [2]. Nevertheless, it must be present,
> because it's required by identity plugin [3]. Attempts to remove or replace
> it with the recommended auth_uri result with these stacktraces [4]
>
> 2. Even if auth_url is set, it can't be used later, because it is not
> registered in oslo_config [5]
>
> So I would like to get an advise from keystone team and understand what I
> should do in such cases. Official documentation doesn't add clarity on the
> matter because it recommends to use auth_uri in some cases and auth_url in
> others.
>

While to a human auth_uri and auth_url might look very similar they're
treated completely differently by auth_token / keystoneauth. One doesn't
replace the other in any way. So it shouldn't be surprising that
documentation would say to use auth_uri for one thing and auth_url for
something else.

 - Brant



> My suggestion is to add auth_url in the list of keystone authtoken
> middleware config options, so that the parameter can be used by the others.
>
> Best,
> Mike
>
> [1] https://review.openstack.org/#/c/473796/
> [2] https://github.com/openstack/keystonemiddleware/blob/
> master/keystonemiddleware/auth_token/_opts.py#L31
> [3] https://github.com/openstack/keystoneauth/blob/master/
> keystoneauth1/loading/identity.py#L37
> [4] http://paste.openstack.org/show/612662/
> [5] http://paste.openstack.org/show/612664/
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][barbican][castellan] Proposal to rename Castellan to oslo.keymanager

[Brant Knudson]

On Wed, Mar 15, 2017 at 5:18 AM, Thierry Carrez 
wrote:

> Julien Danjou wrote:
> > On Tue, Mar 14 2017, Clint Byrum wrote:
> >
> >> +1 for just pulling it under the oslo umbrella but not renaming it. As
> >> much as I like the uniformity oslo.keymanager would bring, I think it's
> >> already adopted well enough we just want to make it clear that it is
> >> blessed and ok to adopt.
> >
> > I don't even get why moving it under the Oslo umbrella is a win.
> >
> > What's the current problem people are trying to solve here?
>
> It's a governance problem. Basically if the abstraction layer is under
> the control of the same group as one of the drivers, it's not really an
> abstraction layer, and nobody will adopt it or develop another driver
> for it.
>
> See Clint's first answer in the thread for a more detailed explanation.
>
> --
> Thierry Carrez (ttx)
>
>
Can the Castellan team be broken out into a new project under the big tent
rather than having to go under oslo? Oslo as a catch-all made more sense
before the big tent. Also, I always thought part of the deal of moving
under oslo is that oslo core reviewers have +2 authority on the repos, but
it doesn't look like that's part of the proposal here which was a rename
and now is changing launchpad to make Castellan a subproject under oslo
(along with some documentation changes).

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] pycrypto is dead, long live pycryptodome... or cryptography...

[Brant Knudson]

On Wed, Mar 8, 2017 at 1:03 PM, Matthew Thode 
wrote:

> So, pycrypto upstream is dead and has been for a while, we should look
> at moving off of it for both bugfix and security reasons.
>
> Currently it's used by the following.
>
> barbican, cinder, trove, glance, heat, keystoneauth, keystonemiddleware,
> kolla, openstack-ansible, and a couple of other smaller places.
>
>
keystoneauth didn't actually use pycrypto even though it was in
test-requirements.txt, so I posted a change to remove it:
https://review.openstack.org/#/c/443318/

 - Brant


> Development of it was forked into pycryptodome, which is supposed to be
> a drop in replacement.  The problem is that due to co-installability
> requirements we can't have half of packages out there using pycrypto and
> the other half using pycryptodome.  We'd need to hard switch everyone as
> both packages install into the same namespace.
>
> Another alternative would be to use something like cryptography instead,
> though it is not a drop in replacement, the migration would be able to
> be done piecemeal.
>
> I'd be interested in hearing about migration plans, especially from the
> affected projects.
>
> --
> Matthew Thode (prometheanfire)
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][api] Changing devstack to not set up keystone on :5000 and :35357

[Brant Knudson]

On Sat, Feb 25, 2017 at 1:10 PM, Lance Bragstad  wrote:

> Nice! Thanks for revisiting this, Brant.
>
> Was this a cross-project goal/discussion at the PTG?
>

I attended the cross-project discussion about the service catalog. There
might have also been some discussion at an API WG meeting that I didn't
know about until later.

- Brant


> On Fri, Feb 24, 2017 at 9:24 AM, Brant Knudson  wrote:
>
>>
>> At the PTG there was some discussion about changing services to not
>> listen on ports[0]. I'd been working on this for devstack keystone off and
>> on for some time[1], so I thought I'd dust off that work again to see what
>> the problems were. Turns out things must have been fixed since, after
>> rebasing and cleaning up the change, it's now passing our gate tests. I
>> split up the change since there was some refactoring to do before making
>> the final change of having Apache HTTPd not listen on :5000 and :35357.
>>
>> This will likely affect devstack plugins using KEYSTONE_SERVICE_PROTOCOL,
>> KEYSTONE_SERVICE_HOST, and KEYSTONE_AUTH_PORT instead of
>> KEYSTONE_SERVICE_URI[2].
>>
>> [0] https://etherpad.openstack.org/p/service-catalog-pike
>> [1] https://review.openstack.org/#/c/312230/
>> [2] https://review.openstack.org/#/c/437760/
>>
>> --
>> - Brant
>>
>>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [keystone][api] Changing devstack to not set up keystone on :5000 and :35357

[Brant Knudson]

At the PTG there was some discussion about changing services to not listen
on ports[0]. I'd been working on this for devstack keystone off and on for
some time[1], so I thought I'd dust off that work again to see what the
problems were. Turns out things must have been fixed since, after rebasing
and cleaning up the change, it's now passing our gate tests. I split up the
change since there was some refactoring to do before making the final
change of having Apache HTTPd not listen on :5000 and :35357.

This will likely affect devstack plugins using KEYSTONE_SERVICE_PROTOCOL,
KEYSTONE_SERVICE_HOST, and KEYSTONE_AUTH_PORT instead of
KEYSTONE_SERVICE_URI[2].

[0] https://etherpad.openstack.org/p/service-catalog-pike
[1] https://review.openstack.org/#/c/312230/
[2] https://review.openstack.org/#/c/437760/

-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Do we really need two listening ports ?

[Brant Knudson]

On Wed, Feb 1, 2017 at 3:54 AM, Attila Fazekas  wrote:

> Hi all,
>
> Typically we have two keystone service listening on two separate ports
> 35357 and 5000.
>
> Historically one of the port had limited functionality, but today I do not
> see why we want
> to have two separate service/port from the same code base for similar
> purposes.
>
> Effective we use double amount of memory than it is really required,
> because both port is served by completely different worker instances,
> typically from the same physical server.
>
> I wonder, would it be difficult to use only a single port or at least the
> same pool of workers for all keystone(identity, auth..) purposes?
>
> Best Regards,
> Attila
>
>
Keystone is a wsgi application and so can be run in a web server that
forwards requests under /identity to the v2 public and v3 handlers; and
/identity_admin to the v2 admin and v3 handlers. 35357 and 5000 are not a
requirement imposed by keystone.

If you run this way you might find that applications using keystone won't
work correctly in all cases since they make the assumption that they can
build identity endpoints by truncating the URL.

Another popular deployment option is to have the different keystones on
different hosts (like identity.example.com and identity_admin.example.com).

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [barbican] [security] Why are projects trying to avoid Barbican, still?

[Brant Knudson]

On Wed, Jan 18, 2017 at 9:58 AM, Dave McCowan (dmccowan)  wrote:

>
> On Mon, Jan 16, 2017 at 7:35 AM, Ian Cordasco 
> wrote:
>
>> Hi everyone,
>>
>> I've seen a few nascent projects wanting to implement their own secret
>> storage to either replace Barbican or avoid adding a dependency on it.
>> When I've pressed the developers on this point, the only answer I've
>> received is to make the operator's lives simpler.
>>
>>
> This is my opinion, but I'd like to see Keystone use Barbican for storing
> credentials. It hasn't happened yet because nobody's had the time or
> inclination (what we have works). If this happened, we could deprecate the
> current way of storing credentials and require Barbican in a couple of
> releases. Then Barbican would be a required service. The Barbican team
> might find this to be the easiest route towards convincing other projects
> to also use Barbican.
>
> - Brant
>
>
> Can you provides some details on how you'd see this work?
> Since Barbican typically uses Keystone to authenticate users before
> determining which secrets they have access to, this leads to a circular
> logic.
>
> Barbican's main purpose is a secret manager.  It supports a variety of
> RBAC and ACL access control methods to determine if a request to
> read/write/delete a secret should be allowed or not.  For secret storage,
> Barbican itself needs a secure backend for storage.  There is a
> customizable plugin interface to access secure storage.  The current
> implementations can support a database with encryption, an HSM via KMIP,
> and Dogtag.
>
>
I haven't thought about it much so don't have details figured out. Keystone
stores many types of secrets for users, and maybe you're thinking about the
user password being tricky. I'm thinking about the users' EC2 credentials
(for example). I don't think this would be difficult and would involve
creating a credentials backend for keystone that supports barbican. Maybe
have a 'keystone' project for credentials keystone is storing? If you're
familiar with the Barbican interface, compare with keystone's credential
interface[0].

[0]
http://git.openstack.org/cgit/openstack/keystone/tree/keystone/credential/backends/base.py#n26

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [barbican] [security] Why are projects trying to avoid Barbican, still?

[Brant Knudson]

On Mon, Jan 16, 2017 at 7:35 AM, Ian Cordasco 
wrote:

> Hi everyone,
>
> I've seen a few nascent projects wanting to implement their own secret
> storage to either replace Barbican or avoid adding a dependency on it.
> When I've pressed the developers on this point, the only answer I've
> received is to make the operator's lives simpler.
>
>
This is my opinion, but I'd like to see Keystone use Barbican for storing
credentials. It hasn't happened yet because nobody's had the time or
inclination (what we have works). If this happened, we could deprecate the
current way of storing credentials and require Barbican in a couple of
releases. Then Barbican would be a required service. The Barbican team
might find this to be the easiest route towards convincing other projects
to also use Barbican.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [congress][oslo.config][keystone] NoSuchOptError: no such option project_domain_name in group [keystone_authtoken]

[Brant Knudson]

On Thu, Jan 12, 2017 at 4:31 PM, Eric K  wrote:

> On a freshly stacked devstack (Jan 12), attempting to access
> `cfg.CONF.keystone_authtoken.project_domain_name` gave the
> error: NoSuchOptError: no such option project_domain_name in group
> [keystone_authtoken]
>
> I’m a little confused because it’s part of the [keystone_authtoken] config
> section generated by devstack. Could anyone point me to where these options
> are declared (I’ve searched several repos) and maybe why this option
> doesn’t exist? Thanks a lot!
>
>
These options are for the auth token middleware. Services shouldn't be
using them directly.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][i18n] how to indicate non-translatable identifiers in translatable strings?

[Brant Knudson]

On Wed, Nov 2, 2016 at 11:34 AM, Brian Rosmaita <
brian.rosma...@rackspace.com> wrote:

> This issue came up during a code review; I've asked around a bit but
> haven't been able to find an answer.
>
> Some of the help output for utility scripts associated with Glance aren't
> being translated, so Li Wei put up a patch to fix this [0], but there are
> at least two problematic cases.
>
> Case 1:
> parser.add_option('-S', '--os_auth_strategy', dest="os_auth_strategy",
> metavar="STRATEGY",
> help=_("Authentication strategy (keystone or noauth)."))
>
> For that one, 'keystone' and 'noauth' are identifiers, so we don't want
> them translated.  Would putting single quotes around them like this be
> sufficient to indicate they shouldn't be translated?  For example,
>
> help=_("Authentication strategy ('keystone' or 'noauth').")
>
>
one option is, don't put the non-translated words in the _(""); for example:

 help=_("Authentication strategy (%r or %r).") % ('keystone', 'noauth')


>
> Andreas Jaeger mentioned that maybe we could use a "translation comment"
> [1].  I guess we'd do something like:
>
> # NOTE: do not translate the stuff in single quotes
> help=_("Authentication strategy ('keystone' or 'noauth').")
>
>
> What are other people doing for this?
>
> Case 2:
> We've got a big block of usage text, roughly
>
> usage = _("""
> %prog  [options] [args]
>
> Commands:
> keyword1what it does
> keyword2what it does
> ...
> keyword8what it does
> """)
>
> We don't want the keywords to be translated, but I'm not sure how to
> convey this to the translators.
>
> Thanks in advance for your help,
> brian
>
>
> [0] https://review.openstack.org/#/c/367795/8
> [1] http://babel.pocoo.org/en/latest/messages.html#translator-comments
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Endpoint structure: a free-for-all

[Brant Knudson]

On Wed, Oct 19, 2016 at 2:27 PM, Brian Curtin  wrote:
...

>
> This started back in August when it came up that we didn't know where
> that Keystone v3 endpoint was. After talking with a few people, Steve
> Martinelli mentioned that at least as of then, hitting the unversioned
> endpoint was the way to solve that. It being unlisted anywhere was
> something for me to figure out (via that path manipulation from the
> given v2), but it was later mentioned that ideally they would like to
> have unversioned endpoints in the catalog anyway.


devstack now sets up the identity/keystone endpoints as unversioned. So you
get an endpoint with "http://192.168.122.102:5000"; for example. So this is
what we're testing with now and you're lucky if a versioned endpoint works
at all ;).


> I'm talking to Steve
> now and perhaps I took that too far in extrapolating which direction
> things were going in reality, but it was a solution that had to be
> undertaken nonetheless and was seen as the best way forward at the
> time. It's also the only one that mostly works at the moment.
>
> In the end, I'll take listing major versions as long as it's accurate
> and complete, but I'll also take listing the service root even if it
> means an extra request for me to determine those versions.
>
>
-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Adding gdavoian to oslo-core

[Brant Knudson]

On Mon, Oct 3, 2016 at 12:40 PM, Joshua Harlow 
wrote:

> Greetings all stackers,
>
> I propose that we add Gevorg Davoian[1] to the oslo-core[2] team.
>
> Gevorg has been actively contributing to oslo for a while now, both in
> helping make oslo better via code contribution(s) and by helping with the
> review load when he can. He has provided quality reviews and is doing an
> awesome job with the various oslo concepts and helping make oslo the best
> it can be!
>
> Overall I think he would make a great addition to the core review team.
>
> Please respond with +1/-1.
>
> Thanks much!
>
> - Joshua Harlow
>
> [1] https://launchpad.net/~gdavoian
> [2] https://launchpad.net/oslo
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

+1

-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Adding ozamiatin to oslo-core

[Brant Knudson]

On Mon, Oct 3, 2016 at 12:42 PM, Joshua Harlow 
wrote:

> Greetings all stackers,
>
> I propose that we add Oleksii Zamiatin[1] to the oslo-core[2] team.
>
> Oleksii has been actively contributing to oslo for a while now, both in
> helping make oslo better via code contribution(s) and by helping with the
> review load when he can. He has provided quality reviews and is doing an
> awesome job with the various oslo concepts and helping make oslo the best
> it can be!
>
> Overall I think he would make a great addition to the core review team.
>
> Please respond with +1/-1.
>
> Thanks much!
>
> - Joshua Harlow
>
> [1] https://launchpad.net/~ozamiatin
> [2] https://launchpad.net/oslo
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

+1

-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] gate-keystoneclient-dsvm-functional-ubuntu-xenial is broken

[Brant Knudson]

On Wed, Sep 21, 2016 at 8:59 AM, Rodrigo Duarte 
wrote:

> After some investigation I've found the possible issue: the functional
> tests run in parallel, some of them create and delete roles and others use
> tokens to perform the creation/update/delete of other types of fixtures.
> The problem is that when we delete a role, we also revoke *all* tokens
> from a user that has any assignment containing that role - so? Race
> condition: if we are executing a not related operation and another test
> deletes a role, the user tokens will be revoked resulting in a request
> error.
>
> The strange part is that reverting this commit [1], the tests seem to work
> fine most of the times - what makes think that commit actually *fixes* a
> big issue in our revoke events (since before it, we would not revoke such
> types of tokens).
>
> I can see a couple of options:
> - Create brand new users and role_assignments to be responsible to handle
> operations in the fixtures for each test
>

This makes sense to me. There was a bug in the tests and this corrects it.


> - Change the "framework" of the tests and rely on tempest plugins
>
>
I don't know why this was suggested.

- Brant


> What to think? Makes sense?
>
> On Tue, Sep 20, 2016 at 11:03 AM, Steve Martinelli  > wrote:
>
>> Since September 14th the keystoneclient functional test job has been
>> broken. Let's be mindful of infra resources and stop rechecking the patches
>> there. Anyone have time to investigate this?
>>
>> See patches https://review.openstack.org/#/c/369469/ or
>> https://review.openstack.org/#/c/371324/
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Rodrigo Duarte Sousa
> Senior Quality Engineer @ Red Hat
> MSc in Computer Science
> http://rodrigods.com
>
> _
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] new core reviewer (rderose)

[Brant Knudson]

On Thu, Sep 1, 2016 at 9:44 AM, Steve Martinelli 
wrote:

> I want to welcome Ron De Rose (rderose) to the Keystone core team. In a
> short time Ron has shown a very positive impact. Ron has contributed
> feature work for shadowing LDAP and federated users, as well as enhancing
> password support for SQL users. Implementing these features and picking up
> various bugs along the way has helped Ron to understand the keystone code
> base. As a result he is able to contribute to the team with quality code
> reviews.
>
> Thanks for all your hard work Ron, we sincerely appreciate it.
>
>
Welcome to the team and keep up the good work!

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Binary Package Dependencies - not only for Python

[Brant Knudson]

On Fri, Aug 12, 2016 at 2:53 PM, Jeremy Stanley  wrote:

> On 2016-08-12 21:20:34 +0200 (+0200), Julien Danjou wrote:
> [...]
> > If bindep.txt is present, are the "standard" packages still installed?
> > If yes, this is going to be more challenging to get bindep.txt right, as
> > a missing entry will go unnoticed.
>
> As Andreas mentioned, we have a fallback list[*] which gets
> installed in most (non-devstack) jobs when you don't have a
> bindep.txt or other-requirements.txt in your repo. That said, the
> addition/modification/removal of that file is accounted for in jobs
> that test a change doing that, so you can see whether it will work
> on our infrastructure simply by proposing the change to your project
> and seeing if any of your jobs fail due to missing packages.
>
> [*] http://git.openstack.org/cgit/openstack-infra/project-
> config/tree/jenkins/data/bindep-fallback.txt >
> --
> Jeremy Stanley
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

Is bindep.txt meant to be used by anything other than OpenStack CI? (As in,
are packagers going to rely on it?)

In keystone's bindep.txt, we have packages listed like:

libldap2-dev [platform:dpkg]


-> Which is only needed if you install with keystone[ldap] (see
keystone's setup.cfg[1]).


libsqlite3-dev [platform:dpkg]


-> Which is only needed for unit tests.

.. and maybe others that aren't needed in all deployments.

So there's a use case for a) integrating with extras, and b) a
"test-bindep.txt".

Maybe this is supported already or is known work to do, or maybe somebody's
looking for something to work on.

[1] http://git.openstack.org/cgit/openstack/keystone/tree/setup.cfg#n28

-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] How to enable SSL in devStack?

[Brant Knudson]

On Wed, Jul 20, 2016 at 12:29 PM, Rob Crittenden 
wrote:

> Rob Crittenden wrote:
>
>> Andrey Pavlov wrote:
>>
>>> Hi,
>>>
>>> When I ran devstack with SSL I found a bug and tried to fix it -
>>> https://review.openstack.org/#/c/242812/
>>> But no one agree with me.
>>> Try to apply this patch - it may help.
>>> Also there is a chance that new bugs present in devstack that
>>> prevented to install it with SSL.
>>>
>>
>> Seeing how some other things in your local.conf might help but when I
>> tried to reproduce it I got the same error and it failed because Apache
>> didn't have an SSL listener on 443.
>>
>> I'm not sure I'd recommend direct SSL in any case. I'd recommend the
>> tls-proxy service instead. Note that I'm pretty sure it has the same
>> problem: it hasn't been updated to handle port 443 for Keystone.
>>
>> I'm working on switching from stud to mod_proxy if you want to take a
>> look and this problem is fixed there, https://review.openstack.org/301172
>>
>> I'll see about adding a SSL listener to Keystone for the USE_SSL case in
>> the next few days.
>>
>> And yeah, it's a moving target. I have an experimental gate test for
>> tlsproxy but it has to be requested explicitly. My plan is to enable it
>> as non-voting once the mod_proxy changes land so it will at least be
>> more obvious when things break (or maybe we can making it voting).
>>
>
> Fixing Keystone is easy. An Apache VirtualHost for 443 needs to be added.
>
> But I found another, deeper problem: cinder won't listen on SSL. When they
> switched to using oslo_service for WSGI they completely removed the ability
> to use SSL. See bug https://bugs.launchpad.net/cinder/+bug/1590901
>
>
> rob
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

Problems like this should make us wonder why we're reimplementing basic
functionality like TLS termination. Existing wsgi containers (uwsgi,
gunicorn, and apache) all handle TLS termination just fine.

-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][all] Incorporating performance feedback into the review process

[Brant Knudson]

On Fri, Jun 3, 2016 at 2:35 PM, Lance Bragstad  wrote:

> Hey all,
>
> I have been curious about impact of providing performance feedback as part
> of the review process. From what I understand, keystone used to have a
> performance job that would run against proposed patches (I've only heard
> about it so someone else will have to keep me honest about its timeframe),
> but it sounds like it wasn't valued.
>
>
We had a job running rally for a year (I think) that nobody ever looked at
so we decided it was a waste and stopped running it.


> I think revisiting this topic is valuable, but it raises a series of
> questions.
>
> Initially it probably only makes sense to test a reasonable set of
> defaults. What do we want these defaults to be? Should they be determined
> by DevStack, openstack-ansible, or something else?
>
>
A performance test is going to depend on the environment (the machines,
disks, network, etc), the existing data (tokens, revocations, users, etc.),
and the config (fernet, uuid, caching, etc.). If these aren't consistent
between runs then the results are not going to be usable. (This is the
problem with running rally on infra hardware.) If the data isn't realistic
(1000s of tokens, etc.) then the results are going to be at best not useful
or at worst misleading.

What does the performance test criteria look like and where does it live?
> Does it just consist of running tempest?
>
>
I don't think tempest is going to give us numbers that we're looking for
for performance. I've seen a few scripts and have my own for testing
performance of token validation, token creation, user creation, etc. which
I think will do the exact tests we want and we can get the results
formatted however we like.

>From a contributor and reviewer perspective, it would be nice to have the
> ability to compare performance results across patch sets. I understand that
> keeping all performance results for every patch for an extended period of
> time is unrealistic. Maybe we take a daily performance snapshot against
> master and use that to map performance patterns over time?
>
>
Where are you planning to store the results?


> Have any other projects implemented a similar workflow?
>
> I'm open to suggestions and discussions because I can't imagine there
> aren't other folks out there interested in this type of pre-merge data
> points.
>
>
Thanks!
>
> Lance
>
>
Since the performance numbers are going to be very dependent on the
machines I think the only way this is going to work is if somebody's
willing to set up dedicated hardware to run the tests on. If you're doing
that then set it up to mimic how you deploy keystone, deploy the patch
under test, run the performance tests, and report the results. I'd be fine
with something like this commenting on keystone changes. None of this has
to involve openstack infra. Gerrit has a REST API to get the current
patches.

Everyone that's got performance requirements should do the same. Maybe I
can get the group I'm in to try it sometime.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon] Horizon in devstack is broken, rechecks are futile

[Brant Knudson]

On Fri, May 27, 2016 at 8:39 AM, Timur Sufiev  wrote:

> The root cause of Horizon issue has been identified and fixed at
> https://review.openstack.org/#/c/321639/
> The next steps are to release new version of django-openstack-auth library
> (which the above fix belongs to), update global-requirements (if it's not
> automatic, I'm not very into the details of release managing of openstack
> components), update horizon requirements from global requirements, and then
> merge the final patch https://review.openstack.org/#/c/321640/ - this
> time into horizon repo. Once all that is done, gate should be unblocked.
>
> Optimistic ETA is by tonight.
>
> On Wed, May 25, 2016 at 10:57 PM Timur Sufiev 
> wrote:
>
>> Dear Horizon contributors,
>>
>> The test job dsvm-integration fails for a reason for the last ~24 hours,
>> please do not recheck your patches if you see that almost all integration
>> tests fail (and only these tests) - it won't help. The fix for
>> django_openstack_auth issue which has been uncovered by the recent devstack
>> change (see https://bugs.launchpad.net/horizon/+bug/1585682) is being
>> worked on. Stay tuned, there will be another notification when rechecks
>> will become meaningful again.
>>
>
>
Thanks for working on this. It will help us eventually get to a devstack
where keystone and potentially the rest of the API servers are listening on
paths rather than on ports. I had to fix a similar issue in tempest.

To request a release send a review with to update
http://git.openstack.org/cgit/openstack/releases/tree/deliverables/mitaka/django-openstack-auth.yaml
with the new library version and commit hash. You'll have to create a new
yaml file for newton since there hasn't been a release yet.

-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] disabling deprecated APIs by config?

[Brant Knudson]

On Wed, May 18, 2016 at 10:20 AM, Sean Dague  wrote:

> nova-net is now deprecated - https://review.openstack.org/#/c/310539/
>
> And we're in the process in Nova of doing some spring cleaning and
> deprecating the proxies to other services -
> https://review.openstack.org/#/c/312209/
>
> At some point in the future after deprecation the proxy code is going to
> stop working. Either accidentally, because we're not going to test or
> fix this forever (and we aren't going to track upstream API changes to
> the proxy targets), or intentionally when we decide to delete it to make
> it easier to address core features and bugs that everyone wants addressed.
>
> However, the world moves forward slowly. Consider the following scenario.
>
> We delete nova-net & the network proxy entirely in Peru (a not entirely
> unrealistic idea). At that release there are a bunch of people just
> getting around to Newton. Their deployments allow all these things to
> happen which are going to 100% break when they upgrade, and people are
> writing more and more OpenStack software every cycle.
>
> How do we signal to users this kind of deprecation? Can we give sites
> tools to help prevent new software being written to deprecated (and
> scheduled for deletion) APIs?
>
> One idea was a "big red switch" in the format of a config option
> ``disable_deprecated_apis=True`` (defaults to False). Which would set
> all deprecated APIs to 404 routes.
>
> One of the nice ideas here is this would allow some API servers to have
> this set, and others not. So users could point to the "clean" API
> server, figure out that they will break, but the default API server
> would still support these deprecated APIs. Or, conversely, the default
> could be the clean API server, and a legacy API server endpoint could be
> provided for projects that really needed it that included these
> deprecated things for now. Either way it would allow some site assisted
> transition. And be something like the -Werror flag in gcc.
>
> In the Nova case the kinds of things ending up in this bucket are going
> to be interfaces that people *really* shouldn't be using any more. Many
> of them data back to when OpenStack was only 2 projects, and the concept
> of splitting out function wasn't really thought about (note: we're
> getting ahead of this one for the 'placement' rest API, so it won't have
> any of these issues). At some point this house cleaning was going to
> have to happen, and now seems to be the time to do get it rolling.
>
> Feedback on this idea would be welcomed. We're going to deprecate the
> proxy APIs regardless, however disable_deprecated_apis is it's own idea
> and consequences, and we really want feedback before pushing forward on
> this.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


oslo.log's fatal_deprecations configuration option[1] will raise an
exception rather than log will raise an exception. So if nova used
oslo.log.versionutils to indicate that the functionality is deprecated then
setting this switch would make it unusable. I'm not sure if anybody has
ever set this option to true.

There could be a gate job that sets deprecations to fatal which would
ensure that there's nothing running that's using deprecated functionality.

[1]
http://docs.openstack.org/developer/oslo.log/opts.html#DEFAULT.fatal_deprecations

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] supporting Go

[Brant Knudson]

I'd be worried about bringing in a language that doesn't integrate well
with Python, since I'd expect the normal route would be to take advantage
of as much of the existing code as we have and only replace those parts
that need replacing. From these web pages it looks like Go integrates with
Python: https://blog.filippo.io/building-python-modules-with-go-1-5/ and
https://github.com/go-python/gopy (I haven't tried these myself).

-- 
Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] [senlin] [keystone] [ceilometer] [telemetry] Questions about api-ref launchpad bugs

[Brant Knudson]

On Tue, May 10, 2016 at 7:53 AM, Anne Gentle 
wrote:

> Great questions, so I'm copying the -docs and -dev lists to make sure
> people know the answers.
>
> On Tue, May 10, 2016 at 5:14 AM, Atsushi SAKAI 
> wrote:
>
>> Hello Anne
>>
>>   I have several question when I am reading through etherpad's (in
>> progress).
>>   It would be appreciated to answer these questions.
>>
>> 1)Should api-ref launchpad **bugs** be moved to each modules
>>   (like keystone, nova etc)?
>>   Also, this should be applied to moved one's only or all components?
>>(compute, baremetal Ref.2)
>>
>>   Ref.
>> https://etherpad.openstack.org/p/austin-docs-newtonplan
>> API site bug list cleanup: move specific service API ref bugs to
>> project's Launchpad
>>
>>   Ref.2
>> http://developer.openstack.org/api-ref/compute/
>> http://developer.openstack.org/api-ref/baremetal/
>
>
> Yes! I definitely got agreement from nova team that they want them. Does
> anyone have a Launchpad script that could help with the bulk filter/export?
> Also, are any teams concerned about taking on their API reference bugs?
> Let's chat.
>
>
>>
>>
>> 2)Status of API-Ref
>>   a)Why keystone and senlin are no person at this moment?
>>
>>
>>
> Keystone -- after the Summit, keystone had someone sign up [1], but sounds
> like we need someone else. Brant, can you help us find someone?
>
>
I put myself down for keystone. I should be able to get to it soon enough
(if somebody else wants to take a stab at it go ahead).

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] Cruft entries found in global-requirements.txt

[Brant Knudson]

On Fri, May 6, 2016 at 1:20 PM, Davanum Srinivas  wrote:

> Folks,
>
> Thanks to Jeremy for pointing to [1]. Please see list below with
> things that are considered cruft as they don't seem to appear in
> requirements/test-requirements in projects. Some of them are clearly
> needed by us :) like libvirt-python. Others are questionable. Example
> sockjs-tornado added for Horizon ended up not being used AFAICT.
>
> Please add notes in etherpad if anyone has an idea if these are needed or
> not:
> https://etherpad.openstack.org/p/requirements-cruft
>
> Thanks,
> Dims
>
> [1]
> http://git.openstack.org/cgit/openstack/requirements/tree/tools/cruft.sh
> [2] https://review.openstack.org/#/q/topic:bp/sparklines,n,z
>
>
> ==
> XStatic-Angular-FileUpload
> XStatic-JQuery.Bootstrap.Wizard
> XStatic-Magic-Search
> XStatic-QUnit
> XenAPI
> aodhclient
> argcomplete
> botocore
> ceilometermiddleware
> dcos
> django-bootstrap-form
> extras
> fairy-slipper
> feedparser
> hgtools
> influxdb
> ironic-discoverd
> ldappool
> libvirt-python
> mimic
> netmiko
> notifier
> os-apply-config
> os-cloud-config
> os-net-config
> os-refresh-config
> posix_ipc
> pyghmi
> pylxd
> pysqlite;python_version
> python-consul
> python-ldap
> python-solumclient
> requestsexceptions
> singledispatch
> sockjs-tornado
> sphinxcontrib-blockdiag
> tripleo-image-elements
> weakrefmethod;python_version
> xmltodict
> ==
>
>
python-ldap and ldappool are in keystone's setup.cfg:
http://git.openstack.org/cgit/openstack/keystone/tree/setup.cfg#n24

-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Token providers and Fernet as the default

[Brant Knudson]

On Tue, May 3, 2016 at 3:46 PM, Clint Byrum  wrote:

> Excerpts from Morgan Fainberg's message of 2016-05-03 11:13:38 -0700:
> > On Tue, May 3, 2016 at 10:28 AM, Monty Taylor 
> wrote:
> >
> > > On 05/03/2016 11:47 AM, Clint Byrum wrote:
> > >
> > >> Excerpts from Monty Taylor's message of 2016-05-03 07:59:21 -0700:
> > >>
> > >>> On 05/03/2016 08:55 AM, Clint Byrum wrote:
> > >>>
> > 
> >  Perhaps we have different perspectives. How is accepting what we
> >  previously emitted and told the user would be valid sneaky or wrong?
> >  Sounds like common sense due diligence to me.
> > 
> > >>>
> > >>> I agree - I see no reason we can't validate previously emitted
> tokens.
> > >>> But I don't agree strongly, because re-authing on invalid token is a
> > >>> thing users do hundreds of times a day. (these aren't oauth API Keys
> or
> > >>> anything)
> > >>>
> > >>>
> > >> Sure, one should definitely not be expecting everything to always work
> > >> without errors. On this we agree for sure. However, when we do decide
> to
> > >> intentionally induce errors for reasons we have not done so before, we
> > >> should weigh the cost of avoiding that with the cost of having it
> > >> happen. Consider this strawman:
> > >>
> > >> - User gets token, it says "expires_at Now+4 hours"
> > >> - User starts a brief set of automation tasks in their system
> > >>that does not use python and has not failed with invalid tokens
> thus
> > >>far.
> > >> - Keystone nodes are all updated at one time (AMAZING cloud ops team)
> > >> - User's automation jobs fail at next OpenStack REST call
> > >> - User begins debugging, wasting hours of time figuring out that
> > >>their tokens, which they stored and show should still be valid,
> were
> > >>rejected.
> > >>
> > >
> > > Ah - I guess this is where we're missing each other, which is good and
> > > helpful.
> > >
> > > I would argue that any user that is _storing_ tokens is doing way too
> much
> > > work. If they are doing short tasks, they should just treat them as
> > > ephemeral. If they are doing longer tasks, they need to deal with
> timeouts.
> > > SO, this:
> > >
> > >
> > > - User gets token, it says "expires_at Now+4 hours"
> > > - User starts a brief set of automation tasks in their system
> > >that does not use python and has not failed with invalid tokens thus
> > >far.
> > >
> > > should be:
> > >
> > > - User starts a brief set of automation tasks in their system
> > > that does not use python and has not failed with invalid tokens thus
> > > far.
> > >
> > > "Get a token" should never be an activity that anyone ever consciously
> > > performs.
> > >
> > >
> > This is my view. Never, ever, ever assume your token is good until
> > expiration. Assume the token might be broken at any request and know how
> to
> > re-auth.
> >
> > > And now they have to refactor their app, because this may happen again,
> > >> and they have to make sure that invalid token errors can bubble up to
> the
> > >> layer that has the username/password, or accept rolling back and
> > >> retrying the whole thing.
> > >>
> > >> I'm not saying anybody has this system, I'm suggesting we're putting
> > >> undue burden on users with an unknown consequence. Falling back to
> UUID
> > >> for a while has a known cost of a little bit of code and checking junk
> > >> tokens twice.
> > >>
> > >
> > Please do not advocate "falling back" to UUID. I am actually against
> making
> > fernet the default (very, very strongly), if we have to have this
> > "fallback" code. It is the wrong kind of approach, we already have
> serious
> > issues with complex code paths that produce subtly different results. If
> > the options are:
> >
> > 1) Make Fernet Default and have "fallback" code
> >
> > or
> >
> > 2) Leave UUID default and highly recommend fernet (plus gate on fernet
> > primarily, default in devstack)
> >
> > I will jump on my soapbox and be very loudly in favor of the 2nd option.
> If
> > we communicate this is a change that will happen (hey, maybe throw an
> > error/make the config option "none" so it has to be explicit) in Newton,
> > and then move to a Fernet default in O - I'd be ok with that.
> >
> > >
> > > Totally. I have no problem with the suggestion that keystone handle
> this.
> > > But I also think that users should quite honestly stop thinking about
> > > tokens at all. Tokens are an implementation detail that if any user
> thinks
> > > about while writing their app they're setting themselves up to be
> screwed -
> > > so we should make sure we're not talking about them in a primary way
> such
> > > as to suggest that people focus a lot of energy on them.
> > >
> > > (I also frequently see users who are using python libraries even get
> > > everything horribly wrong and screw themselves because they think they
> need
> > > to think about tokens)
> > >
> >
> > Better communication that tokens are ephemeral and should not assume to
> > work always (even until thei

Re: [openstack-dev] [keystone] Token providers and Fernet as the default

[Brant Knudson]

On Mon, May 2, 2016 at 6:26 PM, Clint Byrum  wrote:

> Hello! I enjoyed very much listening in on the default token provider
> work session last week in Austin, so thanks everyone for participating
> in that. I did not speak up then, because I wasn't really sure of this
> idea that has been bouncing around in my head, but now I think it's the
> case and we should consider this.
>
> Right now, Keystones without fernet keys, are issuing UUID tokens. These
> tokens will be in the database, and valid, for however long the token
> TTL is.
>
> The moment that one changes the configuration, keystone will start
> rejecting these tokens. This will cause disruption, and I don't think
> that is fair to the users who will likely be shown new bugs in their
> code at a very unexpected moment.
>
> I wonder if one could merge UUID and Fernet into a provider which
> handles this transition gracefully:
>
> if self._fernet_keys:
>

This would have to check that there's an active fernet key and not just a
staging one. You'll want to push out a staging key to all the nodes first
to enable fernet validation before pushing out the active key to enable
token creation. Maybe there's a trick to getting keystone-manage
fernet_setup to only setup a staging key, or you just copy that key around.

Also, we could have keystone keep track of if there aren't any uuid tokens
since there's no need to query the database everytime we get an invalid
token just to see an empty table.

- Brant


>   return self._issue_fernet_token()
> else:
>   return self._issue_uuid_token()
>
> And in the validation, do the same, but also with an eye toward keeping
> the UUID tokens alive:
>
> if self._fernet_keys:

  try:
> self._validate_fernet_token()
>   except InvalidFernetFormatting:
> self._validate_uuid_token()
> else:
>   self._validate_uuid_token()
>
> So that while one is rolling out new keystone nodes and syncing fernet
> keys, all tokens issued would validated properly, with minimal extra
> cost to support both (basically just a number of UUID tokens will need
> to be parsed twice, once as Fernet, and once as UUID).
>
> Thoughts? I think doing this would make changing the default fairly
> uncontroversial.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][nova][oslo] Common backoff & timeout utils

[Brant Knudson]

On Wed, Apr 20, 2016 at 3:10 PM, Boden Russell  wrote:

> Today there are a number of places in nova, neutron and perhaps
> elsewhere that employ backoff + timeout strategies (see [1] - [4]).
> While we are working towards a unified approach in neutron for RPC [5],
> it appears such logic could benefit the greater community as a reusable
> oslo implementation.
>
> IMHO such an oslo implementation could:
> - Enable backoff/timeout irrespective of client/transport. This would
> allow the utils to be used with various clients (HTTP, AMQP RPC, etc.).
> - Support namespacing as inspired by the existing neutron patch [5].
> - In the future, perhaps, allow multiple (pluggable) backoff strategies
> to be used (e.g. configurable backoff).
>
> Anyone adverse to me crafting an initial oslo patch to kick-off the
> details on this one?
>
>
> Thanks
>
>
> [1]
> https://github.com/openstack/nova/blob/master/nova/conductor/api.py#L162
> [2]
>
> https://github.com/openstack/nova/blob/3cdaa30566c17a2add5d9163a0693c97dc1d065b/nova/scheduler/utils.py#L356
> [3]
>
> https://github.com/openstack/neutron/blob/dd4f1253c951d78a5b497680dfb31317ba469a58/neutron/agent/l3/agent.py#L224
> [4]
>
> https://github.com/openstack/neutron/blob/42c6f05f2904cd4c01bcd3f79b1966489f5ad3c1/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py#L189
> [5] https://review.openstack.org/#/c/280595/
>
>
oslo.utils has a decorator that will retry after a delay -
http://docs.openstack.org/developer/oslo.utils/api/excutils.html#oslo_utils.excutils.forever_retry_uncaught_exceptions
, which doesn't do backoff would be easy enough to add. Alternatively, if
an existing library works better, maybe the oslo utility should be
deprecated.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [python-keystoneclient] Return request-id to caller

[Brant Knudson]

On Wed, Apr 20, 2016 at 6:31 AM, Duncan Thomas 
wrote:

> On 20 April 2016 at 08:08, koshiya maho 
> wrote:
>
>
>> This design was discussed, reviewed and approved in cross-projects [1] and
>> already implemented in nova, cinder and neutron.
>> At this point if we change the implementation then it will not be
>> consistent across core OpenStack projects.
>> For maintenance of the whole of OpenStack, I think that the present
>> method is best.
>> Please suggest.
>>
>
>
I haven't been asking for a complete redesign. I just want this to be
opt-in to minimize the chance of an impact on existing applications. We can
eventually deprecate the old behavior.

- Brant



> The fact that a cross-project spec is approved doesn't mean that it will
> end up being practical. If the cinder-client implementation had been found
> to break any none-trivial users then I wouldn't have hesitated.
>
> Cross project specs are not getting massive amounts of detailed attention
> from project teams, end even they were it is not possible to foresee all
> subtle problems at review time - they should be taken as guidance not
> gospel and expect to be reworked if it proves necessary.
>
> --
> Duncan Thomas
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Problem with WSGI on keystone

[Brant Knudson]

On Tue, Apr 19, 2016 at 11:57 AM, Rosensweig, Elisha (Nokia - IL) <
elisha.rosensw...@nokia.com> wrote:

> Hi All,
>
> Recently, I've been having trouble running stack.sh from scratch. With the
> default configuration I've been using for a while, I get the following
> error in /opt/stack/logs/key.log:
>
> 2016-04-19 16:05:26.630 22047 DEBUG keystone.common.controller
> [req-3d384c1f-8b9a-4ff0-aa30-a4aad73963fb 3bdfb473e0e544a4a1889c9192d2ba3c
> fd622e15c12a49a299a63022c5d34088 - default default] RBAC: Authorization
> granted inner /opt/stack/keystone/keystone/common/controller.py:180
> mod_wsgi (pid=22045): Target WSGI script
> '/usr/local/bin/keystone-wsgi-admin' cannot be loaded as Python module.
> mod_wsgi (pid=22045): Exception occurred processing WSGI script
> '/usr/local/bin/keystone-wsgi-admin'.
> Traceback (most recent call last):
>   File "/usr/local/bin/keystone-wsgi-admin", line 36, in 
> application = initialize_admin_application()
>   File "/opt/stack/keystone/keystone/server/wsgi.py", line 62, in
> initialize_admin_application
> return initialize_application('admin')
>   File "/opt/stack/keystone/keystone/server/wsgi.py", line 39, in
> initialize_application
> common.configure()
>   File "/opt/stack/keystone/keystone/server/common.py", line 31, in
> configure
> config.configure()
>   File "/opt/stack/keystone/keystone/common/config.py", line 1131, in
> configure
> help='Do not monkey-patch threading system modules.'))
>   File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line
> 2106, in __inner
> result = f(self, *args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line
> 2282, in register_cli_opt
> raise ArgsAlreadyParsedError("cannot register CLI option")
> ArgsAlreadyParsedError: arguments already parsed: cannot register CLI
> option
>
>
This happens a lot. Especially if you reuse your instance for a while it
typically starts failing due to levels of required packages. You're looking
at the wrong place in the log to figure out what the problem is. Look for
the first error rather than this one.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Brant Knudson]

On Mon, Apr 18, 2016 at 10:20 AM, Matt Fischer  wrote:

> On Mon, Apr 18, 2016 at 8:29 AM, Brant Knudson  wrote:
>
>>
>>
>> On Fri, Apr 15, 2016 at 9:04 PM, Adam Young  wrote:
>>
>>> We all want Fernet to be a reality.  We ain't there yet (Except for
>>> mfish who has no patience) but we are getting closer.  The goal is to get
>>> Fernet as the default token provider as soon as possible. The review to do
>>> this has uncovered a few details that need to be fixed before we can do
>>> this.
>>>
>>> Trusts for V2 tokens were not working correctly.  Relatively easy fix.
>>> https://review.openstack.org/#/c/278693/ Patch is still failing on
>>> Python 3.  The tests are kindof racy due to the revocation event 1 second
>>> granularity.  Some of the tests here have A sleep (1) in them still, but
>>> all should be using the time control aspect of the unit test fixtures.
>>>
>>> Some of the tests also use the same user to validate a token as that
>>> have, for example, a role unassigned.  These expose a problem that the
>>> revocation events are catching too many tokens, some of which should not be
>>> treated as revoked.
>>>
>>> Also, some of the logic for revocation checking has to change. Before,
>>> if a user had two roles, and had one removed, the token would be revoked.
>>> Now, however, the token will validate successful, but the response will
>>> only have the single assigned role in it.
>>>
>>>
>>> Python 3 tests are failing because the Fernet formatter is insisting
>>> that all project-ids be valid UUIDs, but some of the old tests have "FOO"
>>> and "BAR" as ids.  These either need to be converted to UUIDS, or the
>>> formatter needs to be more forgiving.
>>>
>>> Caching of token validations was messing with revocation checking.
>>> Tokens that were valid once were being reported as always valid. Thus, the
>>> current review  removes all caching on token validations, a change we
>>> cannot maintain.  Once all the test are successfully passing, we will
>>> re-introduce the cache, and be far more aggressive about cache invalidation.
>>>
>>> Tempest tests are currently failing due to Devstack not properly
>>> identifying Fernet as the default token provider, and creating the Fernet
>>> key repository.  I'm tempted to just force devstack to always create the
>>> directory, as a user would need it if they ever switched the token provider
>>> post launch anyway.
>>>
>>>
>> There's a review to change devstack to default to fernet:
>> https://review.openstack.org/#/c/195780/ . This was mostly to show that
>> tempest still passes with fernet configured. It uncovered a couple of test
>> issues (similar in nature to the revocation checking issues mentioned in
>> the original note) that have since been fixed.
>>
>> We'd prefer to not have devstack overriding config options and instead
>> use keystone's defaults. The problem is if fernet is the default in
>> keystone then it won't work out of the box since the key database won't
>> exist. One option that I think we should investigate is to have keystone
>> create the key database on startup if it doesn't exist.
>>
>> - Brant
>>
>>
>
> I'm not a devstack user, but as I mentioned before, I assume devstack
> called keystone-manage db_sync? Why couldn't it also call keystone-manage
> fernet_setup?
>
>
When you tell devstack that it's using fernet then it does keystone-manage
fernet_setup. When you tell devstack to use the default, it doesn't
fernet_setup because for now it thinks the default is UUID and doesn't
require keys. One way to have devstack work when fernet is the default is
to have devstack always do keystone-manage fernet_setup.

Really what we want to do is have devstack work like other deployment
methods. We can reasonably expect featureful deployers like puppet to
keystone-manage fernet_setup in the course of setting up keystone. There's
more basic deployers like RPMs or debs that in the past have said they like
the defaults to "just work" (like UUID tokens) and not require extra
commands.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] State of Fernet Token deployment

[Brant Knudson]

On Fri, Apr 15, 2016 at 9:04 PM, Adam Young  wrote:

> We all want Fernet to be a reality.  We ain't there yet (Except for mfish
> who has no patience) but we are getting closer.  The goal is to get Fernet
> as the default token provider as soon as possible. The review to do this
> has uncovered a few details that need to be fixed before we can do this.
>
> Trusts for V2 tokens were not working correctly.  Relatively easy fix.
> https://review.openstack.org/#/c/278693/ Patch is still failing on Python
> 3.  The tests are kindof racy due to the revocation event 1 second
> granularity.  Some of the tests here have A sleep (1) in them still, but
> all should be using the time control aspect of the unit test fixtures.
>
> Some of the tests also use the same user to validate a token as that have,
> for example, a role unassigned.  These expose a problem that the revocation
> events are catching too many tokens, some of which should not be treated as
> revoked.
>
> Also, some of the logic for revocation checking has to change. Before, if
> a user had two roles, and had one removed, the token would be revoked.
> Now, however, the token will validate successful, but the response will
> only have the single assigned role in it.
>
>
> Python 3 tests are failing because the Fernet formatter is insisting that
> all project-ids be valid UUIDs, but some of the old tests have "FOO" and
> "BAR" as ids.  These either need to be converted to UUIDS, or the formatter
> needs to be more forgiving.
>
> Caching of token validations was messing with revocation checking. Tokens
> that were valid once were being reported as always valid. Thus, the current
> review  removes all caching on token validations, a change we cannot
> maintain.  Once all the test are successfully passing, we will re-introduce
> the cache, and be far more aggressive about cache invalidation.
>
> Tempest tests are currently failing due to Devstack not properly
> identifying Fernet as the default token provider, and creating the Fernet
> key repository.  I'm tempted to just force devstack to always create the
> directory, as a user would need it if they ever switched the token provider
> post launch anyway.
>
>
There's a review to change devstack to default to fernet:
https://review.openstack.org/#/c/195780/ . This was mostly to show that
tempest still passes with fernet configured. It uncovered a couple of test
issues (similar in nature to the revocation checking issues mentioned in
the original note) that have since been fixed.

We'd prefer to not have devstack overriding config options and instead use
keystone's defaults. The problem is if fernet is the default in keystone
then it won't work out of the box since the key database won't exist. One
option that I think we should investigate is to have keystone create the
key database on startup if it doesn't exist.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Which projects support wsgi / python3 in mitaka?

[Brant Knudson]

On Fri, Mar 25, 2016 at 9:54 AM, Matthew Thode 
wrote:

> I'm updating the packaging for openstack in preparation for the release
> and need some info to update correctly.  I'm specifically looking at the
> following services.
>
> I don't know if any of these services support python3 fully as a service
> (liberty was kinda wishy washy on that).  Even if it supports python3,
> which version, 3.4 or 3.4 or 3.5 (or multiple).
>
> keystone - probably - keystone/server/wsgi.py
>

For keystone you should use bin/keystone-wsgi-public and
bin/keystone-wsgi-admin as the wsgi entrypoints. The gate tests use
keystone running under mod_wsgi in apache httpd. We've also got a test in
keystone that uses uwsgi as the container.

Keystone doesn't support python 3 in M.

 - Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][zaqar][cloudkitty] Default ports list

[Brant Knudson]

On Thu, Mar 10, 2016 at 10:43 AM, Brant Knudson  wrote:

>
>
> On Thu, Mar 10, 2016 at 8:10 AM, Thomas Herve  wrote:
>
>> On Thu, Mar 10, 2016 at 2:55 PM, Sean Dague  wrote:
>> > On 03/10/2016 08:40 AM, Thomas Herve wrote:
>> >> On Thu, Mar 10, 2016 at 2:28 PM, Chris Dent 
>> wrote:
>> >>> +many. It would be great if we just got rid of the runnable web
>> >>> servers in the projects and just expose wsgi apps (the tools like
>> >>> devstack etc mounted under whatever the available server is).
>> >>
>> >> Isn't devstack meant for development? Running the APIs in a WSGI
>> >> container like Apache or uwsgi makes for a terrible debugging
>> >> experience. Just this morning I had to prevent aodh from running in
>> >> Apache to be able to run it standalone.
>> >>
>> >> Also, those apps that use WSGI still bind a different port. The fact
>> >> that it runs in Apache doesn't really solve the URLs problem.
>> >
>> > But they shouldn't. I do realize it's taking a while to get there, but
>> > this is the push to get rid of all these weird ports. The point is to
>> > get them all on 80 in a subpath or a separate domain.
>> >
>> >
>> > If projects use the pbr wsgi_script directive the wsgi script will run
>> > on wsgi ref on the commandline if you need that for debug -
>> >
>> https://github.com/openstack/keystone/blob/4db54810e58ad86edb92869a608fb5630a6b99e5/setup.cfg#L75
>> > that was built to make this simpler.
>>
>> Right, but if we move to everything in WSGI besides a subpath, you
>> won't be able to switch to the script easily. Unless you actually
>> implement that using a reverse proxy.
>>
>> I totally understand the will to remove those ports from the final
>> product. I question whether it's the right choice for devstack. It
>> would seem that at least keeping those 'weird' ports for internal
>> consumption would be useful. It's not like we're close to use the 65K
>> available.
>>
>> --
>> Thomas
>>
>>
> For some time, devstack has been running with keystone accepting on both
> it's legacy ports (:5000 and :35357), and on subpaths (:80/identity and
> /identity_admin). I tried to change devstack to have keystone only
> listening on subpaths but this is failing in tempest-lib -- see the errors
> in https://review.openstack.org/#/c/193894/. At first I thought it would
> be best to get tempest-lib doing version discovery but this turned into a
> lot of code since version discovery requires quite a bit of parsing and
> searching through dicts which always requires lots of error handling and
> test code (see reviews ending at https://review.openstack.org/#/c/263452/).
> (You might wonder why I didn't use the code that's already in
> keystoneclient/keystoneauth for version discovery -- it's because tempest
> doesn't use the client libs.)
>
> Anyways I think the alternative that's going to be backwards compatible is
> to have the user be able to pass in the identity endpoints for v2 and v3
> and have tempest use those if provided. I haven't had time to propose this
> yet.
>
> My preference would be to have the keystone processes running separately
> under uwsgi or gunicorn, then httpd proxies to that from /identity and
> /identity_admin (and :5000 and :35357 for legacy). Hopefully this would be
> over a unix socket talking uwsgi protocol or whatever the process and httpd
> support. Note that devstack already has a TLS-proxy deployment option
> that's similar. I've been hoping to have time to propose the keystone
> devstack deployment use this but I'm easily distracted.
>
>
Here's the proposed change to devstack to allow keystone running under
uwsgi while apache accepts http / https (using mod_uwsgi_proxy) if you want
to take a look: https://review.openstack.org/#/c/291817/

We've got a few too many deployment options for keystone devstack now. I'd
like to get this down to just mod_wsgi and mod_uwsgi_proxy. The uwsgi
deploy we should be able to get rid of quickly, just add a non-voting job
in keystone for mod_uwsgi_proxy and get rid of the uwsgi job. We should be
able to get rid of eventlet right away in N since it's not supported to run
eventlet keystone anymore.

Next, we should be able to simplify the tls-proxy deployment since keystone
is always running in apache.

Then simplify further: Get rid of the tls-proxy option and instead have
keystone apache listen on https + http by default. Then eventually stop
listening on http and run https all the time.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [stable] Stable* wiki updates

[Brant Knudson]

Since there's no gerrit, I'll leave my comment here: +1 Thanks!

On Fri, Mar 11, 2016 at 1:54 AM, Alan Pevec  wrote:

> Hi stable-maints,
>
> FYI I've updated https://wiki.openstack.org/wiki/StableBranch and
> https://wiki.openstack.org/wiki/StableBranchRelease now that all
> policy and team information has ben included in the Project Team
> Guide. Please review in case I missed something!
>
> Cheers,
> Alan
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][zaqar][cloudkitty] Default ports list

[Brant Knudson]

On Thu, Mar 10, 2016 at 8:10 AM, Thomas Herve  wrote:

> On Thu, Mar 10, 2016 at 2:55 PM, Sean Dague  wrote:
> > On 03/10/2016 08:40 AM, Thomas Herve wrote:
> >> On Thu, Mar 10, 2016 at 2:28 PM, Chris Dent 
> wrote:
> >>> +many. It would be great if we just got rid of the runnable web
> >>> servers in the projects and just expose wsgi apps (the tools like
> >>> devstack etc mounted under whatever the available server is).
> >>
> >> Isn't devstack meant for development? Running the APIs in a WSGI
> >> container like Apache or uwsgi makes for a terrible debugging
> >> experience. Just this morning I had to prevent aodh from running in
> >> Apache to be able to run it standalone.
> >>
> >> Also, those apps that use WSGI still bind a different port. The fact
> >> that it runs in Apache doesn't really solve the URLs problem.
> >
> > But they shouldn't. I do realize it's taking a while to get there, but
> > this is the push to get rid of all these weird ports. The point is to
> > get them all on 80 in a subpath or a separate domain.
> >
> >
> > If projects use the pbr wsgi_script directive the wsgi script will run
> > on wsgi ref on the commandline if you need that for debug -
> >
> https://github.com/openstack/keystone/blob/4db54810e58ad86edb92869a608fb5630a6b99e5/setup.cfg#L75
> > that was built to make this simpler.
>
> Right, but if we move to everything in WSGI besides a subpath, you
> won't be able to switch to the script easily. Unless you actually
> implement that using a reverse proxy.
>
> I totally understand the will to remove those ports from the final
> product. I question whether it's the right choice for devstack. It
> would seem that at least keeping those 'weird' ports for internal
> consumption would be useful. It's not like we're close to use the 65K
> available.
>
> --
> Thomas
>
>
For some time, devstack has been running with keystone accepting on both
it's legacy ports (:5000 and :35357), and on subpaths (:80/identity and
/identity_admin). I tried to change devstack to have keystone only
listening on subpaths but this is failing in tempest-lib -- see the errors
in https://review.openstack.org/#/c/193894/. At first I thought it would be
best to get tempest-lib doing version discovery but this turned into a lot
of code since version discovery requires quite a bit of parsing and
searching through dicts which always requires lots of error handling and
test code (see reviews ending at https://review.openstack.org/#/c/263452/).
(You might wonder why I didn't use the code that's already in
keystoneclient/keystoneauth for version discovery -- it's because tempest
doesn't use the client libs.)

Anyways I think the alternative that's going to be backwards compatible is
to have the user be able to pass in the identity endpoints for v2 and v3
and have tempest use those if provided. I haven't had time to propose this
yet.

My preference would be to have the keystone processes running separately
under uwsgi or gunicorn, then httpd proxies to that from /identity and
/identity_admin (and :5000 and :35357 for legacy). Hopefully this would be
over a unix socket talking uwsgi protocol or whatever the process and httpd
support. Note that devstack already has a TLS-proxy deployment option
that's similar. I've been hoping to have time to propose the keystone
devstack deployment use this but I'm easily distracted.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] audience for release notes

[Brant Knudson]

On Fri, Feb 26, 2016 at 6:34 AM, Doug Hellmann 
wrote:

> I've had a few conversations recently about "appropriate" content for
> release notes, so I thought it was worth starting a separate thread here
> to clarify.
>
> We have 3 potential audiences for release notes:
>
> 1. Developers consuming libraries or other code directly.
> 2. Deployers and operators.
> 3. End-users.
>
> We implemented reno for this cycle to replace the old wiki-based
> system for writing release notes for deployers, operators, and
> end-users. We have in-tree developer documentation for Developers. The
> two sets of documentation are meant for different purposes, so we need
> to think about what information we publish in each.
>
> As you are writing release notes using reno to be published under
> docs.openstack.org/releasenotes, please keep in mind the audience
> is *not* someone who is necessarily going to be looking at code or
> writing apps based on libraries we produce. Highlight information
> that deployers and operators will need, like changes to configuration
> options or service behaviors. Describe REST API changes that an
> end-user may need to know about.
>
>
python-keystoneclient doesn't provide config options, so there should never
be release notes for it. Probably best to disable releasenote generation
for most of the python-*client libraries.

-- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Should we signal backwards incompatible changes in microversions?

[Brant Knudson]

On Tue, Feb 16, 2016 at 1:04 PM, Dean Troyer  wrote:

>
>
> On Tue, Feb 16, 2016 at 12:17 PM, Sean Dague  wrote:
>
>> Honestly, doing per API call version switching is probably going to end
>> in tears. HTTP is stateless, so it's allowed, but it will end in tears
>> of complexity as you need to self modify resources before passing them
>> back. Or follow links that don't exist.
>>
>
> Maybe.  But any code that knows it is using a specific version already is
> handling the special cases.  We are trading the speed of implementation for
> the juggling of multiple versions, no matter if they are labelled
> sequentially or via semver.  I've seen places that mixed Identity v2 and v3
> because of the way the APIs worked and were more convenient.  Same problem,
> bigger delta per API rev vs only one rev.
>
> It's also worth looking at the changes we've actually been making here
>> instead of theoretical examples. The amount of effort to make an
>> application use 2.20 instead of 2.1 is pretty minimal.
>
>
> Sure, but there is nothing in place to prevent that contrived example. The
> first time an addition is made of what would have been an extension in the
> past we will be there.  [and no, this is in no way a defense of extensions
> ;).
>
> If Identity had micro-versioned their way between V2 and v3 would the
> other projects been able to convert faster due to not having to do it all
> at once?
>
>
You didn't have to convert from V2 to V3 all at once, and we didn't. I
assume there is still some use of V2 in gate runs in addition to using V3.

Keystone supports the V2 API on /v2.0 and the V3 API on /v3. So you could
make some requests to V2 and other requests to V3. You could get a token
using V2 and use it on V3, or get a token using V3 and use it to do V2
operations. You can't use new V3 features when accessing the V2 API (like
domains).

This is pretty much the same as your example of specifying a different
version for the nova API on different requests (except they have a lot more
than just 2 and 3). We also keep adding routes to v3 each release, so what
operations are supported keeps changing.

 - Brant

dt
>
> --
>
> Dean Troyer
> dtro...@gmail.com
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] tenant vs. project

[Brant Knudson]

On Fri, Feb 12, 2016 at 7:42 AM, Sean Dague  wrote:

> On 02/12/2016 08:30 AM, Dean Troyer wrote:
> > On Fri, Feb 12, 2016 at 6:01 AM, Sean Dague  > > wrote:
> >
> > OpenStack is wildly inconsistent in it's use of tenant vs. project.
> As
> > someone that wasn't here at the beginning, I'm not even sure which
> one
> > we are supposed to be transitioning from -> to.
> >
> > At a minimum I'd like to make all of devstack use 1 term, which is
> the
> > term we're trying to get to. That will help move the needle.
> >
> > However, again, I'm not sure which one that is supposed to be
> (comments
> > in various places show movement in both directions). So people with
> > deeper knowledge here, can you speak up as to which is the deprecated
> > term and which is the term moving forward.
> >
> >
> > IIRC Nova started with project, until the marriage with Rax, when many
> > things changed, although the project -> tenant change may have never
> > been completed.  Keystone v3 started the movement back to project.
> > OpenStackClient made the commitment from the beginning to present
> > exactly one term to the user in all cases, and we chose project.
> >
> > I've thought about making that change in DevStack many times, and would
> > love to see it happen.  Somehow it never gets to the top of the queue.
> > And now in a plugin world, it'll be a bit harder to maintain
> compatibility.
>
> Hmmm... one issue with that:
>
> keystone only supports tenant_id in their service catalog for
> replacement, not project_id - https://review.openstack.org/#/c/279523/.
>
> Keystone folks, any idea if that's going to be adjusted?
>
> -Sean
>
>
Proposed the change to keystone here:
https://review.openstack.org/#/c/279576/

As an excuse for not supporting this earlier, the replacement of
$(tenant_id)s, $(user_id)s, etc., in the service catalog is considered a
legacy feature so this has been neglected.

- Brant

--
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][ec2-api] Moving EC2 Auth and S3Token to Externally supported

[Brant Knudson]

On Fri, Feb 5, 2016 at 1:03 PM, Dolph Mathews 
wrote:

>
> On Fri, Feb 5, 2016 at 12:37 PM, Andrey Pavlov 
> wrote:
>
>> swift3(s3) works like ec2-api.
>>
>> 1. swift3/ec2-api recieves AWS request
>> 2. it parses signature and access_key (and other headers)
>> 3. it sends these values (and token that calculated from request) to
>> keystone
>> 4. keystone gets secret_key from DB, then calculates signature by
>> recieved access_key and token
>> 5. keystone compares recived signature and claculated signature and
>> then return 'error' or auth_token
>> 6. swift3/ec2-api recieves answer from keystone and return 'forbidden'
>> or continues execution
>> 7. in case of continue swift3/ec2-api uses recieved auth_token for
>> calls other services: nova, cinder, neutron, swift...
>>
>> So I don't understand how implement this functionality outside of
>> keystone...
>>
>
> EC2 support is implemented in middleware on top of keystone, and that
> middleware happens to live in the openstack/keystone repository. This
> change is just proposing to move that middleware code into a dedicated new
> repository and change the community support & maintenance model - it would
> not affect how the code actually operates. The only affect on operators is
> that it would require an extra step to deploy it. End users would not be
> affected.
>
>
> https://github.com/openstack/keystone/blob/5f51912b54dff0a71f00987e9f5c1d6015ad08bd/keystone/contrib/ec2/routers.py#L27
>
>
> https://github.com/openstack/keystone/blob/5f51912b54dff0a71f00987e9f5c1d6015ad08bd/etc/keystone-paste.ini#L27-L31
>
>

One of the things that prompted this discussion is a proposal to make EC2
and S3 required, and not removable by editing the paste config:
https://review.openstack.org/#/c/274973/

Some of us were taking advantage of this ability, but others think that all
APIs should be supported.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] nominating Alexis Lee for oslo-core

[Brant Knudson]

On Thu, Jan 28, 2016 at 4:05 PM, Doug Hellmann 
wrote:

> I am nominating Alexis Lee (lxsli) for oslo-core.
>
> Alexis has been working on adding configuration file reloading to
> oslo.config this cycle. The work isn't complete, but at this point
> he probably knows as much or more about the internals of that library
> as any of us. :-) He has also shown, with some of his other recent
> proposals, that he has a ideas for the future of configuration
> management and an interest in contributing to Oslo.
>
> Please indicate yea or nay with the usual +1/-1 vote here.
>
> Doug
>
>
> http://stackalytics.com/?release=all&project_type=all&metric=commits&user_id=alexisl
>
>
+1! - Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] nominating Ronald Bradford for oslo-core

[Brant Knudson]

On Fri, Jan 29, 2016 at 3:15 PM, Rochelle Grober  wrote:

> I'm not a voting member of the Oslo team, but a BIG
>
> +1
>
> From me.
>
> --Rocky
>
> -Original Message-
> From: Joshua Harlow [mailto:harlo...@fastmail.com]
> Sent: Thursday, January 28, 2016 11:38 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [oslo] nominating Ronald Bradford for
> oslo-core
>
> Let's do it,
>
> +1
>
> -Josh
>
> On 01/28/2016 05:41 PM, Davanum Srinivas wrote:
> > +1 from me!
> >
> > -- Dims
> >
> > On Thu, Jan 28, 2016 at 4:00 PM, Doug Hellmann 
> wrote:
> >> I am nominating Ronald Bradford (rbradfor) for oslo-core.
> >>
> >> Ronald has been working this cycle to learn about oslo.context,
> >> oslo.log, and oslo.config. He anticipates picking up the much-needed
> >> work on the app-agnostic-logging blueprint, and has already started
> >> making incremental changes related to that work.  He has also
> >> contributed to the documentation generation and sample generator
> >> in oslo.config. His understanding of the code, our
> backwards-compatibility
> >> requirements, and the operational needs related to configuration
> >> and logging will make him a valuable addition to the team.
> >>
> >> Please indicate yea or nay with the usual +1/-1 vote here.
> >>
> >> Doug
> >>
> >>
> http://stackalytics.com/?release=all&project_type=all&user_id=ronaldbradford
> >>
> >>
>

+1! - Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] service type vs. project name for use in headers

[Brant Knudson]

On Wed, Jan 27, 2016 at 1:47 PM, michael mccune  wrote:

> hi all,
>
> there have been a few reviews recently where the issue of service type
> versus project name have come up for use in the headers. as usual this
> conversation can get quite murky as there are several good examples where
> service type alone is not sufficient (for example if a service exposes
> several api controllers), and as has been pointed out project name can also
> be problematic (for example projects can change name).
>
> i'm curious if we could come to a consensus regarding the use of service
> type *or* project name for headers. i propose leaving the ultimate decision
> up to the projects involved to choose the most appropriate identifier for
> their custom headers.
>
> i am not convinced that we would ever need to have a standard on how these
> names are chosen for the header values, or if we would even need to have
> header names that could be deduced. for me, it would be much better for the
> projects use an identifier that makes sense to them, *and* for each project
> to have good api documentation.
>
> so, instead of using examples where we have header names like
> "OpenStack-Some-[SERVICE_TYPE]-Header", maybe we should suggest
> "OpenStack-Some-[SERVICE_TYPE or PROJECT_NAME]-Header" as our guideline.
>
> for reference, here are the current reviews that are circling around this
> issue:
>
> https://review.openstack.org/#/c/243429
> https://review.openstack.org/#/c/273158
> https://review.openstack.org/#/c/243414
>
> and one that has already been merged:
>
> https://review.openstack.org/#/c/196918
>
> thoughts?
>
>
Why does the service type or name need to be in the header at all? The
request goes to a specific service so the server and client already know
the service type or name. - Brant

regards,
> mike
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Sachi King for oslo core

[Brant Knudson]

On Sun, Jan 24, 2016 at 8:51 PM, Robert Collins 
wrote:

> Hey, so I'd like to propose Sachi - one of my HPE colleages who has
> been helping out with pbr for a while now, for oslo core. pbr is
> pretty quiet as you know, so its a bit hard to assess her work based
> on reviews done - but I think her error rate will be
> approximately the same as mine - pbr is such a minefield that everyone
> will make mistakes, but she is thoughtful and has been burnt enough by
> the minefield that she's developing quite the paranoia about
> interactions in the wild. \o/
>
> I'd actually like to propose her for oslo core, not just pbr: though she
> doesn't have a big track record there outside of pbr, I don't see much
> point in restricting her to just pbr - she's very
> diligent and I'm sure if we offered it she'd do more for oslo code
> review than I've managed to. I have asked her if she's interested, and
> if she could commit to doing some reviews in oslo outside of pbr
> itself - which she has said she is and can.
>
> -Rob
>
> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
>
>
+1 from me. Thanks!

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [python-keystoneclient] return-request-id-to-caller

[Brant Knudson]

On Tue, Dec 22, 2015 at 2:02 AM, koshiya maho 
wrote:

> Hi,
>
> I want to add support for returning 'x-openstack-request-id' in
> python-keystoneclient as per the design proposed in cross-project specs.
>
> Cross-project spec has been already approved. :
>
> https://specs.openstack.org/openstack/openstack-specs/specs/return-request-id.html
>
> So, I'd like you to approve this BP. :
>
> https://blueprints.launchpad.net/python-keystoneclient/+spec/return-request-id-to-caller
>
> Thank you and best regards,
>
> --
> Maho Koshiya
> E-Mail : koshiya.m...@po.ntts.co.jp
>
>
>
I put this on the meeting agenda for as a no-spec blueprint to approve.[1]
Since there's already a cross-project spec I don't think we need another
spec in keystone.

[1]
https://wiki.openstack.org/wiki/Meetings/KeystoneMeeting#Review_of_Keystone_Blueprints_for_No-Spec_Requires_Status

 - Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][keystone] Move oslo.policy from oslo to keystone

[Brant Knudson]

On Thu, Dec 17, 2015 at 1:51 PM, Davanum Srinivas  wrote:

> Thinking more about it. The only change we'll have is that if someone
> files a oslo-specs for oslo.policy we need to tell them to switch over
> to keystone-specs. We could add notes in README etc to make this
> apparent. So i am +1 to making this move.
>
> Brant, other keystone cores,
> Can you please file the governance review request and we can make sure
> oslo cores chime in there? and make it official?
>
>
Here's the proposed governance change:
https://review.openstack.org/#/c/259169/

 - Brant



> Thanks,
> Dims
>
>
> On Thu, Dec 17, 2015 at 2:40 PM, Flavio Percoco  wrote:
> > On 16/12/15 18:51 -0800, Morgan Fainberg wrote:
> >>
> >> For what is is worth, we originally proposed oslo.policy to graduate to
> >> Keystone when we were converting to the library. I still think it
> belongs
> >> in
> >> keystone (as long as the oslo team doesn't mind that long-term keystone
> >> team
> >> owns something in the oslo. namespace).
> >>
> >> The short term adding keystone-core should get some more eyes on the
> >> reviews,
> >> so +1 to that.
> >
> >
> >
> > Just want to +1 all the above.
> >
> > It'd be great if we can finally hand the library over to the keystone
> > team, where I think it belongs.
> >
> > Cheers,
> > Flavio
> >
> >>
> >> --Morgan
> >>
> >> On Wed, Dec 16, 2015 at 4:08 PM, Davanum Srinivas 
> >> wrote:
> >>
> >>As an interim measure, added keystone-core to oslo-policy-core[1]
> >>
> >>Thanks,
> >>Dims
> >>
> >>[1] https://review.openstack.org/#/admin/groups/556,members
> >>
> >>On Wed, Dec 16, 2015 at 10:40 PM, Dolph Mathews
> >> 
> >>wrote:
> >>>
> >>> On Wed, Dec 16, 2015 at 1:33 PM, Davanum Srinivas <
> dava...@gmail.com>
> >>wrote:
> >>>>
> >>>> Brant,
> >>>>
> >>>> I am ok either way, guess the alternative was to add keystone-core
> >>>> directly to the oslo.policy core group (can't check right now).
> >>>
> >>>
> >>> That's certainly reasonable, and kind of what we did with pycadf.
> >>>
> >>>>
> >>>>
> >>>> The name is very possibly going to create confusion
> >>>
> >>>
> >>> I assume you're not referring to unnecessarily changing the name of
> >> the
> >>> project itself (oslo.policy) just because there might be a shift in
> >> the
> >>> group of maintainers! Either way, let's definitely not do that.
> >>>
> >>>>
> >>>> -- Dims
> >>>>
> >>>> On Wed, Dec 16, 2015 at 7:22 PM, Jordan Pittier
> >>>>  wrote:
> >>>> > Hi,
> >>>> > I am sure oslo.policy would be good under Keystone's governance.
> >> But I
> >>>> > am
> >>>> > not sure I understood what's wrong in having oslo.policy under
> the
> >>oslo
> >>>> > program ?
> >>>> >
> >>>> > Jordan
> >>>> >
> >>>> > On Wed, Dec 16, 2015 at 6:13 PM, Brant Knudson 
> >> wrote:
> >>>> >>
> >>>> >>
> >>>> >> I'd like to propose moving oslo.policy from the oslo program to
> >> the
> >>>> >> keystone program. Keystone developers know what's going on with
> >>>> >> oslo.policy
> >>>> >> and I think are more interested in what's going on with it so
> >> that
> >>>> >> reviews
> >>>> >> will get proper vetting, and it's not like oslo doesn't have
> >> enough
> >>>> >> going on
> >>>> >> with all the other repos. Keystone core has equivalent
> stringent
> >>>> >> development
> >>>> >> policy that we already enforce with keystoneclient and
> >> keystoneauth,
> >>so
> >>>> >> oslo.policy isn't going to be losing any stability.
> >>>> >>

Re: [openstack-dev] [oslo][keystone] Move oslo.policy from oslo to keystone

[Brant Knudson]

On Wed, Dec 16, 2015 at 11:13 AM, Brant Knudson  wrote:

>
> I'd like to propose moving oslo.policy from the oslo program to the
> keystone program. Keystone developers know what's going on with oslo.policy
> and I think are more interested in what's going on with it so that reviews
> will get proper vetting, and it's not like oslo doesn't have enough going
> on with all the other repos. Keystone core has equivalent stringent
> development policy that we already enforce with keystoneclient and
> keystoneauth, so oslo.policy isn't going to be losing any stability.
>
> If there aren't any objections, let's go ahead with this. I heard this
> requires a change to a governance repo, and gerrit permission changes to
> make keystone-core core, and updates in oslo.policy to change some docs or
> links. Any oslo.policy specs that are currently proposed
>

will have to be reproposed to keystone-specs.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][keystone] Move oslo.policy from oslo to keystone

[Brant Knudson]

I'd like to propose moving oslo.policy from the oslo program to the
keystone program. Keystone developers know what's going on with oslo.policy
and I think are more interested in what's going on with it so that reviews
will get proper vetting, and it's not like oslo doesn't have enough going
on with all the other repos. Keystone core has equivalent stringent
development policy that we already enforce with keystoneclient and
keystoneauth, so oslo.policy isn't going to be losing any stability.

If there aren't any objections, let's go ahead with this. I heard this
requires a change to a governance repo, and gerrit permission changes to
make keystone-core core, and updates in oslo.policy to change some docs or
links. Any oslo.policy specs that are currently proposed

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance][keystone][artifacts] Service Catalog name for Glance Artifact Repository API

[Brant Knudson]

On Mon, Dec 14, 2015 at 9:28 AM, Ian Cordasco 
wrote:

>
>
> On 12/14/15, 02:18, "Kuvaja, Erno"  wrote:
>
> >> -Original Message-
> >> From: McLellan, Steven
> >> Sent: Friday, December 11, 2015 6:37 PM
> >> To: OpenStack Development Mailing List (not for usage questions)
> >> Subject: Re: [openstack-dev] [glance][keystone][artifacts] Service
> >>Catalog
> >> name for Glance Artifact Repository API
> >>
> >> Hi Alex,
> >>
> >> Searchlight uses port 9393 (it also made sense to us when we spun out of
> >> Glance!), so we would prefer it if there's another one that makes sense.
> >> Regarding the three hardest things in computer science, searchlight's
> >>already
> >> dealing with cache invalidation so I'll stay out of the naming
> >>discussion.
> >>
> >> Thanks!
> >>
> >> Steve
> >
> >Thanks for the heads up Steve,
> >
> >Mind to make sure that it gets registered for Searchlight as well. It's
> >not listed in config-reference [0] nor iana [1] (seems that at least
> >glance ports are not registered in iana either fwiw):
>
> Are any of the OpenStack projects actually listed on IANA?
>
> Searching through
> https://www.iana.org/assignments/service-names-port-numbers/service-names-p
> ort-numbers.txt for service names (nova, swift, etc.) I don't see *any*
> openstack services.
>
>

We've got Identity's port 35357 reserved:

openstack-id   35357   tcpOpenStack ID Service
[Rackspace_Hosting]   [Ziad_Sawalha]
 2011-08-15

Despite the reservation it still caused issues since this port is in
Linux's default range for ephemeral ports. So some other process might get
this port before keystone did and then keystone would fail to start.

Also note that this is the "admin" port, which keystone only needs for v2
(to be deprecated) since we've got policy support in v3.

Just goes to show that we shouldn't be using distinct ports for web
services, put them on a path on :443 or :80 like the web was designed for.

- Brant



> >
> >[0]
> >
> http://docs.openstack.org/liberty/config-reference/content/firewalls-defau
> >lt-ports.html
> >[1]
> >
> http://www.iana.org/assignments/service-names-port-numbers/service-names-p
> >ort-numbers.xhtml
> >
> >- Erno
> >>
> >> From: Alexander Tivelkov
> >> mailto:ativel...@mirantis.com>>
> >> Reply-To: "OpenStack Development Mailing List (not for usage questions)"
> >> mailto:openstack-
> >> d...@lists.openstack.org>>
> >> Date: Friday, December 11, 2015 at 11:25 AM
> >> To: "OpenStack Development Mailing List (not for usage questions)"
> >> mailto:openstack-
> >> d...@lists.openstack.org>>
> >> Subject: [openstack-dev] [glance][keystone][artifacts] Service Catalog
> >>name
> >> for Glance Artifact Repository API
> >>
> >> Hi folks!
> >>
> >> As it was decided during the Mitaka design summit, we are separating the
> >> experimental Artifact Repository API from the main Glance API. This API
> >>will
> >> have a versioning sequence independent from the main Glance API and will
> >> be run as a standalone optional service, listening on the port
> >>different from
> >> the standard glance-api port (currently the proposed default is 9393).
> >> Meanwhile, it will remain an integral part of the larger Glance
> >>project, sharing
> >> the database, implementation roadmap, development and review teams
> >> etc.
> >>
> >> Since this API will be consumed by both end-users and other Openstack
> >> services, its endpoint should be discoverable via regular service
> >>catalog API.
> >> This rises the question: what should be the service name and service
> >>type for
> >> the appropriate entree in the service catalog?
> >>
> >> We've came out with the idea to call the service "glare" (this is our
> >>internal
> >> codename for the artifacts initiative, being an acronym for "GLance
> >>Artifact
> >> REpository") and set its type to "artifacts". Other alternatives for
> >>the name
> >> may be "arti" or "glance_artifacts" and for the type - "assets" or
> >>"objects"
> >> (the latter may be confusing since swift's type is object-store, so I
> >>personally
> >> don't like it).
> >>
> >> Well... we all know, naming is complicated... anyway, I'll appreciate
> >>any
> >> feedback on this. Thanks!
> >>
> >> --
> >> Regards,
> >> Alexander Tivelkov
> >>
> >> __
> >> 
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: OpenStack-dev-
> >> requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >__
> >OpenStack Development Mailing List (not for usage questions)
> >Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
>

Re: [openstack-dev] [oslo][glance][all] Removing deprecated functions from oslo_utils.timeutils

[Brant Knudson]

On Thu, Dec 10, 2015 at 11:10 AM, Flavio Percoco  wrote:

> On 10/12/15 08:21 -0600, Brant Knudson wrote:
>
>>
>>
>> On Thu, Dec 10, 2015 at 7:26 AM, Sean Dague  wrote:
>>
>>On 12/10/2015 01:56 AM, Joshua Harlow wrote:
>>> Shouldn't be to hard (although it's probably not on each oslo
>> project,
>>> but on the consumers projects).
>>>
>>> The warnings module can turn warnings into raised exceptions with a
>>> simple command line switch btw...
>>>
>>> For example:
>>>
>>> $ python -Wonce
>>> Python 2.7.6 (default, Jun 22 2015, 17:58:13)
>>> [GCC 4.8.2] on linux2
>>> Type "help", "copyright", "credits" or "license" for more
>> information.
>>>>>> import warnings
>>>>>> warnings.warn("I am not supposed to be used", DeprecationWarning)
>>> __main__:1: DeprecationWarning: I am not supposed to be used
>>>
>>> $ python -Werror
>>> Python 2.7.6 (default, Jun 22 2015, 17:58:13)
>>> [GCC 4.8.2] on linux2
>>> Type "help", "copyright", "credits" or "license" for more
>> information.
>>>>>> import warnings
>>>>>> warnings.warn("I am not supposed to be used", DeprecationWarning)
>>> Traceback (most recent call last):
>>>   File "", line 1, in 
>>> DeprecationWarning: I am not supposed to be used
>>>
>>> https://docs.python.org/2/library/warnings.html#the-warnings-filter
>>>
>>> Turn that CLI switch from off to on and I'm pretty sure usage of
>>> deprecated things will become pretty evident real quick ;)
>>
>>It needs to be more targetted than that. There is a long standing
>>warning between paste and pkg_resources that would hard stop everyone.
>>
>>But, yes, the idea of being able to run unit tests with fatal
>>deprecations of oslo easily is what I think would be useful.
>>  -Sean
>>
>>
>>
>> In keystone we set a warnings filter for the unit tests so that if
>> keystone
>> calls any deprecated function it'll raise[1]. So when the oslo timeutils
>> functions were deprecated it broke keystone gate and we fixed it. It
>> would be
>> nicer to have a non-voting gate job to serve as a warning instead, but
>> it's
>> only happened a couple of times where this caused keystone to be blocked
>> for
>> the day that it took to get the fix in. Anyways, it would be easy enough
>> for us
>> to have this enabled/disabled via an environment variable and create a
>> tox job.
>>
>> If we had a non-voting warning job it could also run oslo libs from master
>> rather than released.
>>
>> [1]
>> http://git.openstack.org/cgit/openstack/keystone/tree/keystone/tests/unit/
>> core.py?id=4f8c4a7a10d85080d6db9b30ae1759d45a38a32c#n460
>>
>
> I like this!
>
> Will look into what needs to be done to make it happen in Glance and
> get feedback from the rest of the folks. As you said, it's really few
> times when this would totally break a project's gate and I think
> that's manageable.
> A non-voting gate doesn't have the same effect, FWIW, but I also see
> reasons for having one instead of stopping a project.
>
> Flavio
>
>
If other projects are interested (and since we already copy-pasted it to
other keystone projects), I've got a pull request in to fixtures to add a
fixture that does the same thing, see
https://github.com/testing-cabal/fixtures/pull/19 .

 - Brant


>
>> - Brant
>>
>>
> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> --
> @flaper87
> Flavio Percoco
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][glance][all] Removing deprecated functions from oslo_utils.timeutils

[Brant Knudson]

On Thu, Dec 10, 2015 at 7:26 AM, Sean Dague  wrote:

> On 12/10/2015 01:56 AM, Joshua Harlow wrote:
> > Shouldn't be to hard (although it's probably not on each oslo project,
> > but on the consumers projects).
> >
> > The warnings module can turn warnings into raised exceptions with a
> > simple command line switch btw...
> >
> > For example:
> >
> > $ python -Wonce
> > Python 2.7.6 (default, Jun 22 2015, 17:58:13)
> > [GCC 4.8.2] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
>  import warnings
>  warnings.warn("I am not supposed to be used", DeprecationWarning)
> > __main__:1: DeprecationWarning: I am not supposed to be used
> >
> > $ python -Werror
> > Python 2.7.6 (default, Jun 22 2015, 17:58:13)
> > [GCC 4.8.2] on linux2
> > Type "help", "copyright", "credits" or "license" for more information.
>  import warnings
>  warnings.warn("I am not supposed to be used", DeprecationWarning)
> > Traceback (most recent call last):
> >   File "", line 1, in 
> > DeprecationWarning: I am not supposed to be used
> >
> > https://docs.python.org/2/library/warnings.html#the-warnings-filter
> >
> > Turn that CLI switch from off to on and I'm pretty sure usage of
> > deprecated things will become pretty evident real quick ;)
>
> It needs to be more targetted than that. There is a long standing
> warning between paste and pkg_resources that would hard stop everyone.
>
> But, yes, the idea of being able to run unit tests with fatal
> deprecations of oslo easily is what I think would be useful.
>
> -Sean
>
>
In keystone we set a warnings filter for the unit tests so that if keystone
calls any deprecated function it'll raise[1]. So when the oslo timeutils
functions were deprecated it broke keystone gate and we fixed it. It would
be nicer to have a non-voting gate job to serve as a warning instead, but
it's only happened a couple of times where this caused keystone to be
blocked for the day that it took to get the fix in. Anyways, it would be
easy enough for us to have this enabled/disabled via an environment
variable and create a tox job.

If we had a non-voting warning job it could also run oslo libs from master
rather than released.

[1]
http://git.openstack.org/cgit/openstack/keystone/tree/keystone/tests/unit/core.py?id=4f8c4a7a10d85080d6db9b30ae1759d45a38a32c#n460

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Brant Knudson]

On Tue, Dec 1, 2015 at 12:57 AM, Steve Martinelli 
wrote:

> Trying to summarize here...
>
> - There isn't much interest in keeping eventlet around.
> - Folks are OK with running keystone in a WSGI server, but feel they are
> constrained by Apache.
> - uWSGI could help to support multiple web servers.
>
> My opinion:
>
> - Adding support for uWSGI definitely sounds like it's worth
> investigating, but not achievable in this release (unless someone already
> has something cooked up).
>

What needs to change to support uWSGI? You can already run keystone in
python uwsgi and then front it with nginx:

 $ uwsgi --socket 127.0.0.1:5001 --wsgi-file $(which keystone-wsgi-public)
--honour-stdin --enable-threads --workers 6
 $ uwsgi --socket 127.0.0.1:35358 --wsgi-file $(which keystone-wsgi-admin)
--honour-stdin --enable-threads --workers 6

 $ sudo vi /etc/nginx/sites-available/keystone

server {
  listen 5000 default_server;
  server_name localhost;
  location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:5001;
uwsgi_param SCRIPT_NAME /;
  }
}
server {
  listen 35357 default_server;
  server_name localhost;
  location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:35358;
uwsgi_param SCRIPT_NAME /;
  }
}

 $ sudo ln -x /etc/nginx/sites-available/keystone /etc/nginx/sites-enabled/

 $ sudo nginx

and then you can make your regular curl calls.

Also, you can run keystone with regular http in python uwsgi (uwsgi --http)
and then just do normal reverse proxy (from Apache or nginx or whatever),
which I think would be adequate for keystone.

We don't do anything in keystone to stop deployments in web servers other
than Apache. Keystone is just a regular wsgi app. We document Apache since
it's popular and it provides mod_shib, which is the only saml2 module for
web servers that I know of. Keystone can work with other saml2 modules and
in different servers, it just takes the environment variables that the
module sets and runs it through some mapping code. The mapping code has
been shown to work alternative authentication modules (for ldap and
kerberos).

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] Auth_version from 'old style' URLs in the database

[Brant Knudson]

On Thu, Dec 3, 2015 at 10:24 AM, Bunting, Niall 
wrote:

> Hi,
>
> Currently glance will use an auth_url if in the database. Eg.
> 10.0.0.8:5000/v2.0
>
> However glance currently takes the auth_version from the config
> files. Therefore this can lead to a mismatch of keystone version to be used
> between the url and the config files. This is problematic due to a
> different
> resource id being required in different version of keystone (in keystone v2
> it was /v2.0/tokens in keystone v3 it is /v3/auth/tokens).
>
> Using a v2 url and config file with keystone v3:
> 10.0.0.8:5000/v2.0/auth/tokens -- Fails to authenticate the user,
> and user can't download image.
>
> See https://bugs.launchpad.net/glance-store/+bug/1507610 for a bug report
> on this.
>
> This means that the fix proposed by
> https://review.openstack.org/#/c/238074/ parses the URL for an
> auth_version
> and then if found will use the parsed value as the auth_version rather than
> the one from the config files. Taking the url as the true source.
> Therefore the image will still work as the auth_version used by glance is
> the
> one defined in the URL meaning the correct resource id appended.
>
> Whilst discussing it with Kairat it was proposed that we ignore the
> keystone version in the URL and if it does not support the auth_version
> in the configs, then the image would fail to be downloaded. This is due to
> a
> preference to have a centralised auth_version value.
>
> I am wondering what people would prefer to do, to support the 'old style'
> urls
> and therefore parse the version from the url. Or to make the auth_version
> common and potentially break the 'old style' database entries.
>
> Thanks,
> Niall Bunting
>
>
If you want to know the version supported by the auth_url, do a GET on it
and the JSON returned will tell you what version(s) are supported. This is
preferred since parsing the value is error-prone and may not support all
deployment options (maybe somebody puts keystone v2 on /v2_old or
something). The keystoneauth library provides functions for getting the
version of the service[1].

Also, the keystoneauth library provides functions for all this stuff.
Glance shouldn't have to deal with this at all since you can just point
keystoneauth to the config section and it'll load the correct plugin and
create a session that your app can use to do its operations without
worrying about the specifics of authentication.

[1]
http://docs.openstack.org/developer/keystoneauth/api/keystoneauth1.html#module-keystoneauth1.discover

- Brant



>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone][stable] nominating lin hua cheng for keystone-stable-maint

[Brant Knudson]

On Tue, Nov 17, 2015 at 5:24 PM, Steve Martinelli 
wrote:

> I'd like to nominate Lin Hua Cheng for keystone-stable-maint. He has been
> doing reviews on keystone's liberty and kilo stable branches since mitaka
> development has opened, and being a member of horizon-stable-maint, he is
> already familiar with stable branch policies. If there are no objections
> from the current stable-maint-core and keystone-stable-maint teams, then
> I'd like to add him.
>
> Thanks,
>
> Steve Martinelli
> Keystone Project Team Lead
>
>
+1 from me. - Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Oslo][TaskFlow] Proposal for new core reviewer (greg hill)

[Brant Knudson]

Looks like he's doing good reviews so +1. - Brant

On Wed, Nov 11, 2015 at 2:02 PM, Joshua Harlow 
wrote:

> Greetings all stackers,
>
> I propose that we add Greg Hill[1] to the taskflow-core[2] team.
>
> Greg (aka jimbo) has been actively contributing to taskflow for a
> while now, both in helping make taskflow better via code
> contribution(s) and by helping spread more usage/knowledge of taskflow
> at rackspace (since the big-data[3] team uses taskflow internally).
> He has helped provided quality reviews and is doing an awesome job
> with the various taskflow concepts and helping make taskflow the best
> it can be!
>
> Overall I think he would make a great addition to the core review team.
>
> Please respond with +1/-1.
>
> Thanks much!
>
> - Joshua Harlow
>
> [1] https://launchpad.net/~greg-hill
> [2] https://launchpad.net/taskflow
> [3] http://www.rackspace.com/cloud/big-data
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] [infra] speeding up gate runs?

[Brant Knudson]

On Wed, Nov 4, 2015 at 6:47 AM, Sean Dague  wrote:

> I was spot checking the grenade multinode job to make sure it looks like
> it was doing the correct thing. In doing so I found that ~15minutes of
> it's hour long build time is compiling lxml and numpy 3 times each.
>
>
I've always wondered why lxml is used rather than python's built-in XML
support. Is there some function that xml.etree is missing that lxml.etree
provides? The only thing I know about is that lxml has better support for
some XPATH features.

:: Brant


> Due to our exact calculations by upper-constraints.txt we ensure exactly
> the right version of each of those in old & new & subnode (old).
>
> Is there a nodepool cache strategy where we could pre build these? A 25%
> performance win comes out the other side if there is a strategy here.
>
> -Sean
>
>



> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][bandit] Handling bandit configuration files in Oslo.

[Brant Knudson]

On Tue, Nov 3, 2015 at 9:41 AM, Cyril Roelandt  wrote:

> On 11/02/2015 07:40 PM, Brant Knudson wrote:
>
>>
>> We could use something like this in keystone since we've got a few
>> repositories. There should be a way to document why the test was skipped
>> since otherwise we'll have to figure it out every time we update the
>> file. Putting a comment on the command line would wind up being
>> unwieldy, so we should have a config file for bandit-conf-generator...
>> but then why not just have bandit know how to read the
>> bandit-conf-generator config file and skip the extra step?
>>
>
>
> The bandit.yaml from python-keystoneclient supports multiple profiles,
> which is already something my tool, in its current state, cannot do.
>
>
The only reason for that is because I haven't updated it yet to match the
normal usage where there is only the 1 gate profile.

:: Brant


> I don't know exactly which set of features should be supported by a
> configuration generator. If it becomes too hard to write the configuration
> for the configuration generator, we might as well just write the
> configuration for bandit manually :⁾
>
> See my answer to Victor about enhancing Bandit so that it can read a
> "simpler" config file. I'm not a big fan of it.
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][bandit] Handling bandit configuration files in Oslo.

[Brant Knudson]

On Mon, Nov 2, 2015 at 12:22 PM, Cyril Roelandt  wrote:

> Hello,
>
> The libraries from the Oslo project are used everywhere in OpenStack,
> which means that a security issue in Olso code might have an impact on a
> lot of other projects. This is why I am currently trying to add support
> for the bandit[1] static checker in all of the Oslo libraries.
>
> While reviewing one of my patches[2], Victor Stinner noticed that the
> bandit configuration file (bandit.yaml) I proposed, which is basically a
> copy of the example config file[3] provided by the bandit project with
> some minor changes, might be a bit hard to maintain across all Oslo
> projects. Indeed, all configuration files could potentially have to be
> changed whenever a new checker is added to bandit, for instance.
>
> In order to make it easier to keep an up-to-date configuration file, I
> quickly wrote a proof of concept[4] that allows developers to generate a
> configuration file that fits their needs. One can now generate a working
> bandit.yaml configuration file by typing something like:
>
> $ bandit-conf-generator --disable try_except_pass --out bandit.yaml
> oslo.messaging ~/openstack/bandit/bandit/config/bandit.yaml
>
> Whenever a new version of bandit comes out, one can grab the latest
> config file example from the bandit release, and re-run the above
> command. The generated config file will include all the new checkers.
>
> What do you think? Could this be a useful tool to handle bandit
> configurations?
>
>
We could use something like this in keystone since we've got a few
repositories. There should be a way to document why the test was skipped
since otherwise we'll have to figure it out every time we update the file.
Putting a comment on the command line would wind up being unwieldy, so we
should have a config file for bandit-conf-generator... but then why not
just have bandit know how to read the bandit-conf-generator config file and
skip the extra step?

- Brant


> Cyril Roelandt.
> ---
>
> [1] https://wiki.openstack.org/wiki/Security/Projects/Bandit
> [2] https://review.openstack.org/#/c/239666/
> [3]
> https://github.com/openstack/bandit/blob/master/bandit/config/bandit.yaml
> [4] https://github.com/CyrilRoelandteNovance/bandit_conf_generator
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Require documenting changes with versionadded and versionchanged

[Brant Knudson]

On Thu, Oct 15, 2015 at 5:52 AM, Victor Stinner  wrote:

> Hi,
>
> I propose that changes must now be documented in Oslo libraries. If a
> change is not documented, it must *not* be approved.
>
> IMHO it's very important to document all changes. Otherwise, it becomes
> really hard to guess if a specific parameter or a specific function can be
> used just by reading the doc :-/ And we should not force users to always
> upgrading the Oslo libraries to the latest versions. It doesn't work on
> stable branches :-)
>
> Currently, ".. versionadded:: x.y" and ".. versionchanged:: x.y" are not
> (widely) used in Oslo libraries. A good start would be to dig the Git
> history to add these tags. I started to do this for the oslo.config library:
> https://review.openstack.org/#/c/235232/
>
> I'm interested to write similar changes for other Oslo libraries.
>
> Because I created this change, I started to vote -1 on all patches which
> oslo.config changes the API without documenting the doc.
>
> What do you think?
>
>
Victor
>
>
Submitters don't know what release their change is going to get into. They
might submit the review when version 1.1.0 is current so they mark it with
added in 1.2.0, but then the change doesn't get merged until after 1.4.0 is
tagged. Also, the submitter doesn't know what the next release is going to
be tagged as, since it might be 1.2.0 or it might be 2.0.0.

So this will create extra churn as reviews have to be updated with the
release #, and the docs will likely be wrong when reviewers forget to check
it (unless this can be automated).

We have the same problem with documenting when something is deprecated.

I don't think it's worth it to document in which release something is added
in the oslo libs. We're not charging for this stuff so if you want the
online docs to match your code use the latest. Or check out the tag and
generate the docs for the release you're on to look at to see if the
feature is there.

:: Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Optional Dependencies

[Brant Knudson]

On Tue, Sep 29, 2015 at 9:38 AM, Douglas Mendizábal <
douglas.mendiza...@rackspace.com> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> Hi openstack-dev,
>
> I was wondering what the correct way of handling optional dependencies
> is? I'm specifically talking about libraries that are only required
> for a specific driver for example, so they're not technically a hard
> requirement and the project is expected to function without them when
> using a driver that does not require the lib.
>
>
We've got some in keystone, so for example the packages for ldap won't get
installed unless it's requested. One of the reasons we chose this one
specifically is that the python ldap package requires a C library.

We set up the different extra package groups in setup.cfg [extras]:

[1]
http://git.openstack.org/cgit/openstack/keystone/tree/setup.cfg?h=stable/liberty#n24

See pbr docs: http://docs.openstack.org/developer/pbr/#extra-requirements

Here's the tox.ini line so that they're all installed for unit tests:
http://git.openstack.org/cgit/openstack/keystone/tree/tox.ini?h=stable/liberty#n10



> I read through the README in openstack/requirements [1] but I didn't
> see anything about it.
>
>
> Thanks,
> Douglas Mendizábal
>
> [1]
> https://git.openstack.org/cgit/openstack/requirements/tree/README.rst
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - https://gpgtools.org
>
> iQIcBAEBCgAGBQJWCqJlAAoJEB7Z2EQgmLX72D8P/RROP9qT7DRY1jDnbK0Aj/TZ
> lYujurHS70nXCj/Pw6uqsq41TttmwMdAx85yXwoLv/XBASaFYZ6eT6i0scfHBKAu
> z5f0IomaJMQDGJ27By/amcE5eMiST5sEW/OCwHyZxdM8zgo3mzX1jIslmFEyPJ0z
> wSah5DoZZh3J0RfQuBg8MOQgJVZo74KiNRou1uKE82cbVXJzVKjlfn+r7yO9TUtx
> 9hB/77a8sDFBWI4nXluTP+Dfpy6NSW1kqwwUoDtsZACtrhTDNCDWxUUIjfyBlIKT
> LdY+oVrhqWSUI/WwCop4+Aim64obaAq5yWPR6fjTlcQ3+iCYbBzzgP/9VOm/+0Nr
> AGzVbIW7ah2yEDhM0yTymaay8+G1mc+jxhvwAtTxJVIJLcJXdC3XK6b00OFkO2Kt
> 0dkjx/i8/riP56sb62P2a3heS3gOFqzqzwlh9SD8Omvhot3NkOr2e1QR7Cvjh1le
> W5U/61vGKxmtv+iIaFXd86CRO46+4UiD1V+T0lKz083J9XuC49nkhyfuMP3ev6lc
> /qD6uOnbJfyVWKRdf2PkTEe9C8YsXlxEWZ72GFC+u1jvL5K/NATUkLLWmGuv/JH+
> tPyAOPISKHh44mhJqM/K37NvJO/TloOhz0a2fW2FV8kOX1V5wVAZiQBSEWtCAI8u
> 29up4yIgvi13ZkrRb94n
> =fj9z
> -END PGP SIGNATURE-
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] How to make a mock effactive for all method of a testclass

[Brant Knudson]

On Wed, Sep 23, 2015 at 3:06 AM, liuxinguo  wrote:

> Hi,
>
>
>
> In a.py we have a function:
>
> def *_change_file_mode*(filepath):
>
> utils.execute(*'chmod'*, *'600'*, filepath, run_as_root=True)
>
>
>
> In test_xxx.py, there is a testclass:
>
> clas*s DriverTestCase(test.TestCase):*
>
> *def test_a(self)*
>
> *…*
>
> *Call a. _change_file_mode*
>
> *…*
>
>
>
> *def test_b(self)*
>
> *…*
>
> *Call a. _change_file_mode*
>
> *…*
>
>
>
> I have tried to mock like mock out function *_change_file_mode *like this:
>
> *@mock.patch.object*(a, *'_change_file_mode', *return_value=None)
>
> clas*s DriverTestCase(test.TestCase):*
>
> *def test_a(self)*
>
> *…*
>
> *Call a. _change_file_mode*
>
> *…*
>
>
>
> *def test_b(self)*
>
> *…*
>
> *Call a. _change_file_mode*
>
> *…*
>
>
>
> But the mock takes no effort, the real function _change_file_mode is still
> executed.
>
> So how to make a mock effactive for all method of a testclass?
>
> Thanks for any input!
>
>
>

Use oslotest's mockpatch.PatchObject fixture:
http://docs.openstack.org/developer/oslotest/api.html#oslotest.mockpatch.PatchObject


> Wilson Liu
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] New Gerrit translations change proposals from Zanata

[Brant Knudson]

On Tue, Sep 15, 2015 at 4:30 PM, Elizabeth K. Joseph 
wrote:

> Hi everyone,
>
>
...


>
> The Gerrit topic for these change proposals for all projects with
> translations has been changed from "transifex/translations" to
> "zanata/translations". After a test with oslo.versionedobjects last
> week[1], we're moving forward this Wednesday morning UTC time to have
> the jobs run so that all translations changes proposed to Gerrit are
> made by Zanata.
>
>
According to [0], we'll have the final releases of libraries today, so if
translations aren't there yet it's already too late for them.

[0]
http://lists.openstack.org/pipermail/openstack-dev/2015-September/074367.html

:: Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] creating new users with invalid mail addresses possible

[Brant Knudson]

On Fri, Sep 11, 2015 at 7:26 AM, Christian Berendt 
wrote:

> At the moment it is possible to create new users with invalid mail
> addresses. I pasted the output of my test at
> http://paste.openstack.org/show/456642/. (the listing of invalid mail
> addresses is available at
> http://codefool.tumblr.com/post/15288874550/list-of-valid-and-invalid-email-addresses
> ).
>
> Is it intended that addresses are not be validated?
>
> Does it makes sense to validate addresses (e.g. with
> https://github.com/mailgun/flanker)?
>
> Christian.
>
> --
> Christian Berendt
> Cloud Solution Architect
> Mail: bere...@b1-systems.de
>
> B1 Systems GmbH
> Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
> GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

If you back keystone users with ldap and map the keystone email address
property to an attribute that the ldap server validates then you'll get
email validation. (I haven't tried it but that's the theory.)

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] contextlib.nested and Python3 failing

[Brant Knudson]

On Wed, Aug 19, 2015 at 6:51 PM, Sylvain Bauza  wrote:

> Hi,
>
> I was writing some tests so I added a contextlib.nested to a checked
> TestCase [1]. Unfortunately, contextlib.nested is no longer available in
> Python3 and there is no clear solution on how to provide a compatible
> import for both python2 and python3:
>  - either providing a python3 compatible behaviour by using
> contextlib.ExitStack but that class is not available in Python 2
>  - or provide contextlib2 for python2 (and thus adding it to the
> requirements)
>
> That sounds really disruptive and blocking as we are close to the
> FeatureFreeze. Many other users of contextlib.nested are not impacted by
> the job because it excludes all of them but since the test I'm changing is
> part of the existing validated tests, that leaves Jenkins -1'ing my change.
>
> Of course, a 3rd solution would consist of excluding my updated test from
> the python3 check but I can hear others yelling at that :-)
>
> Ideas appreciated.
>
> -Sylvain
>
> [1]
> https://review.openstack.org/#/c/199205/18/nova/tests/unit/scheduler/test_rpcapi.py,cm
>
>
>
Mock provides a context that patches multiple things so that no nesting is
needed: http://www.voidspace.org.uk/python/mock/patch.html#patch-multiple

oslotest provides fixtures for mock, so you don't need a context:
http://docs.openstack.org/developer/oslotest/api.html#module-oslotest.mockpatch

  __ Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Need volunteers to test coverage 4.0 alpha release

[Brant Knudson]

On Mon, Aug 3, 2015 at 6:32 AM, Doug Hellmann  wrote:

> Do we have anyone willing to spend some time looking at the
> incompatibilities Ned mentions below to ensure that we have a smooth
> transition when 4.0 moves out of alpha?
>
> Doug
>
>
Seems to work ok with keystone. There were a couple of problems running it
but they were easy to work around.

When starting, "Unrecognized option '[report] ignore-errors=' in config
file .coveragerc"; removed the section and didn't get the error:
-
-[report]
-ignore-errors = True

At end of run, "Couldn't read data from '/opt/stack/keystone/.coverage':
CoverageException: Doesn't seem to be a coverage.py data file"; deleted
.coverage and ran again, I think it was left over from old run.

Results are like:
Old summary: Total   16846   1442194 4239434 90%
New summary: Total   16846   1442194 4239435 90%
it found another "Partial" line somewhere.

-- Brant



> --- Begin forwarded message from Ned Batchelder ---
> From: Ned Batchelder 
> To: testing-in-python 
> Date: Mon, 03 Aug 2015 07:22:30 -0400
> Subject: [TIP] Coverage.py 4.0 beta 1
>
> Hi all,
>
> After more than 18 months of development, the first beta of Coverage.py
> 4.0 is available: https://pypi.python.org/pypi/coverage/4.0b1
>
> There are a number of very significant changes since v3.7.1, including
> backward-incompatibilities.  I'd really appreciate getting some testing
> from people using third-party coverage integrations, or using unusual
> configurations.
>
> Bug reports heartily welcomed: https://bitbucket.org/ned/coveragepy
>
> A quick list of the biggest changes are below, more at
> http://coverage.readthedocs.org/en/coverage-4.0b1/changes.html
>
> Thanks,
>
> --Ned.
>
>
>
> Backward incompatibilities:
>
> - CPython versions supported are now Python 2.6, 2.7, 3.3, 3.4 and 3.5b4.
>PyPy2 2.4, 2.6, and PyPy3 2.4 are also supported.
>
> - The original command line switches (`-x` to run a program, etc) are no
>longer supported.
>
> - The ``COVERAGE_OPTIONS`` environment variable is no longer supported.
> It was
>a hack for ``--timid`` before configuration files were available.
>
> - The original module-level function interface to coverage.py is no longer
>supported.  You must now create a ``coverage.Coverage`` object, and use
>methods on it.
>
> - The ``Coverage.use_cache`` method is no longer supported.
>
> - The private method ``Coverage._harvest_data`` is now called
>``Coverage.get_data``, and returns the ``CoverageData`` containing the
>collected data.
>
> - Coverage.py is now licensed under the Apache 2.0 license. See
> NOTICE.txt for
>details.
>
>
> Major new features:
>
> - Plugins: third parties can write plugins to add file support for
> non-Python
>files, such as web application templating engines, or languages that
> compile
>down to Python.  A plugin for measuring Django template coverage is
>available: `django_coverage_plugin`_
>
> - The data storage has been re-written, using JSON instead of pickle.  The
>:class:`CoverageData` class is a new supported API to the contents of
> the
>data file.
>
> - Gevent, eventlet, and greenlet are now supported, closing `issue
> 149`_.  The
>``concurrency`` setting, or the ``--concurrency`` command line switch,
>specifies the concurrency library in use.  Huge thanks to Peter
> Portante for
>initial implementation, and to Joe Jevnik for the final insight that
>completed the work.
>
> - Wildly experimental: support for measuring processes started by the
>multiprocessing module.  To use, set ``--concurrency=multiprocessing``,
>either on the command line or in the .coveragerc file (`issue 117`_).
> Thanks,
>Eduardo Schettino.  Currently, this does not work on Windows.
>
>
> New features:
>
> - Options are now also read from a setup.cfg file, if any.  Sections are
>prefixed with "coverage:", so the ``[run]`` options will be read from
> the
>``[coverage:run]`` section of setup.cfg.  Finishes `issue 304`_.
>
> - A new option: `coverage report --skip-covered` (or ``[report]
> skip_covered``)
>will reduce the number of files reported by skipping files with 100%
>coverage.  Thanks, Krystian Kichewko.  This means that empty
> `__init__.py`
>files will be skipped, since they are 100% covered, closing `issue
> 315`_.
>
> - You can now specify the ``--fail-under`` option in the ``.coveragerc``
> file
>as the ``[report] fail_under`` options.  This closes `issue 314`_.
>
> - The ``report`` command can now show missing branches when reporting on
> branch
>coverage.  Thanks, Steve Leonard. Closes `issue 230`_.
>
> - The ``coverage combine`` command now accepts any number of directories or
>files as arguments, and will combine all the data from them.  This
> means you
>don't have to copy the files to one directory before combining. Thanks,
>Christine Lytwynec.  Finishes `issue 354`_.
>
> - A new configuration option for the XML

Re: [openstack-dev] [keystone] the status of ec2 api

[Brant Knudson]

On Wed, Jul 29, 2015 at 1:49 PM, Brant Knudson  wrote:

>
>
> On Mon, Jul 13, 2015 at 9:50 PM, Ken'ichi Ohmichi 
> wrote:
>
>> Hi Keystone team,
>>
>> Now the test of ec2 credentials[1] is been proposed to Tempest, and
>> I'd like to know current situation of ec2 api on Keystone as a Tempest
>> reviewer.
>>
>>
Why are keystone functional tests being proposed to tempest and not to
keystone?


> On Nova instead, ec2 api is deprecated in Nova and the standalone
>> service of ec2 api is separated from Nova to
>> https://github.com/stackforge/ec2-api .
>> As the result, ec2 api tests of Nova are defined as "thirdparty" and
>> these tests don't run on normal checks/gates on the gate.
>> If ec2 api is deprecated on Keystone side also, it seems better to
>> define these tests as "thirdparty".
>> That is a reason why I'd like to know current situation of Keystone ec2
>> api.
>>
>> Thanks
>> Ken Ohmichi
>>
>>
> There hasn't been any discussion of deprecating EC2 APIs in keystone, and
> I don't know why we would since it hasn't been causing any problems. Every
> once in a while we get a proposal from someone to implement a new version.
>
>   - Brant
>
> ---
>> [1]: https://review.openstack.org/#/c/194584
>>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] the status of ec2 api

[Brant Knudson]

On Mon, Jul 13, 2015 at 9:50 PM, Ken'ichi Ohmichi 
wrote:

> Hi Keystone team,
>
> Now the test of ec2 credentials[1] is been proposed to Tempest, and
> I'd like to know current situation of ec2 api on Keystone as a Tempest
> reviewer.
>
> On Nova instead, ec2 api is deprecated in Nova and the standalone
> service of ec2 api is separated from Nova to
> https://github.com/stackforge/ec2-api .
> As the result, ec2 api tests of Nova are defined as "thirdparty" and
> these tests don't run on normal checks/gates on the gate.
> If ec2 api is deprecated on Keystone side also, it seems better to
> define these tests as "thirdparty".
> That is a reason why I'd like to know current situation of Keystone ec2
> api.
>
> Thanks
> Ken Ohmichi
>
>
There hasn't been any discussion of deprecating EC2 APIs in keystone, and I
don't know why we would since it hasn't been causing any problems. Every
once in a while we get a proposal from someone to implement a new version.

  - Brant

---
> [1]: https://review.openstack.org/#/c/194584
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Python 3: 5 more projects with a py34 voting gate, only 4 remaing

[Brant Knudson]

On Fri, Jul 17, 2015 at 7:32 AM, Victor Stinner  wrote:

> Hi,
>
> We are close to having a voting py34 gate on all OpenStack libraries and
> applications. I just made the py34 gate voting for the 5 following projects:
>
> * keystone
>

...

> Victor
>
>
While there's a py34 gate for keystone, it doesn't actually test much, so
let's not declare victory just yet. The short list of tests to run are in
keystone's tox.ini[1]. Here's the error I get when I add
keystone/tests/unit/backend/domain_config/test_sql.py to the list:
http://paste.openstack.org/show/392971/ . So another library that we use
that doesn't support py34 is dogpile.cache (or maybe keystone isn't calling
it correctly).

[1] http://git.openstack.org/cgit/openstack/keystone/tree/tox.ini#n20

Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Python 3: 5 more projects with a py34 voting gate, only 4 remaing

[Brant Knudson]

On Fri, Jul 17, 2015 at 7:32 AM, Victor Stinner  wrote:

> Hi,
>
> ...

> (3) keystonemiddleware: blocked by python-memcached, I sent a pull request
> 3 months ago and I'm still waiting...
>
> https://github.com/linsomniac/python-memcached/pull/67
>
> I may fork the project if the maintainer never reply. Read the current
> thread "[all] Non-responsive upstream libraries (python34 specifically)" on
> openstack-dev.
>
>
...


> Victor
>
>
keystonemiddleware has had a py34 gate for a long time. The tests run
without python-memcache installed since the tests are skipped if memcache
isn't available. We've got a separate test-requirements-py34.txt that
doesn't include python-memcache. This has been causing problems lately
since the requirements job now fails since there are duplicate requirements
in multiple files (test-requirements-py34 is just test-requirements with
python-memcache removed).

I proposed a change to global-requirements to mark python-memcache as not
working on py34: https://review.openstack.org/#/c/203437/

and then we'll have to change keystonemiddleware to merge the
test-requirements-py34 into test-requirements:
https://review.openstack.org/#/c/197254/ .

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] apache wsgi application support

[Brant Knudson]

On Wed, Jun 24, 2015 at 7:27 AM, Chris Dent  wrote:

> On Wed, 24 Jun 2015, Sean Dague wrote:
>
>> On 06/24/2015 07:57 AM, Chris Dent wrote:
>>
>>> If the primary reason is so that we can rely on the console_scripts
>>> entry point to handle getting the application somewhere useful then
>>> that too feels a bit crufty, in the sense of "that's a hack".
>>>
>>
> [snip]
>
>  The reason I want this is so that the upgrade process for keystone is:
>>
>> pip install ./keystone
>>
>> And not have to also have knowledge about the contents of the keystone
>> source directory. Today a lot of installation and upgrade logic for
>> packages is "left as an exercise for the reader", which means devstack,
>> ansible, rpms, debs all end up doing a bunch of work beyond pip install.
>> Minimizing that by bringing more of this back into what pip install does
>> will make for a more common base moving forward for everyone.
>>
>
> Makes sense. My only concern would be the somewhat of a mismatch in
> semantics between console_scripts and what is not a console script.
>
> Does anyone who is more familiar with entry points know of an
> alternative that will accomplish the same sort of thing?
>
> If there's not, then awesome, let's do it.
>
> --
> Chris Dent tw:@anticdent freenode:cdent
> https://tank.peermore.com/tanks/cdent
>
>
I tried console_scripts and there is a mismatch between what a console
script is and a wsgi script. A wsgi script must export an "application"
symbol and the generated console script doesn't. Maybe setuptools will be
enhanced someday to also support wsgi scripts but until then I think we're
stuck.

One thing I should have looked at earlier is how devstack handles horizon,
since it's already running in Apache Httpd. The devstack horizon setup uses
the wsgi script out of the devstack source directories:

WSGIScriptAlias /
/opt/stack/horizon/openstack_dashboard/wsgi/django.wsgi

Docs:
[1]
https://pythonhosted.org/setuptools/setuptools.html#automatic-script-creation
-- console_scripts and gui_scripts
[2] docs.openstack.org/developer/pbr/#files -- I couldn't get this to work
for some reason

-Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] apache wsgi application support

[Brant Knudson]

On Wed, Jun 17, 2015 at 1:21 PM, Sean Dague  wrote:

> On 06/16/2015 05:25 PM, Chris Dent wrote:
> > On Tue, 16 Jun 2015, Sean Dague wrote:
> >
> >> I was just looking at the patches that put Nova under apache wsgi for
> >> the API, and there are a few things that I think are going in the wrong
> >> direction. Largely I think because they were copied from the
> >> lib/keystone code, which we've learned is kind of the wrong direction.
> >
> > Yes, that's certainly what I've done the few times I've done it.
> > devstack is deeply encouraging of cargo culting for reasons that are
> > not entirely clear.
>
> Yeh, hence why I decided to put the brakes on a little here and get this
> on the list.
>
> >> The first is the fact that a big reason for putting {SERVICES} under
> >> apache wsgi is we aren't running on a ton of weird unregistered ports.
> >> We're running on 80 and 443 (when appropriate). In order to do this we
> >> really need to namespace the API urls. Which means that service catalog
> >> needs to be updated appropriately.
> >
> > So:
> >
> > a) I'm very glad to hear of this. I've been bristling about the weird
> >ports thing for the last year.
> >
> > b) You make it sound like there's been a plan in place to not use
> >those ports for quite some time and we'd get to that when we all
> >had some spare time. Where do I go to keep abreast of such plans?
>
> Unfortunately, this is one of those "in the ether" kinds of plans. It's
> been talked about for so long, but it never really got written down.
> Hopefully this can be driven into the service catalog standardization
> spec (or tag along somewhere close).
>
> Or if nothing else, we're documenting it now on the mailing list as
> permanent storage.
>
> >> I also think this -
> >>
> https://github.com/openstack-dev/devstack/blob/master/lib/nova#L266-L268
> >> is completely wrong.
> >>
> >> The Apache configs should instead specify access rules such that the
> >> installed console entry point of nova-api can be used in place as the
> >> WSGIScript.
> >
> > I'm not able to parse this paragraph in any actionable way. The lines
> > you reference are one of several ways of telling mod wsgi where the
> > virtualenv is, which has to happen in some fashion if you are using
> > a virtualenv.
> >
> > This doesn't appear to have anything to do with locating the module
> > that contains the WSGI app, so I'm missing the connection. Can you
> > explain please?
> >
> > (Basically I'm keen on getting gnocchi and ceilometer wsgi servers
> > in devstack aligned with whatever the end game is, so knowing the plan
> > makes it a bit easier.)
>
> Gah, the problem of linking to 'master' with line numbers. The three
> lines I cared about were:
>
> # copy proxy vhost and wsgi helper files
> sudo cp $NOVA_DIR/nova/wsgi/nova-api.py $NOVA_WSGI_DIR/nova-api
> sudo cp $NOVA_DIR/nova/wsgi/nova-ec2-api.py $NOVA_WSGI_DIR/nova-ec2-api
>
> I don't think that we should be copying py files around to other
> directories outside of normal pip install process. We should just have
> mod_wsgi reference a thing that is installed in /usr/{local}/bin or
> /usr/share via the python install process.
>
> >> This should also make lines like -
> >> https://github.com/openstack-dev/devstack/blob/master/lib/nova#L272 and
> >> L274 uneeded. (The WSGI Script will be in a known place). It will also
> >> make upgrades much more friendly.
> >
> > It sounds like maybe you are saying that the api console script and
> > the module containing the wsgi 'application' variable ought to be the
> > same thing. I don't reckon that's a great idea as the api console
> > scripts will want to import a bunch of stuff that the wsgi application
> > will not.
> >
> > Or I may be completely misreading you. It's been a long day, etc.
>
> They don't need to be actually the same thing. They could be different
> scripts, but they should be scripts that install via the normal pip
> install process to a place, and we reference them by known name.
>
> >> I think that we need to get these things sorted before any further
> >> progression here. Volunteers welcomed to help get us there.
> >
> > Find me, happy to help. The sooner we can kill wacky port weirdness
> > the better.
>
> Agreed.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
>
I've got a few related changes proposed to keystone and devstack:

https://review.openstack.org/#/c/193891/ - Changes Apache Httpd config so
that /identity is the keystone public (aka main) handler and
/identity_admin is the keystone admin handler. Httpd can have multiple
aliases for the same wsgi handler so :5000 and :35357 still work. The
follow-on patch at https://review.openstack.org/193894 shows some further
work to change config so that the new endpoints are used by the tests.
There are a lot of devstack variables that aren't going to apply or are
going to be reinterpreted if we switch to this so I'll have to think about
how that's going to work.

https://review.op

Re: [openstack-dev] [ceilometer] can we get ceilometermiddleware to use a config file instead of transport_url?

[Brant Knudson]

On Mon, Jun 22, 2015 at 2:43 PM, Doug Hellmann 
wrote:

> Excerpts from Sean Dague's message of 2015-06-22 13:30:44 -0400:
> > On 06/22/2015 01:15 PM, Michael Krotscheck wrote:
> > > Having _just_ done this, a couple of suggestions.
> > >
> > > - If the middleware is NOT optional - that is, it provides some kind of
> > > a fundamental component or specification of the API, like ETag caching,
> > > CORS, or DB Session management - then the middleware should be added
> > > during the app initialization routine, likely in the app factory. In
> > > this case, we have tight control over the initialization order, and can
> > > make sure that oslo.config is there first. Yay config.
> > >
> > > - If the middleware _is_ optional, then I really feel this problem is
> > > solved by pastedeploy's filter_factory pattern. It lets you define as
> > > many required parameters as you want, and spits back a middleware
> > > instance. That way you have the freedom to set whatever configuration
> > > options you want, or omit the middleware altogether.
> > >
> > > http://pythonpaste.org/deploy/#paste-filter-factory
> > >
> > > Ultimately, what you should want is to ask a factory method for a thing
> > > with some configuration options, and it spits an instance back at you.
> > > If your project doesn't support pastedeploy, that limits your options
> > > (ahem-ironic-ahem). Otherwise, it's really a team decision on whether
> > > something is a 'Fundamental feature' or something that's optional.
> > >
> > > In the case of Ceilometer? Sorry, but I think it's optional. You should
> > > be able to run any component of openstack without ceilometer. So I
> don't
> > > really think it should depend on oslo_config.
> >
> > Ok, here is where I'm confused. Ceilometer middleware *already* depends
> > on oslo_config.
> >
> >
> https://github.com/openstack/ceilometermiddleware/blob/master/ceilometermiddleware/swift.py#L42
> >
> > And it does stuff with it here:
> >
> >
> https://github.com/openstack/ceilometermiddleware/blob/master/ceilometermiddleware/swift.py#L107
> >
> > But regardless, it uses oslo_messaging, which uses oslo_config. So it
> > seems like the only issue at hand is middleware retriggering conf
> parsing.
> >
> > Because the current model of requiring simultaneous code lands in
> > oslo_messaging and ceilometermiddleware, and simultaneous config updates
> > in every installer and config management system to configure the same
> > thing in 2 different ways, does seem like a long term win. And honestly
> > just prohibits most of what people seem to want to do with new messaging
> > approaches.
>
> I explained this on IRC, but I'll repeat it here as a more permanent
> record.
>
> We should not have the ceilometer middleware (re)initializing
> oslo.config. That combines 2 concerns (setting up configuration and
> doing whatever ceilometermiddleware is already doing) in one piece
> of middleware, and that will make it more difficult to combine
> ceilometermiddleware with other middleware that also wants access
> to the configuration settings.
>
> If the application code isn't setting up oslo.config, then we should
> have another piece of middleware do just that one thing. It can take
> options from paste's config to tell it how to do that work, and then
> other middleware later in the pipeline can rely on oslo.config being set
> up and the config files being loaded.
>
> Doug
>
>
Whatever is decided for this, I assume it'll also apply to the auth_token
middleware, since that uses oslo.config, too. We already have a bug[1] and
a proposed change[2] that isn't passing jenkins.

[1] https://bugs.launchpad.net/keystonemiddleware/+bug/1406218
[2] https://review.openstack.org/#/c/143063/

Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] [all] To changes-since or not to changes-since

[Brant Knudson]

Keystone also has a resource that provides changes since[1], the query
parameter used is "since".

[1]
http://specs.openstack.org/openstack/keystone-specs/api/v3/identity-api-v3-os-revoke-ext.html#list-revocation-events

Ciao, Brant



On Fri, Jun 19, 2015 at 3:25 PM, Sean Dague  wrote:

> On 06/19/2015 05:07 AM, Chris Dent wrote:
> >
> > There's an open question in the API-WG on whether to formalize or
> > otherwise enshrine the concept of a "changes-since" query parameter
> > on collection oriented resources across the projects. The original
> > source of this concept is from Nova's API:
> >
> >
> >
> http://docs.openstack.org/developer/nova/v2/polling_changes-since_parameter.html
> >
> >
> > There are arguments for and against but we've been unable to reach a
> > consensus so the agreed next step was to bring the issue to the
> > mailing list so more people can hash it out and provide their input.
> > The hope is that concerns or constraints that those in the group
> > are not aware of will be revealed and a better decision will be
> > reached.
> >
> > The basic idea of "changes-since" is that it can be used as a way to
> > signal that the requestor is doing some polling and would like to
> > ask "Give me stuff that has changed since the last time I checked".
> > As I understand it, for the current implementations (in Nova and
> > Glance) this means "including stuff that has been deleted". Repeated
> > requests to the resource with a "changes-since" parameter gives a
> > running report on the evolving state of the resource. This is intended
> > to allow "efficient polling"[0].
> >
> > The pro camp on this likes it because this is very useful and quite
> > humane: The requestor doesn't need to know the details of how the
> > query is is implemented under the hood. That is, if there are
> > several timestamps associated with the singular resources in the
> > collection which of those are used for time comparison and which
> > attributes (such as "state" with a value of "deleted") are used to
> > determine if a singular resource is included. The service gets to
> > decide these things and in order for "efficient polling" to actually
> > be achieved it needs to do in order to make effective queries of the
> > data store.
> >
> > The con camp doesn't like it because it introduces magic, ambiguity
> > and inconsistency into the API (when viewed from a cross-project
> > perspective) and one of the defining goals of the working group is
> > to slowly guide things to some measure of consistency. The
> > alternate approach is to formulate a fairly rigorous query system
> > for both filtering[1] and sorting[2] and use that to specify
> > explicit queries that state "I want resources that are newer than time
> > X in timestamp attribute 'updated_at' _and_ have attribute 'state'
> > that is one of 'foo', 'bar' or 'baz'".
> >
> > (I hope I have represented the two camps properly here and not
> > introduced any bias. Myself I'm completely on the fence. If you
> > think I've misrepresented the state of things please post a
> > clarifying response.)
> >
> > The questions come down to:
> >
> > * Are there additional relevant pros and cons for the two proposals?
> > * Are there additional proposals which can address the shortcomings
> >   in either?
> >
> > Thanks for your input.
> >
> > [0] Please try to refrain from responses on the line of "ha!
> > efficiency! that's hilarious!" and "ZOMG, polling, that's so
> > last century". Everybody knows this already and it's not
> > germane to the immediate concerns. We'll get to a fully message
> > driven architecture next week. This week we're still working
> > with what we've got.
> > [1] filtering guideline proposal
> > https://review.openstack.org/#/c/177468/
> > [2] sorting guideline proposal
> > https://review.openstack.org/#/c/145579/
>
> I think that is a reasonable summation. My personal feeling is that if
> 'changes-since' is strictly defined as the AND of 2 other standard
> filters, it's not feeling very relevant to recommend it existing. It's
> now just a 2nd way to do exactly the same thing, and all we can do is
> screw it up ("A man with one watch always knows what time it is. A man
> with two watches is never quite sure.").
>
> If it's left a little more vague of "for resources that you expect to be
> regularly polled, this can provide an interface to only show the
> consumer relevent resources. If you choose to implement this, you must
> document what criteria is used to return resources from the url." And
> allow the possibility that their might be different sensible definitions
> depending on the resource, I'm happier about it.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bi

Re: [openstack-dev] Specify a domain in mapping rules

[Brant Knudson]

You might need these changes, which are proposed but not merged:
https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:stable/kilo+topic:bug/1442787,n,z

Salut, Brant

On Thu, Jun 18, 2015 at 6:04 AM, J. Pablo Martín Cobos 
wrote:

> Hi all,
>
> I'm a Python/Django software developer [1].  We have to do an integration
> of OpenStack and a Shibboleth IdP in my current project.
>
> This is not a easy feature to configure... but finally we got it :-) Now
> we only need specify a domain for the user different to the "Federated"
> default domain. This domain depends on an attribute from the IdP.
>
> Is it possible to get with stable/kilo branch? Is it a feature for the
> next  release? [2] These are my rules:
>
> [
> {
> "local": [
> {
> "user": {
> "name": "{0}",
> "domain": {
> "name": "{1}"
> }
> }
> },
> {
> "group": {
> "id": "0ff59ec2f97646eb9350fe75478f9600"
> }
> }
> ],
> "remote": [
> {
> "type": "identity"
> },
> {
> "type": "domain"
> }
> ]
> }
> ]
>
> I have tested with a lot of rules with little changes:
>
> "domain": {
> "name": "Default"
> }
>
> or
>
> "domain": {
> "id": "default"
> }
>
> or
>
> "domain": {
> "id": "14321243"
> }
>
> etc... and this never works :-(
>
> Could you help me?
>
> REF's
>
> 1. https://github.com/goinnn
> 2.
> https://github.com/openstack/keystone-specs/blob/master/api/v3/identity-api-v3-os-federation-ext.rst
>
> Thanks a lot!!,
>
> --
>
> Pablo Martín
> Software engineer
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][security] Enable user password complexity verification

[Brant Knudson]

On Wed, Jun 3, 2015 at 6:49 AM, David Stanek  wrote:

>
> On Wed, Jun 3, 2015 at 6:04 AM liusheng  wrote:
>
>>  Thanks for this topic, also, I think it is similar situation when
>> talking about keystone users, not only the instances's password.
>>
>>
> In the past we've talked about having more advanced password management
> features in Keystone (complexity checks, rotation, etc). The end result is
> that we are not adding them because we would like to get away from managing
> users in Keystone that way. Instead we are pushing for users to integrate
> Keystone with more fully featured identity products.
>
>

We typically reject it for our SQL backend implementation since there are
other ways to configure the Keystone that support the functionality
already. You can configure keystone to use an LDAP backend or you can use
federation. So there's no reason for us to re-implement and support all
this functionality.

That said, if there was a python library that did password complexity
validation that nova was using and it only required a couple of lines of
code in keystone to support it I wouldn't be against it.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone]Why not common definition about normal HTTP status code like 2xx and 3xx?

[Brant Knudson]

On Tue, Jun 2, 2015 at 4:32 AM, Chenhong Liu 
wrote:

> There is keystone/exception.py which contains Exceptions defined and used
> inside keystone provide 4xx and 5xx status code. And we can use it like:
> exception.Forbidden.code, exception.forbiddent.title
> exception.NotFound.code, exception.NotFound.title
>
> This makes the code looks pretty and avoid error prone. But I can't find
> definition for other status code, like 200, 201, 204, 302, and so on. The
> code in keystone, especially the unit test cases,  just write these status
> code and title explicitly.
>
> How about add those definitions?
>
>
Looks like Python 3.5 added constants in http.HTTPStatus[1]. So someday
we'll be able to use those instead.

[1] https://docs.python.org/3.5/library/http.html#http.HTTPStatus

Sincerely, Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model

[Brant Knudson]

On Fri, May 29, 2015 at 8:11 AM, Thomas Goirand  wrote:

> On 05/28/2015 06:41 PM, Devananda van der Veen wrote:
> > Hi all,
> >
> > tl;dr;
> >
> > At the summit, the Ironic team discussed the challenges we've had with
> > the current release model and came up with some ideas to address them.
> > I had a brief follow-up conversation with Doug and Thierry, but I'd
> > like this to be discussed more openly and for us (the Ironic dev
> > community) to agree on a clear plan before we take action.
> >
> > If Ironic moves to a release:independent model, it shouldn't have any
> > direct effect on other projects we integrate with -- we will continue
> > to follow release:at-6mo-cycle-end -- but our processes for how we get
> > there would be different, and that will have an effect on the larger
> > community.
>
> Just a quick follow-up to voice the Distro's opinion (I'm sure other
> package maintainers will agree with what I'm going to write).
>
> It's fine if you use whatever release schedule that you want. Though
> what's important for us, downstream distro, is that you make sure to
> release a longer term version at the same time as everything else, and
> that you make sure security follow-up is done properly. It's really
> important that you clearly identify the versions for which you will do
> security patch backports, so that we don't (by mistake) release a stable
> version of a given distribution with a something you wont ever give long
> term support for.
>
>

The only way I know of to get this information now is to look in the git
repos, so for example for keystoneclient I can find the security-supported
releases are
0.11 (icehouse) --
http://git.openstack.org/cgit/openstack/python-keystoneclient/log/?h=stable/icehouse
1.1 (juno)
1.3 (kilo)

The rest (for example, 1.2) are not supported.

The place I would expect to find this information is here:
https://wiki.openstack.org/wiki/Releases

tschüß -- Brant



> I'm sure you guys know that already, but better be safe than sorry, and
> other less experience projects may find the above useful.
>
> Cheers,
>
> Thomas Goirand (zigo)
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [pbr] 1.0.1 released

[Brant Knudson]

On Thu, May 21, 2015 at 10:17 AM, Robert Collins 
wrote:

> On 22 May 2015 at 05:14, Brant Knudson  wrote:
> >
> > On Thu, May 21, 2015 at 2:22 AM, YAMAMOTO Takashi <
> yamam...@valinux.co.jp>
> > wrote:
> >>
> >> hi,
> >>
> >> can you explain why you want to use pbr 1.0.1 in the first place?
> >> what's wrong with requiring pbr<1.0 ?
> >>
> >
> > Global requirements currently has "pbr>=0.11,<2.0"[1], and
> > keystonemiddleware has the updated requirement which conflicts with
> > oslo.config, so now the keystonemiddleware gate is broken[2].
>
> You can fix this immediately:
>
> 2015-05-21 13:51:45.876 |   File
> "keystonemiddleware/tests/unit/test_opts.py", line 80, in
> test_entry_point
> 2015-05-21 13:51:45.876 | list_fn = ep.load()
>
> is the problem: test_opts is unnecessarily cross-checking dependencies are
> met.
>
> See https://review.openstack.org/#/c/184319/
>
> (Yes we do need to release all the oslo things, but the test isn't
> interested in dep management, its interested in whether the plugins
> implement the right config options. There is a stevedore API that
> would be even better to use - see the review linked above.)
>
> -Rob
>
>
Here's the proposed change for keystonemiddleware:
https://review.openstack.org/#/c/184864/

Worked for me. Thanks!

 - Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [pbr] 1.0.1 released

[Brant Knudson]

On Thu, May 21, 2015 at 2:22 AM, YAMAMOTO Takashi 
wrote:

> hi,
>
> can you explain why you want to use pbr 1.0.1 in the first place?
> what's wrong with requiring pbr<1.0 ?
>
>
Global requirements currently has "pbr>=0.11,<2.0"[1], and
keystonemiddleware has the updated requirement which conflicts with
oslo.config, so now the keystonemiddleware gate is broken[2].

According to the pkg_resources output, all of these packages are
conflicting:
'oslo.utils', 'python-keystoneclient', 'oslo.config', 'oslo.serialization',
'oslo.i18n', 'oslo.context'.

[1]
http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n94
[2]
http://logs.openstack.org/60/179460/3/check/gate-keystonemiddleware-python26/335fcfa/console.html#_2015-05-21_13_51_45_876

- Brant



> YAMAMOTO Takashi
>
> > Hi openstack-dev team,
> >
> > Can you also release latest oslo.config to allow pbr 1.0.1?
> > In my convenience I want to use it.
> >
> >
> > Thanks,
> > kakuma
> >
> > On Wed, 20 May 2015 11:07:36 +1200
> > Robert Collins  wrote:
> >
> >> We are super psyched to announce the release of:
> >>
> >> pbr 1.0.1: Python Build Reasonableness
> >>
> >> With source available at:
> >>
> >> http://git.openstack.org/cgit/openstack-dev/pbr
> >>
> >> For more details, please see the git log history below and:
> >>
> >> http://launchpad.net/pbr/+milestone/1.0.1
> >>
> >> Please report issues through launchpad:
> >>
> >> http://bugs.launchpad.net/pbr
> >>
> >> Changes in pbr 1.0.0..1.0.1
> >> ---
> >>
> >> b72e446 Remove self.pre_run calls in packaging.py
> >> 8e87679 Update hacking to 0.10.x series
> >>
> >> Diffstat (except docs and test files)
> >> -
> >>
> >> pbr/packaging.py  | 2 --
> >> test-requirements.txt | 2 +-
> >> 2 files changed, 1 insertion(+), 3 deletions(-)
> >>
> >>
> >> Requirements updates
> >> 
> >>
> >> diff --git a/test-requirements.txt b/test-requirements.txt
> >> index 2b33504..6e4521c 100644
> >> --- a/test-requirements.txt
> >> +++ b/test-requirements.txt
> >> @@ -4 +4 @@ fixtures>=0.3.14
> >> -hacking>=0.9.2,<0.10
> >> +hacking>=0.10.0,<0.11
> >>
> >>
> >>
> >> --
> >> Robert Collins 
> >> Distinguished Technologist
> >> HP Converged Cloud
> >>
> >>
> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > --
> > fumihiko kakuma 
> >
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder][neutron][security] Rootwrap discussions at OSSG mid-cycle

[Brant Knudson]

On Thu, May 14, 2015 at 2:48 AM, Angus Lees  wrote:

> On Wed, 13 May 2015 at 02:16 Thierry Carrez  wrote:
>
>> Lucas Fisher wrote:
>> > We spent some time at the OSSG mid-cycle meet-up this week discussing
>> root wrap, looking at the existing code, and considering some of the
>> mailing list discussions.
>> >
>> > Summary of our discussions:
>> https://github.com/hyakuhei/OSSG-Security-Practices/blob/master/ossg_rootwrap.md
>> >
>> > The one line summary is we like the idea of a privileged daemon with
>> higher level interfaces to the commands being run. It has a number of
>> advantages such as easier to audit, enables better input sanitization,
>> cleaner interfaces, and easier to take advantage of Linux capabilities,
>> SELinux, AppArmour, etc. The write-up has some more details.
>>
>> For those interested in that topic and willing to work on the next
>> stage, we'll have a work session on the future of rootwrap in the Oslo
>> track at the Design Summit in Vancouver:
>>
>> http://sched.co/3B2B
>>
>>
> Fwiw, I've continued work on my privsep proposal(*) and how it interacts
> with existing rootwrap.  I look forward to discussing it and alternatives
> at the session.
>
> (*) https://review.openstack.org/#/c/155631
>
>  - Gus
>
>
As part of the OSSG work, I started prototyping changes in Nova where the
goal is to

1) Get all the code that's calling rootwrap into one place so that it's
easy to find, and get tests for this code.
2) Next (or even in step 1 if it's easy enough), tighten the interfaces, so
that rather than providing a function to do "chmod %s %s" it would only
allow whatever chmod nova actually has to do, maybe passing in a server ID
rather than a bare file name.

With this, we should be able to tighten up the rootwrap filters in the same
way, or switch to privsep or whatever we decide to do in the future. So
maybe it looks like rearranging deckchairs on the titanic, but in this case
the deckchairs are blocking the emergency exits.

I didn't get too far in it to even see how viable the approach is since I
was working on other things.

I'll put this session on my calendar.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [CI] gate wedged by tox >= 2.0

[Brant Knudson]

On Thu, May 14, 2015 at 9:41 AM, Matt Riedemann 
wrote:

>
>
> On 5/14/2015 5:46 AM, Sean Dague wrote:
>
>> On 05/14/2015 04:16 AM, Robert Collins wrote:
>>
>>> Tox 2.0 just came out, and it isolates environment variables - which
>>> is good, except if you use them (which we do). So everything is
>>> broken.
>>>
>>> https://review.openstack.org/182966
>>>
>>> Should fix it until projects have had time to fix up their local
>>> tox.ini's to let through the needed variables.
>>>
>>> As an aside it might be nice to get this specifier from
>>> global-requirements, so that its managed in the same place as all our
>>> other specifiers.
>>>
>>
>> This will only apply to tempest jobs, and I see lots of tempest jobs
>> passing without it. Do we have a bug with some failures linked because
>> of it?
>>
>> If this is impacting unit tests, that has to be directly fixed there.
>>
>> -Sean
>>
>>
> python-novaclient, neutron and python-manilaclient are being tracked
> against bug https://bugs.launchpad.net/neutron/+bug/1455102.
>
> Heat is being tracked against bug
> https://bugs.launchpad.net/heat/+bug/1455065.
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
>
Here's the fix in keystoneclient if you need an example:
https://review.openstack.org/#/c/182900/

It just added passenv = OS_*

If you're seeing jobs pass without the workaround then those jobs are
probably not running with tox>=2.0.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] [Murano] Congress devstack installation is failing in murano gate job

[Brant Knudson]

On Wed, Apr 29, 2015 at 9:11 AM, Sean Dague  wrote:

> On 04/29/2015 10:05 AM, Sean Dague wrote:
> > On 04/29/2015 09:29 AM, Filip Blaha wrote:
> >> Hi Serg
> >>
> >> I checked devstack log [1] and it seems that stevedore==1.4.0 was
> >> installed due to congress
> >>
> >> 2015-04-29 11:12:16.120
> >> <
> http://logs.openstack.org/61/178461/1/check/gate-murano-congress-devstack-dsvm/fa01b8f/logs/devstacklog.txt.gz#_2015-04-29_11_12_16_120
> >|
> >> Successfully installed PyYAML-3.11 argparse-1.3.0 jsonpatch-1.9
> >> jsonpointer-1.7 oslo.config-1.11.0 pyOpenSSL-0.15.1
> >> python-cloudfoundryclient-1.0.2 requests-2.6.2 stevedore-1.4.0
> >>
> >> It is last entry in log mentioning installation stevedore
> >>
> >> [1]
> >>
> http://logs.openstack.org/61/178461/1/check/gate-murano-congress-devstack-dsvm/fa01b8f/logs/devstacklog.txt.gz
> >
> > Not, it was installed due to keystonemiddleware -
> >
> http://logs.openstack.org/61/178461/1/check/gate-murano-congress-devstack-dsvm/fa01b8f/logs/devstacklog.txt.gz#_2015-04-29_11_02_13_727
>
> Actually, it's more insideous than this. It's installed at 1.4.0 there,
> then it's downgraded to 1.3.0 when glance is installed, and then it's
> pushed back to 1.4.0 because of:
>
>
> http://logs.openstack.org/61/178461/1/check/gate-murano-congress-devstack-dsvm/fa01b8f/logs/devstacklog.txt.gz#_2015-04-29_11_12_08_418
>
> pip install -U should not be used in the general case, only in *very*
> specific cases.
>
> Whatever is hardcoding -U is the source of your problem.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
>
There's a review posted to update keystonemiddleware's requirements:
https://review.openstack.org/#/c/173972/ . It was broken for a while after
the branch was created due to pycadf installing oslo.messaging. For some
reason it's working now even without the pycadf changes, but Doug H has a
-2 on it. I'd like to know what the reqs are supposed to be for all the
stable/kilo libraries -- I assumed we wanted them all to match
global-requirements.

I think all of our tox.ini's have -U:
http://git.openstack.org/cgit/openstack/nova/tree/tox.ini#n6

- Brant



> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Deprecation path?

[Brant Knudson]

On Fri, Apr 24, 2015 at 2:21 PM, Robert Collins 
wrote:

> I think it needs a three-step dance.
>
> 1) Commit the intent to deprecate but don't signal it programmatically.
>

Python's warnings package has a PendingDeprecationWarning for this --
https://docs.python.org/2.6/library/warnings.html#warning-categories

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][Keystone] [Nova] How to validate teanant-id for admin operation

[Brant Knudson]

On Fri, Apr 24, 2015 at 11:53 AM, Eichberger, German <
german.eichber...@hp.com> wrote:

> All,
>
> Following up from the last Neutron meeting:
>
> If Neutron is performing an operation as an admin on behalf of a user that
> user's tenant-id (or project-id) isn't validated - in particular an admin
> can mistype and create object on behalf of non existent users. I am
> wondering how other projects (e.g. Nova) deal with that and if there is
> some API support in keystone to save us a round trip (e.g. authenticate
> admin + validate additional user-id).
>
>
Not to long ago we got support in the auth_token middleware for a "service"
token in addition to the user's token. The user token is sent in the
x-auth-token header and the service token is sent in the x-service-token,
and then fields from both tokens are available to the application (e.g.,
the user project is in HTTP_X_PROJECT_ID and the service token roles are in
HTTP_X_SERVICE_ROLES). So you could potentially have a policy rule on the
server for the operation that required the service token to have the
'service' role, and what neutron could do is send the original user token
in x-auth-token and send its own token as the service token. This seems to
be what you're asking for here.

- Brant



> Thanks,
> German
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please stop reviewing code while asking questions

[Brant Knudson]

On Fri, Apr 24, 2015 at 10:06 AM, Salvatore Orlando 
wrote:

>
>
> On 24 April 2015 at 16:50, Chris Friesen 
> wrote:
>
>> On 04/24/2015 07:26 AM, Salvatore Orlando wrote:
>>
>>  If you think it might be beneficial to adjust tooling to that these
>>> "contributions" get counted this is fine by me. I just wanted to point
>>> out that
>>> I do not consider those contributions at all (and btw it would be at
>>> least more
>>> polite to put a +1 rather than a -1).
>>>
>>
>> If you're asking a question to elicit information, then it's quite
>> possible you don't have enough information for a +1 yet.
>
>
> This makes sense in general. I was referring to the specific cases posed
> by Julien - curiosities, pedantry, or questions unrelated to the scope of
> the patch.
> Julien clarified that there actually questions which grant a -1, and
> surely never a +1. For instance the kind of "what if" questions listed by
> Doug. In this case it make sense for a reviewer to put a hold a patch while
> waiting for an answer.
>
>
Would anybody be willing to codify this into a document that we can point
offenders to so that we can get better review quality over time? Maybe
http://docs.openstack.org/infra/manual/developers.html#peer-review (was
https://wiki.openstack.org/wiki/ReviewChecklist ).

(Just don't be surprised when some joker posts a question with a -1 on the
review.)

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Please stop reviewing code while asking questions

[Brant Knudson]

On Fri, Apr 24, 2015 at 3:14 AM, Julien Danjou  wrote:

> Hi there,
>
> This is now happening weekly to me now, probably because I write too
> many patches touching almost all OpenStack projects once a cycle, and
> I'm really tired of that behavior, so PLEASE:
>
>   *Stop sending Code-Review-1 when asking a question in a patch*
>
> _Sometimes_ there are good reasons to set -1 even when asking a
> question. For example, when the question is a hint sent to the patch
> author so that (s)he improves is commit message, a code comment or a
> piece of code.
>
> But most of the time, if you ask a question because there's something
> YOU DO NOT KNOW OR UNDERSTAND, do not put a score to a patchset. You
> don't know the answer, so you have absolutely no right to evaluate a
> patchset with -1. Just don't set a score, it's OK, and wait for the
> answer before deciding if the patch is worth [-1..+2].
>
>
If other developers can't understand your code then the code should be
changed to be clearer. There's no reason for OpenStack code to be so
complex that a developer can't understand it. Having code that developers
don't understand leads to bugs and sometimes security vulnerabilties that
affect our users.

There's also be a corollary to this -- don't +1 (and especially +2) code
that you don't understand. In the past I've ignored changes that I wasn't
going to understand, typically this has been because it's implementing an
external standard that I'm not familiar with and don't want to spend hours
reading the spec. These changes tend to sit around unreviewed for a long
time, so it would help the submitter to make the code and reasoning
clearer. So there's a flip side to this -- if reviewers are going to have
trouble understanding the change then expect it to take a long time to
merge, if it ever does.

Maybe it's worth it to have a job that removes -1s after some time, so the
reviewer will know to go back and check on it.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

[Brant Knudson]

On Thu, Apr 23, 2015 at 4:41 AM, Thomas Goirand  wrote:

>
>
> On 04/10/2015 09:41 AM, Thierry Carrez wrote:
>
>> Victor Stinner wrote:
>>
>>> I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(),
>>> threading.RLock, threading.Thread). Good news: the just released eventlet
>>> 0.17.3 includes these fixes and it is now fully compatible with Python 3!
>>> For example, the Oslo Messaging test suite now pass with this eventlet
>>> version! Currently, eventlet is disabled in Oslo Messaging on Python 3
>>> (eventlet tests are skipped).
>>>
>>
>> Great news ! That makes the "port to Python 3" question independent of
>> the "Moving off eventlet" question, which should facilitate immediate
>> progress on the former.
>>
>> On the latter, do you plan to file a Concurrency models cross-project
>> session ? That sounds like a good topic to discuss face to face...
>>
>> See
>> http://lists.openstack.org/pipermail/openstack-dev/2015-April/061070.html
>> for
>> details on how to file there.
>>
>>
> Also, on the Python 3 topic, there's still a big issue with memcached
> (aka: python-memcache). It's blocking me from adding Python3 support to
> keystoneclient, and as a consequence, to almost all of OpenStack.
>
> BTW, the Eventlet module for 0.17.3 is available from here:
>
> http://kilo-jessie.pkgs.mirantis.com/debian/pool/jessie-kilo-backports-nochange/main/p/python-eventlet/
>
> and I will upload this to Experimental as soon as Jessie is released
> (that's in 3 days now...).
>
> Cheers,
>
> Thomas Goirand (zigo)
>
>

The part of keystoneclient that uses the memcached client was deprecated in
Juno (as it was moved to the keystonemiddleware repo), so I think we can
remove it now. You might want to patch it out of your keystoneclient
package if you know everything's using the auth_token middleware from
keystonemiddleware.

- Brant



>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] keystonemiddleware stable/juno broken

[Brant Knudson]

On Tue, Apr 21, 2015 at 12:10 PM, gordon chung  wrote:

> __
> > Date: Tue, 21 Apr 2015 11:12:12 -0500
> > From: b...@acm.org
> > To: openstack-dev@lists.openstack.org
> > Subject: Re: [openstack-dev] [keystone] keystonemiddleware stable/juno
> broken
> >
> >
> >
> > On Sun, Apr 19, 2015 at 2:21 PM, Brant Knudson
> > mailto:b...@acm.org>> wrote:
> >
> > Ever since the stable/juno branch was created for keystonemiddleware
> > it's been broken and there are several changes piling up there. The fix
> > for bug 1408838 "Tests fail with new oslo.utils 1.2.1" allows the tests
> > to pass. I cherry-picked the fix here[1].
> >
> > Based on the title of the bug, you'd think that capping
> > oslo.utils<1.2.1 would be an alternative fix, but I tried installing
> > older versions of oslo.utils and eventually it failed with a completely
> > different error, so I think that's a dead end.
> >
> > [1] https://review.openstack.org/#/c/175232/
> >
> >
> > So this fix has merged, but keystonemiddleware has another problem --
> > the proposal bot posted a change to get the requirements up to date and
> > that's failing all the -python* jobs. Looks like what's happening is
> > that during the tox venv installation the oslo.messaging package is
> > being updated to version 1.9.0, which is way newer than the other oslo
> > packages and this is causing a test that uses pkg_resource to fail with
> > a dependency issue (oslo.messaging 1.9.0 requires
> > oslo.utils<1.5.0,>1.4.0 but global requirements has oslo.utils<1.2.0,
> > so that's what's installed). This all appears to be related to other
> > issues we've seen where the dep resolver isn't working as expected.
> >
> > I tried using an older pycadf (which, from the tox logs is where
> > oslo.messaging is being upgraded) but it was only with a very old
> > version of pycadf that things worked.
>
> does it make sense to make a stable/juno branch for pycadf and remove the
> oslo.messaging requirement? it actually isn't a mandatory requirement and
> was moved to test-requirements in 0.7.0
>
>
Gordon -

I think that would also work... if I have some time I'll take a look at it,
or maybe someone else will get to it first.

:: Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] keystonemiddleware stable/juno broken

[Brant Knudson]

On Sun, Apr 19, 2015 at 2:21 PM, Brant Knudson  wrote:

>
> Ever since the stable/juno branch was created for keystonemiddleware it's
> been broken and there are several changes piling up there. The fix for bug
> 1408838 "Tests fail with new oslo.utils 1.2.1" allows the tests to pass. I
> cherry-picked the fix here[1].
>
> Based on the title of the bug, you'd think that capping oslo.utils<1.2.1
> would be an alternative fix, but I tried installing older versions of
> oslo.utils and eventually it failed with a completely different error, so I
> think that's a dead end.
>
> [1] https://review.openstack.org/#/c/175232/
>
>
So this fix has merged, but keystonemiddleware has another problem -- the
proposal bot posted a change to get the requirements up to date and that's
failing all the -python* jobs. Looks like what's happening is that during
the tox venv installation the oslo.messaging package is being updated to
version 1.9.0, which is way newer than the other oslo packages and this is
causing a test that uses pkg_resource to fail with a dependency issue
(oslo.messaging 1.9.0 requires oslo.utils<1.5.0,>1.4.0 but global
requirements has oslo.utils<1.2.0, so that's what's installed). This all
appears to be related to other issues we've seen where the dep resolver
isn't working as expected.

I tried using an older pycadf (which, from the tox logs is where
oslo.messaging is being upgraded) but it was only with a very old version
of pycadf that things worked.

So the current proposed fix is to move oslo.messaging from
test-requirements.txt to requirements.txt. The correct oslo.messaging gets
installed when both pycadf and oslo.messaging are in the same file.
Hopefully this is ok and can be done in a fix release... oslo.messaging was
already a requirement for pycadf so this shouldn't cause an issue for
packagers.[1]

[1] https://review.openstack.org/#/c/175688/

If this sounds ok I'll squash the change into the parent review so that it
can get merged.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] keystoneclient stable/icehouse broken

[Brant Knudson]

On Sun, Apr 19, 2015 at 3:07 PM, Brant Knudson  wrote:

>
> Ever since the stable/icehouse branch was created for
> python-keystoneclient it's been broken. There are a couple of issues:
>
> 1) The gate-python-keystoneclient-python34 job is failing with the error
> "ERROR: InterpreterNotFound: python3.4". I don't know why it's not
> available. Maybe python3 needs to be installed on the instance or maybe the
> job should not be run for stable/icehouse. No fix has been proposed for
> this as far as I know. I'll need some help from infra to figure it out.
>
>
For this one, the job was just removed via an infra change, see
https://review.openstack.org/#/c/175235/ .

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [keystone] keystoneclient stable/icehouse broken

[Brant Knudson]

Ever since the stable/icehouse branch was created for python-keystoneclient
it's been broken. There are a couple of issues:

1) The gate-python-keystoneclient-python34 job is failing with the error
"ERROR: InterpreterNotFound: python3.4". I don't know why it's not
available. Maybe python3 needs to be installed on the instance or maybe the
job should not be run for stable/icehouse. No fix has been proposed for
this as far as I know. I'll need some help from infra to figure it out.

2) The gate-tempest-dsvm-neutron-src-python-keystoneclient job is failing
since swift-proxy isn't starting due to a requirements mismatch. This is
the same issue we had in the stable/juno branch and was fixed by raising
the cap. I proposed the change to global-requirements[1], and then that
should eventually get to keystoneclient's requirements.txt. (This is the
same issue that was plaguing stable/juno.)

[1] https://review.openstack.org/#/c/175234/

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [keystone] keystonemiddleware stable/juno broken

[Brant Knudson]

Ever since the stable/juno branch was created for keystonemiddleware it's
been broken and there are several changes piling up there. The fix for bug
1408838 "Tests fail with new oslo.utils 1.2.1" allows the tests to pass. I
cherry-picked the fix here[1].

Based on the title of the bug, you'd think that capping oslo.utils<1.2.1
would be an alternative fix, but I tried installing older versions of
oslo.utils and eventually it failed with a completely different error, so I
think that's a dead end.

[1] https://review.openstack.org/#/c/175232/

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Problems with keystoneclient stable branch (and maybe yours too)

[Brant Knudson]

On Mon, Apr 13, 2015 at 9:33 PM, gordon chung  wrote:

> >
> >> 2) Incorrect cap in requirements.txt
> >>
> >> python-keystoneclient in stable/juno was capped at <=1.1.0, and 1.1.0 is
> >> the version tagged for the stable branch. When you create a review in
> >> stable/juno it installs python-keystoneclient and now the system has
> got a
> >> version like 1.1.0.post1, which is>1.1.0, so now python-keystoneclient
> >> doesn't match the requirements and swift-proxy fails to start
> (swift-proxy
> >> is very good at catching this problem for whatever reason). The cap
> should
> >> have been <1.2.0 so that we can propose patches and also make fix
> releases
> >> (1.1.1, 1.1.2, etc.).[3]
> >>
> >> [3] https://review.openstack.org/#/c/172718/
> >
> > Approved.
>
> we have the same issue for ceilometerclient for both icehouse[1] and
> juno[2], i put up requirement patches for each [3][4]
>
> [1] https://review.openstack.org/#/c/173085/
> [2] https://review.openstack.org/#/c/173086/
> [3] https://review.openstack.org/#/c/173149/
> [4] https://review.openstack.org/#/c/173148/
>
>
Those caps were originally <=1.0.12 and the new cap is <1.0.13, but these
caps are equivalent since the next fix release after 1.0.12 is 1.0.13
(unless you can use a version with 1.0.12.1?). So I'm not sure how this is
going to work.


> >
> >>
> >> I tried to recap all of the clients but that didn't pass Jenkins,
> probably
> >> because one or more clients didn't use semver correctly and have
> >> requirements updates in a micro release.[4]
> >>
> >> [4] https://review.openstack.org/#/c/172719/
> >
> > Did you literally update them all, or only the ones that looked like
> > they might be wrong? It looks like those caps came from the cap.py
> > script in the repository, which makes me wonder if we were just too
> > aggressive with defining what the cap should be.
> >
>
>
I updated all the python-*clients to allow for fixes, e.g., from
<=[major].[minor].[fix] to <[major].[minor+1].0. When this didn't work I
thought about checking all the client repos to see which ones have problems
but that was more work than I was willing to do.

don't know about others but full disclosure, we didn't use SEMVER
> correctly. :\
>
>
Maybe projects that didn't use semver correctly can't have stable branches
for now. I'm not sure what all the drawbacks are... maybe just that you
don't get to backport security fixes.

- Brant


> cheers,
> gord
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] Problems with keystoneclient stable branch (and maybe yours too)

[Brant Knudson]

There were several problems with the keystoneclient stable/juno branch that
have been or are in the process of being fixed since its creation.
Hopefully this note will be useful to other projects that create stable
branches for their libraries.


1) Unit tests didn't pass with earlier packages

The supported versions of several of the packages in requirements.txt in
the stable branch are in the process of being capped[0], so that the tests
are now running with older versions of the packages. Since we don't
normally test with the older packages we didn't know that the
keystoneclient unit tests don't actually pass with the old version of the
package. This is fixed by correcting the tests to work with the older
versions of the packages.[1][2]

[0] https://review.openstack.org/#/c/172220/
[1] https://review.openstack.org/#/c/172655/
[2] https://review.openstack.org/#/c/172256/

It would be great if we were testing with the minimum versions of the
packages that we say we support somehow since that would have caught this.


2) Incorrect cap in requirements.txt

python-keystoneclient in stable/juno was capped at <=1.1.0, and 1.1.0 is
the version tagged for the stable branch. When you create a review in
stable/juno it installs python-keystoneclient and now the system has got a
version like 1.1.0.post1, which is >1.1.0, so now python-keystoneclient
doesn't match the requirements and swift-proxy fails to start (swift-proxy
is very good at catching this problem for whatever reason). The cap should
have been <1.2.0 so that we can propose patches and also make fix releases
(1.1.1, 1.1.2, etc.).[3]

[3] https://review.openstack.org/#/c/172718/

I tried to recap all of the clients but that didn't pass Jenkins, probably
because one or more clients didn't use semver correctly and have
requirements updates in a micro release.[4]

[4] https://review.openstack.org/#/c/172719/


3) Unsupported functional tests

We added support for functional tests (tox -e functional) in K, but Jenkins
was configured to run the functional job on all branches and it fails when
the tox target doesn't exist. The fix was to exclude the current stable/
branches for keystoneclient.[5]

[5] https://review.openstack.org/#/c/172658/


4) Tempest -juno job?

For some reason keystoneclient has 2 tempest-neutron jobs:
gate-tempest-dsvm-neutron-src-python-keystoneclient and
...-keystoneclient-juno , and the -juno job is failing in stable/juno. It
didn't make sense to me that we needed to run both in python-keystoneclient
stable/juno. I was told that we didn't need the -juno one anymore on any
branch since we have a stable/juno branch, so that job is removed.[6]

[6] https://review.openstack.org/#/c/172662/


Hopefully with these changes the python-keystoneclient stable/juno branch
will be working.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][oslo][clients] Let's speed up start of OpenStack libs and clients by optimizing imports with profimp

[Brant Knudson]

On Wed, Apr 8, 2015 at 7:59 AM, Doug Hellmann  wrote:

> Excerpts from Robert Collins's message of 2015-04-07 10:43:30 +1200:
> > On 7 April 2015 at 05:11, Joe Gordon  wrote:
> > >
> > >
> > > On Mon, Apr 6, 2015 at 8:39 AM, Dolph Mathews  >
> > > wrote:
> > >>
> > >>
> > >> On Mon, Apr 6, 2015 at 10:26 AM, Boris Pavlovic 
> wrote:
> > >>>
> > >>> Jay,
> > >>>
> > >>>
> >  Not far, IMHO. 100ms difference in startup time isn't something we
> >  should spend much time optimizing. There's bigger fish to fry.
> > >>>
> > >>>
> > >>> I agree that priority of this task shouldn't be critical or even
> high,
> > >>> and that there are other places that can be improved in OpenStack.
> > >>>
> > >>> In other hand this one is as well big source of UX issues that we
> have in
> > >>> OpenStack..
> > >>>
> > >>> For example:
> > >>>
> > >>> 1) You would like to run some command X times where X is pretty big
> > >>> (admins likes to do this via bash loops). If you can execute all of
> them for
> > >>> 1 and not 10 minutes you will get happier end user.
> > >>
> > >>
> > >> +1 I'm fully in support of this effort. Shaving 100ms off the startup
> time
> > >> of a frequently used library means that you'll save that 100ms over
> and
> > >> over, adding up to a huge win.
> > >>
> > >
> > >
> > > Another data point on how slow our libraries/CLIs can be:
> > >
> > > $ time openstack -h
> > > 
> > > real0m2.491s
> > > user0m2.378s
> > > sys 0m0.111s
> >
> >
> > pbr should be snappy - taking 100ms to get the version is wrong.
>
> I have always considered pbr a packaging/installation time tool, and not
> something that would be used at runtime. Why are we using pbr to get the
> version of an installed package, instead of asking pkg_resources?
>
> Doug
>
> >
> > -Rob
> >
>
>
I have no idea if one is better than the other, but I figured It was easy
enough to post the change for keystoneclient:
https://review.openstack.org/#/c/171720/ .

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [docs] [End User Guide] [Keystone] keyring support for python-keystoneclient and python-openstackclient

[Brant Knudson]

On Tue, Apr 7, 2015 at 3:52 AM, Olena Logvinova 
wrote:

> Good day to everyone!
>
> My name is Olena, I am new in OpenStack (working as a tech writer). And
> I'm stuck on a bug https://launchpad.net/bugs/1419990 (patch
> https://review.openstack.org/#/c/163503/).
>
> I have 2 questions, please:
>
> Does the page http://docs.openstack.org/user-guide/content/cli_openrc.html 
> contain
> info about python-keystoneclient only, or both python-keystoneclient and
> python-openstackclient?
>

The page contains info about all the CLIs, not just keystone and openstack.
Users should be able to create an OpenStack RC file that works with the
keystone command or the openstack command or nova, glance, etc. Note that
the keystone command is deprecated in favor of the openstack command, so I
think we should be removing any use of it from the documentation.

And should we remove the keyring support part here (because it was removed
> from python-openstackclient), or do some re-wording (because it is still
> left in python-keystoneclient)?
> (Here is a patch saying that keyring is still used by keystoneclient:
> https://review.openstack.org/#/c/107719/ )
>
>
I tried the keystone command and it doesn't have an --os-use-keyring
option, and neither does the openstack command. I think the "Keyring
support" section should be removed.

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][oslo][clients] Let's speed up start of OpenStack libs and clients by optimizing imports with profimp

[Brant Knudson]

On Sun, Apr 5, 2015 at 5:41 PM, Boris Pavlovic  wrote:

> Brant,
>
> I run profimp with and without patch
> https://review.openstack.org/#/c/164066/:
> And it really works well:
>
> before 170ms:
> http://boris-42.github.io/keystone/before.html
>
> after 76ms:
> http://boris-42.github.io/keystone/after.html
>
>
>
Looks like now the issue is that keystoneclient imports pbr.version so that
it can get pbr to generate the __version__ value. Maybe pbr could be more
efficient about the work it does on import, or we could figure out a
different / lazy way to generate the version string.
http://git.openstack.org/cgit/openstack/python-keystoneclient/tree/keystoneclient/__init__.py#n35

How far do we need to go with this?

- Brant



> Best regards,
> Boris Pavlovic
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][oslo][clients] Let's speed up start of OpenStack libs and clients by optimizing imports with profimp

[Brant Knudson]

On Thu, Apr 2, 2015 at 4:52 PM, Boris Pavlovic  wrote:

> Hi stackers,
>
> Recently, I started working on speeding up Rally cli.
>
> What I understand immediately  is that I don't understand why it takes
> 700-800ms
> to just run "rally version" command and it is impossible hard task to find
> what takes so much time just by reading the code.
>
> I started playing with patching __import__ and make a simple but powerful
> tool
> that allows you to trace any imports and get pretty graphs of nested
> importing with timings:
>
> https://github.com/boris-42/profimp
>
> So now it's simple to understand what imports take most of time, just by
> running:
>
>   profimp "import " --html
>
>
> Let's optimize OpenStack libs and clients?
>
>
> Best regards,
> Boris Pavlovic
>
>
There's a review in python-keystoneclient to do lazy importing of modules
here: https://review.openstack.org/#/c/164066/ . It would be interesting to
know if this improves the initial import time significantly. Also, this can
be an example of how to improve other libraries.

 - Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev