Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Aliasgar Ginwala]

So in any case i am using 20 processes and 20 threads to have optimal
utilization. I did try with 50 50 values too for apache and the utilization
spikes a bit high but its fine for load test. Utilization do touch around
40% for 20 20 values of processes and threads.I am running it on BM with
128gb memory.I am spinning 100 concurrent users. Rps is bit killing as
create token does sign tokeb operation too which consumes some time.

Regards,
Ali

On Wednesday, December 9, 2015, Dolph Mathews 
wrote:

>
>
> On Wednesday, December 9, 2015, Ginwala, Aliasgar  > wrote:
>
>> Hi Dolph/team:
>>
>> As requested I have outlined most of the files and configs to give more
>> clear picture @ https://gist.github.com/noah8713/7d5554d78b60cd9a4999:
>>
>
> The number of threads per API your config files show range from 20 to
> 10,000 (processes * threads). On one end, your server might be idling
> during the benchmark. On the other end, you're probably exhausting the
> server's memory during the benchmark run. So, absolutely nothing about this
> benchmark appears to be fair.
>
> What is the maximum number of concurrent users you're spinning up to in
> locust? (not the spawn rate)
>
>
>>
>> * keystone.conf   —uploaded
>> * distro used for testing, in case they override the project's defaults
>>  —mentioned
>> * all nginx config files —uploaded nginx-keystone.conf
>> * all uwsgi config files —keystone-main.ini, keystone-admin.ini and
>> upstart file
>> * apache config, including virtual hosts and mods —apache-keystone.conf
>> and python files (common for nginx)
>> * your test client and it's configuration —mentioned
>> * server & client architecture, and at least some idea of what lies in
>> between (networking, etc)  —briefly outlined
>> * whatever else I'm forgetting —feel free to add in the comments
>>
>>
>>
>> Regards,
>> Ali
>>
>> From: Dolph Mathews 
>> Reply-To: "OpenStack Development Mailing List (not for usage questions)"
>> 
>> Date: Wednesday, December 9, 2015 at 5:42 AM
>> To: "OpenStack Development Mailing List (not for usage questions)" <
>> openstack-dev@lists.openstack.org>
>> Subject: Re: [openstack-dev] [Openstack-operators] [keystone] Removing
>> functionality that was deprecated in Kilo and upcoming deprecated
>> functionality in Mitaka
>>
>> Benchmarks always appreciated!
>>
>> But, these types of benchmarks are *entirely* useless unless you can
>> provide the exact configuration you used for each scenario so that others
>> can scrutinize the test method and reproduce your results. So, off the top
>> of my head, I'm looking for:
>>
>> * keystone.conf
>> * distro used for testing, in case they override the project's defaults
>> * all nginx config files
>> * all uwsgi config files
>> * apache config, including virtual hosts and mods
>> * your test client and it's configuration
>> * server & client architecture, and at least some idea of what lies in
>> between (networking, etc)
>> * whatever else I'm forgetting
>>
>> A mailing list is probably not the best method to provide anything other
>> than a summary; so I'd suggest publishing the details in a gist, blog post,
>> or both.
>>
>> And to comment on the results themselves: you shouldn't be seeing that
>> big of a performance gap between httpd and everything else; something is
>> fundamentally different about that configuration. These are just web
>> servers, after all. Choosing between them should not be a matter of
>> performance, but it should be a choice of documentation, licensing,
>> community, operability, supportability, reliability, etc. Performance
>> should be relatively similar, and thus a much lower priority in making your
>> selection.
>>
>> On Tue, Dec 8, 2015 at 10:09 PM, Ginwala, Aliasgar 
>> wrote:
>>
>>>
>>>
>>> Hi All:
>>>
>>> Just to inform Steve and all the folks who brought up this talk ;We did
>>> some benchmarking using wsgi, apache and nginx for keystone with mysql as
>>> token backend and we got following results on Juno version. Hence I am just
>>> giving you brief highlight about the results we got.
>>>
>>> spawning 100 users per sec for create token, below are the results:
>>>
>>> *Using nginx with uwsgi: *
>>> rps *32*—(reqests/sec)
>>> median time ~ 3.3 sec
>>> no of processes 20
>>>
>>> *using apache *
>>> rp

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Ginwala, Aliasgar]

Hi Dolph/team:

As requested I have outlined most of the files and configs to give more clear 
picture @ https://gist.github.com/noah8713/7d5554d78b60cd9a4999:

* keystone.conf   —uploaded
* distro used for testing, in case they override the project's defaults  
—mentioned
* all nginx config files —uploaded nginx-keystone.conf
* all uwsgi config files —keystone-main.ini, keystone-admin.ini and upstart file
* apache config, including virtual hosts and mods —apache-keystone.conf and 
python files (common for nginx)
* your test client and it's configuration —mentioned
* server & client architecture, and at least some idea of what lies in between 
(networking, etc)  —briefly outlined
* whatever else I'm forgetting —feel free to add in the comments



Regards,
Ali

From: Dolph Mathews mailto:dolph.math...@gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Date: Wednesday, December 9, 2015 at 5:42 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Openstack-operators] [keystone] Removing 
functionality that was deprecated in Kilo and upcoming deprecated functionality 
in Mitaka

Benchmarks always appreciated!

But, these types of benchmarks are *entirely* useless unless you can provide 
the exact configuration you used for each scenario so that others can 
scrutinize the test method and reproduce your results. So, off the top of my 
head, I'm looking for:

* keystone.conf
* distro used for testing, in case they override the project's defaults
* all nginx config files
* all uwsgi config files
* apache config, including virtual hosts and mods
* your test client and it's configuration
* server & client architecture, and at least some idea of what lies in between 
(networking, etc)
* whatever else I'm forgetting

A mailing list is probably not the best method to provide anything other than a 
summary; so I'd suggest publishing the details in a gist, blog post, or both.

And to comment on the results themselves: you shouldn't be seeing that big of a 
performance gap between httpd and everything else; something is fundamentally 
different about that configuration. These are just web servers, after all. 
Choosing between them should not be a matter of performance, but it should be a 
choice of documentation, licensing, community, operability, supportability, 
reliability, etc. Performance should be relatively similar, and thus a much 
lower priority in making your selection.

On Tue, Dec 8, 2015 at 10:09 PM, Ginwala, Aliasgar 
mailto:aginw...@ebay.com>> wrote:


Hi All:

Just to inform Steve and all the folks who brought up this talk ;We did some 
benchmarking using wsgi, apache and nginx for keystone with mysql as token 
backend and we got following results on Juno version. Hence I am just giving 
you brief highlight about the results we got.

spawning 100 users per sec for create token, below are the results:

Using nginx with uwsgi:
rps 32—(reqests/sec)
median time ~ 3.3 sec
no of processes 20

using apache
rps 75
median time ~ 1.3 sec
avg time - 1.5 sec
no of processes 20

using wsgi
rps 28
median time ~ 3.4
avg 3.5
no of processes 20


We are planning to switch to apache since we are not seeing good results using 
nginx with uwsgi. May be some more added support is required for nginx 
performance.We did not encounter this auto restart issue in test yet and hence 
are open to more inputs.

Any other suggestions are welcome too. Let us know in case of queries.

Regards,
Aliasgar

On 8 December 2015 at 07:53, Thomas Goirand 
mailto:z...@debian.org>> wrote:
On 12/01/2015 07:57 AM, Steve Martinelli wrote:
> Trying to summarize here...
>
> - There isn't much interest in keeping eventlet around.
> - Folks are OK with running keystone in a WSGI server, but feel they are
> constrained by Apache.
> - uWSGI could help to support multiple web servers.
>
> My opinion:
>
> - Adding support for uWSGI definitely sounds like it's worth
> investigating, but not achievable in this release (unless someone
> already has something cooked up).
> - I'm tempted to let eventlet stick around another release, since it's
> causing pain on some of our operators.
> - Other folks have managed to run keystone in a web server (and
> hopefully not feel pain when doing so!), so it might be worth getting
> technical details on just how it was accomplished. If we get an OK from
> the operator community later on in mitaka, I'd still be OK with removing
> eventlet, but I don't want to break folks.
>
> stevemar
>
> From: John Dewey mailto:j...@dewey.ws>>
> 100% agree.
>
> We should look at uwsgi as the reference architecture. Nginx/Apache/etc
> should be intercha

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Thomas Goirand]

On 12/08/2015 06:39 AM, Jamie Lennox wrote:
> The main problem I see with running Keystone (or any other service) in a
> web server, is that *I* (as a package maintainer) will loose the control
> over when the service is started. Let me explain why that is important
> for me.
> 
> In Debian, many services/daemons are run, then their API is used by the
> package. In the case of Keystone, for example, it is possible to ask,
> via Debconf, that Keystone registers itself in the service catalog. If
> we get Keystone within Apache, it becomes at least harder to do so.
> 
> I was going to leave this up to others to comment on here, but IMO -
> excellent. Anyone that is doing an even semi serious deployment of
> OpenStack is going to require puppet/chef/ansible or some form of
> orchestration layer for deployment. Even for test deployments it seems
> to me that it's crazy for this sort of functionality be handled from
> debconf. The deployers of the system are going to understand if they
> want to use eventlet or apache and should therefore understand what
> restarting apache on a system implies.

It is often what everyone from within the community says. However,
there's lots of users who hardly do a single deployment, maybe 2. I
don't agree that they should all invest a huge amount of time in some
automation tools, and for them, packages should be enough.

Anyway, the debconf handling is completely optional, and most of the
helpers are completely disabled by default. So it is *not* in the way of
using any deployment tool like puppet.

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Ginwala, Aliasgar]

Sounds good Dolph. Will try to post  the details as requested on gist/ blog 
shortly.

Regards,
Ali

From: Dolph Mathews mailto:dolph.math...@gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Date: Wednesday, December 9, 2015 at 5:42 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Openstack-operators] [keystone] Removing 
functionality that was deprecated in Kilo and upcoming deprecated functionality 
in Mitaka

Benchmarks always appreciated!

But, these types of benchmarks are *entirely* useless unless you can provide 
the exact configuration you used for each scenario so that others can 
scrutinize the test method and reproduce your results. So, off the top of my 
head, I'm looking for:

* keystone.conf
* distro used for testing, in case they override the project's defaults
* all nginx config files
* all uwsgi config files
* apache config, including virtual hosts and mods
* your test client and it's configuration
* server & client architecture, and at least some idea of what lies in between 
(networking, etc)
* whatever else I'm forgetting

A mailing list is probably not the best method to provide anything other than a 
summary; so I'd suggest publishing the details in a gist, blog post, or both.

And to comment on the results themselves: you shouldn't be seeing that big of a 
performance gap between httpd and everything else; something is fundamentally 
different about that configuration. These are just web servers, after all. 
Choosing between them should not be a matter of performance, but it should be a 
choice of documentation, licensing, community, operability, supportability, 
reliability, etc. Performance should be relatively similar, and thus a much 
lower priority in making your selection.

On Tue, Dec 8, 2015 at 10:09 PM, Ginwala, Aliasgar 
mailto:aginw...@ebay.com>> wrote:


Hi All:

Just to inform Steve and all the folks who brought up this talk ;We did some 
benchmarking using wsgi, apache and nginx for keystone with mysql as token 
backend and we got following results on Juno version. Hence I am just giving 
you brief highlight about the results we got.

spawning 100 users per sec for create token, below are the results:

Using nginx with uwsgi:
rps 32—(reqests/sec)
median time ~ 3.3 sec
no of processes 20

using apache
rps 75
median time ~ 1.3 sec
avg time - 1.5 sec
no of processes 20

using wsgi
rps 28
median time ~ 3.4
avg 3.5
no of processes 20


We are planning to switch to apache since we are not seeing good results using 
nginx with uwsgi. May be some more added support is required for nginx 
performance.We did not encounter this auto restart issue in test yet and hence 
are open to more inputs.

Any other suggestions are welcome too. Let us know in case of queries.

Regards,
Aliasgar

On 8 December 2015 at 07:53, Thomas Goirand 
mailto:z...@debian.org>> wrote:
On 12/01/2015 07:57 AM, Steve Martinelli wrote:
> Trying to summarize here...
>
> - There isn't much interest in keeping eventlet around.
> - Folks are OK with running keystone in a WSGI server, but feel they are
> constrained by Apache.
> - uWSGI could help to support multiple web servers.
>
> My opinion:
>
> - Adding support for uWSGI definitely sounds like it's worth
> investigating, but not achievable in this release (unless someone
> already has something cooked up).
> - I'm tempted to let eventlet stick around another release, since it's
> causing pain on some of our operators.
> - Other folks have managed to run keystone in a web server (and
> hopefully not feel pain when doing so!), so it might be worth getting
> technical details on just how it was accomplished. If we get an OK from
> the operator community later on in mitaka, I'd still be OK with removing
> eventlet, but I don't want to break folks.
>
> stevemar
>
> From: John Dewey mailto:j...@dewey.ws>>
> 100% agree.
>
> We should look at uwsgi as the reference architecture. Nginx/Apache/etc
> should be interchangeable, and up to the operator which they choose to
> use. Hell, with tcp load balancing now in opensource Nginx, I could get
> rid of Apache and HAProxy by utilizing uwsgi.
>
> John

The main problem I see with running Keystone (or any other service) in a
web server, is that *I* (as a package maintainer) will loose the control
over when the service is started. Let me explain why that is important
for me.

In Debian, many services/daemons are run, then their API is used by the
package. In the case of Keystone, for example, it is possible to ask,
via Debconf, that Keystone registers itself in the service catalog. If
we get Keystone within Apache, it becomes at least harde

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Dolph Mathews]

Benchmarks always appreciated!

But, these types of benchmarks are *entirely* useless unless you can
provide the exact configuration you used for each scenario so that others
can scrutinize the test method and reproduce your results. So, off the top
of my head, I'm looking for:

* keystone.conf
* distro used for testing, in case they override the project's defaults
* all nginx config files
* all uwsgi config files
* apache config, including virtual hosts and mods
* your test client and it's configuration
* server & client architecture, and at least some idea of what lies in
between (networking, etc)
* whatever else I'm forgetting

A mailing list is probably not the best method to provide anything other
than a summary; so I'd suggest publishing the details in a gist, blog post,
or both.

And to comment on the results themselves: you shouldn't be seeing that big
of a performance gap between httpd and everything else; something is
fundamentally different about that configuration. These are just web
servers, after all. Choosing between them should not be a matter of
performance, but it should be a choice of documentation, licensing,
community, operability, supportability, reliability, etc. Performance
should be relatively similar, and thus a much lower priority in making your
selection.

On Tue, Dec 8, 2015 at 10:09 PM, Ginwala, Aliasgar 
wrote:

>
>
> Hi All:
>
> Just to inform Steve and all the folks who brought up this talk ;We did
> some benchmarking using wsgi, apache and nginx for keystone with mysql as
> token backend and we got following results on Juno version. Hence I am just
> giving you brief highlight about the results we got.
>
> spawning 100 users per sec for create token, below are the results:
>
> *Using nginx with uwsgi: *
> rps *32*—(reqests/sec)
> median time ~ 3.3 sec
> no of processes 20
>
> *using apache *
> rps *75*
> median time ~ 1.3 sec
> avg time - 1.5 sec
> no of processes 20
>
> *using wsgi *
> rps *28*
> median time ~ 3.4
> avg 3.5
> no of processes 20
>
>
> We are planning to switch to apache since we are not seeing good results
> using nginx with uwsgi. May be some more added support is required for
> nginx performance.We did not encounter this auto restart issue in test
> yet and hence are open to more inputs.
>
> Any other suggestions are welcome too. Let us know in case of queries.
>
> Regards,
> Aliasgar
>
> On 8 December 2015 at 07:53, Thomas Goirand  wrote:
>
>> On 12/01/2015 07:57 AM, Steve Martinelli wrote:
>> > Trying to summarize here...
>> >
>> > - There isn't much interest in keeping eventlet around.
>> > - Folks are OK with running keystone in a WSGI server, but feel they are
>> > constrained by Apache.
>> > - uWSGI could help to support multiple web servers.
>> >
>> > My opinion:
>> >
>> > - Adding support for uWSGI definitely sounds like it's worth
>> > investigating, but not achievable in this release (unless someone
>> > already has something cooked up).
>> > - I'm tempted to let eventlet stick around another release, since it's
>> > causing pain on some of our operators.
>> > - Other folks have managed to run keystone in a web server (and
>> > hopefully not feel pain when doing so!), so it might be worth getting
>> > technical details on just how it was accomplished. If we get an OK from
>> > the operator community later on in mitaka, I'd still be OK with removing
>> > eventlet, but I don't want to break folks.
>> >
>> > stevemar
>> >
>> > From: John Dewey 
>> > 100% agree.
>> >
>> > We should look at uwsgi as the reference architecture. Nginx/Apache/etc
>> > should be interchangeable, and up to the operator which they choose to
>> > use. Hell, with tcp load balancing now in opensource Nginx, I could get
>> > rid of Apache and HAProxy by utilizing uwsgi.
>> >
>> > John
>>
>> The main problem I see with running Keystone (or any other service) in a
>> web server, is that *I* (as a package maintainer) will loose the control
>> over when the service is started. Let me explain why that is important
>> for me.
>>
>> In Debian, many services/daemons are run, then their API is used by the
>> package. In the case of Keystone, for example, it is possible to ask,
>> via Debconf, that Keystone registers itself in the service catalog. If
>> we get Keystone within Apache, it becomes at least harder to do so.
>>
>
> I was going to leave this up to others to comment on here, but IMO -
> excellent. Anyone that is doing an even semi serious deployment of
> OpenStack is going to require puppet/chef/ansible or some form of
> orchestration layer for deployment. Even for test deployments it seems to
> me that it's crazy for this sort of functionality be handled from debconf.
> The deployers of the system are going to understand if they want to use
> eventlet or apache and should therefore understand what restarting apache
> on a system implies.
>
>
>>
>> The other issue is that if all services are sharing the same web server,
>> restarting the web server restarts all services. 

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Aliasgar Ginwala]

Hi All:


Just to inform Steve and all the folks who brought up this talk ;We did
some benchmarking using wsgi, apache and nginx for keystone with mysql as
token backend and we got following results on Juno version. Hence I am just
giving you brief highlight about the results we got.


spawning 100 users per sec for create token, below are the results:


*Using nginx with uwsgi: *

rps *32*—(reqests/sec)

median time ~ 3.3 sec

no of processes 20


*using apache *

rps *75*

median time ~ 1.3 sec

avg time - 1.5 sec

no of processes 20


*using wsgi *

rps *28*

median time ~ 3.4

avg 3.5

no of processes 20



We are planning to switch to apache since we are not seeing good results
using nginx with uwsgi. May be some more added support is required for
nginx performance.We did not encounter this auto restart issue in test yet
and hence are open to more inputs.


Any other suggestions are welcome too. Let us know in case of queries.


Regards,

Aliasgar

On Mon, Dec 7, 2015 at 9:39 PM, Jamie Lennox  wrote:

>
>
> On 8 December 2015 at 07:53, Thomas Goirand  wrote:
>
>> On 12/01/2015 07:57 AM, Steve Martinelli wrote:
>> > Trying to summarize here...
>> >
>> > - There isn't much interest in keeping eventlet around.
>> > - Folks are OK with running keystone in a WSGI server, but feel they are
>> > constrained by Apache.
>> > - uWSGI could help to support multiple web servers.
>> >
>> > My opinion:
>> >
>> > - Adding support for uWSGI definitely sounds like it's worth
>> > investigating, but not achievable in this release (unless someone
>> > already has something cooked up).
>> > - I'm tempted to let eventlet stick around another release, since it's
>> > causing pain on some of our operators.
>> > - Other folks have managed to run keystone in a web server (and
>> > hopefully not feel pain when doing so!), so it might be worth getting
>> > technical details on just how it was accomplished. If we get an OK from
>> > the operator community later on in mitaka, I'd still be OK with removing
>> > eventlet, but I don't want to break folks.
>> >
>> > stevemar
>> >
>> > From: John Dewey 
>> > 100% agree.
>> >
>> > We should look at uwsgi as the reference architecture. Nginx/Apache/etc
>> > should be interchangeable, and up to the operator which they choose to
>> > use. Hell, with tcp load balancing now in opensource Nginx, I could get
>> > rid of Apache and HAProxy by utilizing uwsgi.
>> >
>> > John
>>
>> The main problem I see with running Keystone (or any other service) in a
>> web server, is that *I* (as a package maintainer) will loose the control
>> over when the service is started. Let me explain why that is important
>> for me.
>>
>> In Debian, many services/daemons are run, then their API is used by the
>> package. In the case of Keystone, for example, it is possible to ask,
>> via Debconf, that Keystone registers itself in the service catalog. If
>> we get Keystone within Apache, it becomes at least harder to do so.
>>
>
> I was going to leave this up to others to comment on here, but IMO -
> excellent. Anyone that is doing an even semi serious deployment of
> OpenStack is going to require puppet/chef/ansible or some form of
> orchestration layer for deployment. Even for test deployments it seems to
> me that it's crazy for this sort of functionality be handled from debconf.
> The deployers of the system are going to understand if they want to use
> eventlet or apache and should therefore understand what restarting apache
> on a system implies.
>
>
>>
>> The other issue is that if all services are sharing the same web server,
>> restarting the web server restarts all services. Or, said otherwise: if
>> I need to change a configuration value of any of the services served by
>> Apache, I will need to restart them all, which is very annoying: I very
>> much prefer to just restart *ONE* service if I need.
>>
>> Also, something which we learned the hard way at Mirantis: it is *very*
>> annoying that Apache restarts every Sunday morning by default in
>> distributions like Ubuntu and Debian (I'm not sure for the other
>> distros). No, the default config of logrotate and Apache can't be
>> changed in distros just to satisfy OpenStack users: there's other users
>> of Apache in these distros.
>>
>
> :O
>
>
>>
>> Then, yes, uWSGI becomes a nice option. I used it for the Barbican
>> package, and it worked well. Though the uwsgi package in Debian isn't
>> very well maintained, and multiple times, Barbican could have been
>> removed from Debian testing because of RC bugs against uWSGI.
>>
>> So, all together, I'm a bit reluctant to see the Eventlet based servers
>> going away. If it's done, then yes, I'll work around it. Though I'd
>> prefer if it didn't.
>>
>> It is also my view that it's up to the deployers to decide how they want
>> to implement things. For many small use cases, Eventlet performs well
>> enough.
>>
>> Finally, one thing which I never understood: if Eventlet is bad as an
>> HTTP server, can't we use a

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Jamie Lennox]

On 8 December 2015 at 07:53, Thomas Goirand  wrote:

> On 12/01/2015 07:57 AM, Steve Martinelli wrote:
> > Trying to summarize here...
> >
> > - There isn't much interest in keeping eventlet around.
> > - Folks are OK with running keystone in a WSGI server, but feel they are
> > constrained by Apache.
> > - uWSGI could help to support multiple web servers.
> >
> > My opinion:
> >
> > - Adding support for uWSGI definitely sounds like it's worth
> > investigating, but not achievable in this release (unless someone
> > already has something cooked up).
> > - I'm tempted to let eventlet stick around another release, since it's
> > causing pain on some of our operators.
> > - Other folks have managed to run keystone in a web server (and
> > hopefully not feel pain when doing so!), so it might be worth getting
> > technical details on just how it was accomplished. If we get an OK from
> > the operator community later on in mitaka, I'd still be OK with removing
> > eventlet, but I don't want to break folks.
> >
> > stevemar
> >
> > From: John Dewey 
> > 100% agree.
> >
> > We should look at uwsgi as the reference architecture. Nginx/Apache/etc
> > should be interchangeable, and up to the operator which they choose to
> > use. Hell, with tcp load balancing now in opensource Nginx, I could get
> > rid of Apache and HAProxy by utilizing uwsgi.
> >
> > John
>
> The main problem I see with running Keystone (or any other service) in a
> web server, is that *I* (as a package maintainer) will loose the control
> over when the service is started. Let me explain why that is important
> for me.
>
> In Debian, many services/daemons are run, then their API is used by the
> package. In the case of Keystone, for example, it is possible to ask,
> via Debconf, that Keystone registers itself in the service catalog. If
> we get Keystone within Apache, it becomes at least harder to do so.
>

I was going to leave this up to others to comment on here, but IMO -
excellent. Anyone that is doing an even semi serious deployment of
OpenStack is going to require puppet/chef/ansible or some form of
orchestration layer for deployment. Even for test deployments it seems to
me that it's crazy for this sort of functionality be handled from debconf.
The deployers of the system are going to understand if they want to use
eventlet or apache and should therefore understand what restarting apache
on a system implies.


>
> The other issue is that if all services are sharing the same web server,
> restarting the web server restarts all services. Or, said otherwise: if
> I need to change a configuration value of any of the services served by
> Apache, I will need to restart them all, which is very annoying: I very
> much prefer to just restart *ONE* service if I need.
>
> Also, something which we learned the hard way at Mirantis: it is *very*
> annoying that Apache restarts every Sunday morning by default in
> distributions like Ubuntu and Debian (I'm not sure for the other
> distros). No, the default config of logrotate and Apache can't be
> changed in distros just to satisfy OpenStack users: there's other users
> of Apache in these distros.
>

:O


>
> Then, yes, uWSGI becomes a nice option. I used it for the Barbican
> package, and it worked well. Though the uwsgi package in Debian isn't
> very well maintained, and multiple times, Barbican could have been
> removed from Debian testing because of RC bugs against uWSGI.
>
> So, all together, I'm a bit reluctant to see the Eventlet based servers
> going away. If it's done, then yes, I'll work around it. Though I'd
> prefer if it didn't.
>
> It is also my view that it's up to the deployers to decide how they want
> to implement things. For many small use cases, Eventlet performs well
> enough.
>
> Finally, one thing which I never understood: if Eventlet is bad as an
> HTTP server, can't we use anything else written in Python? Isn't it
> possible to write a decent HTTP server in Python? Why are we forced into
> just Eventlet for doing the job? I haven't searched around, but there
> must be loads of alternatives, no?
>
> Cheers,
>
> Thomas Goirand (zigo)
>

So I'd be ok with keeping eventlet around until after we can figure out
something for multiple virtual envs (i think you'd replace virtualenvs with
containers) , but i don't think the packaging should have anything to do
with this.


> ___
> OpenStack-operators mailing list
> openstack-operat...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Thomas Goirand]

On 12/01/2015 07:57 AM, Steve Martinelli wrote:
> Trying to summarize here...
> 
> - There isn't much interest in keeping eventlet around.
> - Folks are OK with running keystone in a WSGI server, but feel they are
> constrained by Apache.
> - uWSGI could help to support multiple web servers.
> 
> My opinion:
> 
> - Adding support for uWSGI definitely sounds like it's worth
> investigating, but not achievable in this release (unless someone
> already has something cooked up).
> - I'm tempted to let eventlet stick around another release, since it's
> causing pain on some of our operators.
> - Other folks have managed to run keystone in a web server (and
> hopefully not feel pain when doing so!), so it might be worth getting
> technical details on just how it was accomplished. If we get an OK from
> the operator community later on in mitaka, I'd still be OK with removing
> eventlet, but I don't want to break folks.
> 
> stevemar
> 
> From: John Dewey 
> 100% agree.
> 
> We should look at uwsgi as the reference architecture. Nginx/Apache/etc
> should be interchangeable, and up to the operator which they choose to
> use. Hell, with tcp load balancing now in opensource Nginx, I could get
> rid of Apache and HAProxy by utilizing uwsgi.
> 
> John

The main problem I see with running Keystone (or any other service) in a
web server, is that *I* (as a package maintainer) will loose the control
over when the service is started. Let me explain why that is important
for me.

In Debian, many services/daemons are run, then their API is used by the
package. In the case of Keystone, for example, it is possible to ask,
via Debconf, that Keystone registers itself in the service catalog. If
we get Keystone within Apache, it becomes at least harder to do so.

The other issue is that if all services are sharing the same web server,
restarting the web server restarts all services. Or, said otherwise: if
I need to change a configuration value of any of the services served by
Apache, I will need to restart them all, which is very annoying: I very
much prefer to just restart *ONE* service if I need.

Also, something which we learned the hard way at Mirantis: it is *very*
annoying that Apache restarts every Sunday morning by default in
distributions like Ubuntu and Debian (I'm not sure for the other
distros). No, the default config of logrotate and Apache can't be
changed in distros just to satisfy OpenStack users: there's other users
of Apache in these distros.

Then, yes, uWSGI becomes a nice option. I used it for the Barbican
package, and it worked well. Though the uwsgi package in Debian isn't
very well maintained, and multiple times, Barbican could have been
removed from Debian testing because of RC bugs against uWSGI.

So, all together, I'm a bit reluctant to see the Eventlet based servers
going away. If it's done, then yes, I'll work around it. Though I'd
prefer if it didn't.

It is also my view that it's up to the deployers to decide how they want
to implement things. For many small use cases, Eventlet performs well
enough.

Finally, one thing which I never understood: if Eventlet is bad as an
HTTP server, can't we use anything else written in Python? Isn't it
possible to write a decent HTTP server in Python? Why are we forced into
just Eventlet for doing the job? I haven't searched around, but there
must be loads of alternatives, no?

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Matthew Treinish]

On Mon, Dec 07, 2015 at 06:18:04PM -0500, Steve Martinelli wrote:
> 
> ... re-adding the operators mailing list.
> 
> sounds like we should document how to do this, with the assertion that it
> is not tested with our CI.
> 
> with that said, we should try to have a job that sets up keystone with
> nginx that is run periodically (similar to our eventlet job at the moment).

So, we actually run keystone with eventlet on every tempest-dsvm-postgres-full
job. It runs way more than periodically:

http://status.openstack.org/openstack-health/#/job/gate-tempest-dsvm-postgres-full
 

That's just a 24 hr window in the gate queue, including check it's much more.

This has been long standing behavior ever since keystone under mod_wsgi support
was added to devstack:

https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml#L1395-L1429

It's 1 of 3 things that are different that make the postgres job different. I've
always viewed that job config overloading as a bug, for this exact reason.

-Matt Treinish
 
> 
> From: Brant Knudson 
> To:   "OpenStack Development Mailing List (not for usage questions)"
>         
> Date: 2015/12/07 05:52 PM
> Subject:  Re: [openstack-dev] [Openstack-operators] [keystone] Removing
>     functionality that was deprecated in Kilo and upcoming
> deprecated functionality in Mitaka
> 
> 
> 
> 
> 
> On Tue, Dec 1, 2015 at 12:57 AM, Steve Martinelli 
> wrote:
>   Trying to summarize here...
> 
>   - There isn't much interest in keeping eventlet around.
>   - Folks are OK with running keystone in a WSGI server, but feel they are
>   constrained by Apache.
>   - uWSGI could help to support multiple web servers.
> 
>   My opinion:
> 
>   - Adding support for uWSGI definitely sounds like it's worth
>   investigating, but not achievable in this release (unless someone already
>   has something cooked up).
> 
> 
> 
> What needs to change to support uWSGI? You can already run keystone in
> python uwsgi and then front it with nginx:
> 
>  $ uwsgi --socket 127.0.0.1:5001 --wsgi-file $(which keystone-wsgi-public)
> --honour-stdin --enable-threads --workers 6
>  $ uwsgi --socket 127.0.0.1:35358 --wsgi-file $(which keystone-wsgi-admin)
> --honour-stdin --enable-threads --workers 6
> 
>  $ sudo vi /etc/nginx/sites-available/keystone
> 
> server {
>   listen 5000 default_server;
>   server_name localhost;
>   location / {
>     include uwsgi_params;
>     uwsgi_pass 127.0.0.1:5001;
>     uwsgi_param SCRIPT_NAME /;
>   }
> }
> server {
>   listen 35357 default_server;
>   server_name localhost;
>   location / {
>     include uwsgi_params;
>     uwsgi_pass 127.0.0.1:35358;
>     uwsgi_param SCRIPT_NAME /;
>   }
> }
> 
>  $ sudo ln -x /etc/nginx/sites-available/keystone /etc/nginx/sites-enabled/
> 
>  $ sudo nginx
> 
> and then you can make your regular curl calls.
> 
> Also, you can run keystone with regular http in python uwsgi (uwsgi --http)
> and then just do normal reverse proxy (from Apache or nginx or whatever),
> which I think would be adequate for keystone.
> 
> We don't do anything in keystone to stop deployments in web servers other
> than Apache. Keystone is just a regular wsgi app. We document Apache since
> it's popular and it provides mod_shib, which is the only saml2 module for
> web servers that I know of. Keystone can work with other saml2 modules and
> in different servers, it just takes the environment variables that the
> module sets and runs it through some mapping code. The mapping code has
> been shown to work alternative authentication modules (for ldap and
> kerberos).
> 
> - Brant
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 



> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Steve Martinelli]

... re-adding the operators mailing list.

sounds like we should document how to do this, with the assertion that it
is not tested with our CI.

with that said, we should try to have a job that sets up keystone with
nginx that is run periodically (similar to our eventlet job at the moment).

stevemar



From:   Brant Knudson 
To: "OpenStack Development Mailing List (not for usage questions)"

Date:   2015/12/07 05:52 PM
Subject:    Re: [openstack-dev] [Openstack-operators] [keystone] Removing
    functionality that was deprecated in Kilo and upcoming
    deprecated functionality in Mitaka





On Tue, Dec 1, 2015 at 12:57 AM, Steve Martinelli 
wrote:
  Trying to summarize here...

  - There isn't much interest in keeping eventlet around.
  - Folks are OK with running keystone in a WSGI server, but feel they are
  constrained by Apache.
  - uWSGI could help to support multiple web servers.

  My opinion:

  - Adding support for uWSGI definitely sounds like it's worth
  investigating, but not achievable in this release (unless someone already
  has something cooked up).



What needs to change to support uWSGI? You can already run keystone in
python uwsgi and then front it with nginx:

 $ uwsgi --socket 127.0.0.1:5001 --wsgi-file $(which keystone-wsgi-public)
--honour-stdin --enable-threads --workers 6
 $ uwsgi --socket 127.0.0.1:35358 --wsgi-file $(which keystone-wsgi-admin)
--honour-stdin --enable-threads --workers 6

 $ sudo vi /etc/nginx/sites-available/keystone

server {
  listen 5000 default_server;
  server_name localhost;
  location / {
    include uwsgi_params;
    uwsgi_pass 127.0.0.1:5001;
    uwsgi_param SCRIPT_NAME /;
  }
}
server {
  listen 35357 default_server;
  server_name localhost;
  location / {
    include uwsgi_params;
    uwsgi_pass 127.0.0.1:35358;
    uwsgi_param SCRIPT_NAME /;
  }
}

 $ sudo ln -x /etc/nginx/sites-available/keystone /etc/nginx/sites-enabled/

 $ sudo nginx

and then you can make your regular curl calls.

Also, you can run keystone with regular http in python uwsgi (uwsgi --http)
and then just do normal reverse proxy (from Apache or nginx or whatever),
which I think would be adequate for keystone.

We don't do anything in keystone to stop deployments in web servers other
than Apache. Keystone is just a regular wsgi app. We document Apache since
it's popular and it provides mod_shib, which is the only saml2 module for
web servers that I know of. Keystone can work with other saml2 modules and
in different servers, it just takes the environment variables that the
module sets and runs it through some mapping code. The mapping code has
been shown to work alternative authentication modules (for ldap and
kerberos).

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Morgan Fainberg]

On Dec 7, 2015 17:51, "Brant Knudson"  wrote:
>
>
>
> On Tue, Dec 1, 2015 at 12:57 AM, Steve Martinelli 
wrote:
>>
>> Trying to summarize here...
>>
>> - There isn't much interest in keeping eventlet around.
>> - Folks are OK with running keystone in a WSGI server, but feel they are
constrained by Apache.
>> - uWSGI could help to support multiple web servers.
>>
>> My opinion:
>>
>> - Adding support for uWSGI definitely sounds like it's worth
investigating, but not achievable in this release (unless someone already
has something cooked up).
>
>
> What needs to change to support uWSGI? You can already run keystone in
python uwsgi and then front it with nginx:
>
>  $ uwsgi --socket 127.0.0.1:5001 --wsgi-file $(which
keystone-wsgi-public) --honour-stdin --enable-threads --workers 6
>  $ uwsgi --socket 127.0.0.1:35358 --wsgi-file $(which
keystone-wsgi-admin) --honour-stdin --enable-threads --workers 6
>
>  $ sudo vi /etc/nginx/sites-available/keystone
>
> server {
>   listen 5000 default_server;
>   server_name localhost;
>   location / {
> include uwsgi_params;
> uwsgi_pass 127.0.0.1:5001;
> uwsgi_param SCRIPT_NAME /;
>   }
> }
> server {
>   listen 35357 default_server;
>   server_name localhost;
>   location / {
> include uwsgi_params;
> uwsgi_pass 127.0.0.1:35358;
> uwsgi_param SCRIPT_NAME /;
>   }
> }
>
>  $ sudo ln -x /etc/nginx/sites-available/keystone
/etc/nginx/sites-enabled/
>
>  $ sudo nginx
>
> and then you can make your regular curl calls.
>
> Also, you can run keystone with regular http in python uwsgi (uwsgi
--http) and then just do normal reverse proxy (from Apache or nginx or
whatever), which I think would be adequate for keystone.
>
> We don't do anything in keystone to stop deployments in web servers other
than Apache. Keystone is just a regular wsgi app. We document Apache since
it's popular and it provides mod_shib, which is the only saml2 module for
web servers that I know of. Keystone can work with other saml2 modules and
in different servers, it just takes the environment variables that the
module sets and runs it through some mapping code. The mapping code has
been shown to work alternative authentication modules (for ldap and
kerberos).

Recently I  looked and there seemed to be a similar module for nginx but I
don't know how mature it is. It might be worth exploring with more
discussion on alternatives to apache.

>
> - Brant
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Brant Knudson]

On Tue, Dec 1, 2015 at 12:57 AM, Steve Martinelli 
wrote:

> Trying to summarize here...
>
> - There isn't much interest in keeping eventlet around.
> - Folks are OK with running keystone in a WSGI server, but feel they are
> constrained by Apache.
> - uWSGI could help to support multiple web servers.
>
> My opinion:
>
> - Adding support for uWSGI definitely sounds like it's worth
> investigating, but not achievable in this release (unless someone already
> has something cooked up).
>

What needs to change to support uWSGI? You can already run keystone in
python uwsgi and then front it with nginx:

 $ uwsgi --socket 127.0.0.1:5001 --wsgi-file $(which keystone-wsgi-public)
--honour-stdin --enable-threads --workers 6
 $ uwsgi --socket 127.0.0.1:35358 --wsgi-file $(which keystone-wsgi-admin)
--honour-stdin --enable-threads --workers 6

 $ sudo vi /etc/nginx/sites-available/keystone

server {
  listen 5000 default_server;
  server_name localhost;
  location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:5001;
uwsgi_param SCRIPT_NAME /;
  }
}
server {
  listen 35357 default_server;
  server_name localhost;
  location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:35358;
uwsgi_param SCRIPT_NAME /;
  }
}

 $ sudo ln -x /etc/nginx/sites-available/keystone /etc/nginx/sites-enabled/

 $ sudo nginx

and then you can make your regular curl calls.

Also, you can run keystone with regular http in python uwsgi (uwsgi --http)
and then just do normal reverse proxy (from Apache or nginx or whatever),
which I think would be adequate for keystone.

We don't do anything in keystone to stop deployments in web servers other
than Apache. Keystone is just a regular wsgi app. We document Apache since
it's popular and it provides mod_shib, which is the only saml2 module for
web servers that I know of. Keystone can work with other saml2 modules and
in different servers, it just takes the environment variables that the
module sets and runs it through some mapping code. The mapping code has
been shown to work alternative authentication modules (for ldap and
kerberos).

- Brant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Fox, Kevin M]

Not really sure. If its reproducible with hammer+, I'd say yes. giant's 
keystone support was very basic, and it may not work against Mitaka anyway? It 
only supports v2.

Thanks,
Kevin

From: Adam Young [ayo...@redhat.com]
Sent: Wednesday, December 02, 2015 8:00 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Openstack-operators] [keystone] Removing 
functionality that was deprecated in Kilo and upcoming deprecated functionality 
in Mitaka

On 12/01/2015 03:50 PM, Fox, Kevin M wrote:
> I just upgraded to keystone liberty for one of my production clouds, and went 
> with apache since eventlet was listed as deprecated. It was pretty easy. Just 
> ran into one issue. RadosGW wouldn't work against it until I added 
> "WSGIChunkedRequest On'" in the config. otherwise, the config as shipped with 
> RDO worked fine. I am running giant radosgw, so future versions may not 
> require that.

Thanks for the note.  Should this be bug?

>
> Thanks,
> Kevin
> 
> From: Sean Dague [s...@dague.net]
> Sent: Tuesday, December 01, 2015 4:05 AM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [Openstack-operators] [keystone] Removing 
> functionality that was deprecated in Kilo and upcoming deprecated 
> functionality in Mitaka
>
> On 12/01/2015 01:57 AM, Steve Martinelli wrote:
>> Trying to summarize here...
>>
>> - There isn't much interest in keeping eventlet around.
>> - Folks are OK with running keystone in a WSGI server, but feel they are
>> constrained by Apache.
>  From an interop perspective, this concerns me a bit. My understanding is
> that Apache is specifically needed for Federation. Federation is the
> norm that we want for environments in the future.
>
> I'd hate to go down a path where the reference architecture we put out
> there doesn't support this. It's going to be all the pain of cells /
> non-cells that Nova's or nova-net / neutron bifurcation.
>
> Whatever the reference architecture is, it should support Federation. A
> non federation capable keystone should be the exception.
>
>> - uWSGI could help to support multiple web servers.
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Adam Young]

On 12/03/2015 07:51 AM, Morgan Fainberg wrote:


Chunked Encoding is a bad idea with mod_wsgi in general. While 
enabling it like that is fine, you are not guaranteed to get 100% 
consistent results simply because the wsgi spec did not/does not 
support it. Not all versions of mod_wsgi can enable it.


So, in short, officially keystone does not support chunked encoding. 
If radosgw cannot work without it, it is a bug against radosgw or 
something RDO itself is doing with radosgw.




RDO is not doing Keystone in HTTPD by default yet, and I have not come 
across the radosgw module at all.  Didn't think it was a real issue, but 
want to track if it is.



--morgan

On Dec 2, 2015 23:00, "Adam Young" <mailto:ayo...@redhat.com>> wrote:


On 12/01/2015 03:50 PM, Fox, Kevin M wrote:

I just upgraded to keystone liberty for one of my production
clouds, and went with apache since eventlet was listed as
deprecated. It was pretty easy. Just ran into one issue.
RadosGW wouldn't work against it until I added
"WSGIChunkedRequest On'" in the config. otherwise, the config
as shipped with RDO worked fine. I am running giant radosgw,
so future versions may not require that.


Thanks for the note.  Should this be bug?


Thanks,
Kevin

From: Sean Dague [s...@dague.net <mailto:s...@dague.net>]
Sent: Tuesday, December 01, 2015 4:05 AM
To: openstack-dev@lists.openstack.org
<mailto:openstack-dev@lists.openstack.org>
    Subject: Re: [openstack-dev] [Openstack-operators] [keystone]
        Removing functionality that was deprecated in Kilo and
upcoming deprecated functionality in Mitaka

On 12/01/2015 01:57 AM, Steve Martinelli wrote:

Trying to summarize here...

- There isn't much interest in keeping eventlet around.
- Folks are OK with running keystone in a WSGI server, but
feel they are
constrained by Apache.

 From an interop perspective, this concerns me a bit. My
understanding is
that Apache is specifically needed for Federation. Federation
is the
norm that we want for environments in the future.

I'd hate to go down a path where the reference architecture we
put out
there doesn't support this. It's going to be all the pain of
cells /
non-cells that Nova's or nova-net / neutron bifurcation.

Whatever the reference architecture is, it should support
Federation. A
non federation capable keystone should be the exception.

- uWSGI could help to support multiple web servers.


--
Sean Dague
http://dague.net


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Morgan Fainberg]

Chunked Encoding is a bad idea with mod_wsgi in general. While enabling it
like that is fine, you are not guaranteed to get 100% consistent results
simply because the wsgi spec did not/does not support it. Not all versions
of mod_wsgi can enable it.

So, in short, officially keystone does not support chunked encoding. If
radosgw cannot work without it, it is a bug against radosgw or something
RDO itself is doing with radosgw.

--morgan
On Dec 2, 2015 23:00, "Adam Young"  wrote:

> On 12/01/2015 03:50 PM, Fox, Kevin M wrote:
>
>> I just upgraded to keystone liberty for one of my production clouds, and
>> went with apache since eventlet was listed as deprecated. It was pretty
>> easy. Just ran into one issue. RadosGW wouldn't work against it until I
>> added "WSGIChunkedRequest On'" in the config. otherwise, the config as
>> shipped with RDO worked fine. I am running giant radosgw, so future
>> versions may not require that.
>>
>
> Thanks for the note.  Should this be bug?
>
>
>> Thanks,
>> Kevin
>> 
>> From: Sean Dague [s...@dague.net]
>> Sent: Tuesday, December 01, 2015 4:05 AM
>> To: openstack-dev@lists.openstack.org
>> Subject: Re: [openstack-dev] [Openstack-operators] [keystone] Removing
>> functionality that was deprecated in Kilo and upcoming deprecated
>> functionality in Mitaka
>>
>> On 12/01/2015 01:57 AM, Steve Martinelli wrote:
>>
>>> Trying to summarize here...
>>>
>>> - There isn't much interest in keeping eventlet around.
>>> - Folks are OK with running keystone in a WSGI server, but feel they are
>>> constrained by Apache.
>>>
>>  From an interop perspective, this concerns me a bit. My understanding is
>> that Apache is specifically needed for Federation. Federation is the
>> norm that we want for environments in the future.
>>
>> I'd hate to go down a path where the reference architecture we put out
>> there doesn't support this. It's going to be all the pain of cells /
>> non-cells that Nova's or nova-net / neutron bifurcation.
>>
>> Whatever the reference architecture is, it should support Federation. A
>> non federation capable keystone should be the exception.
>>
>> - uWSGI could help to support multiple web servers.
>>>
>>
>> --
>> Sean Dague
>> http://dague.net
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Adam Young]

On 12/01/2015 03:50 PM, Fox, Kevin M wrote:

I just upgraded to keystone liberty for one of my production clouds, and went with apache 
since eventlet was listed as deprecated. It was pretty easy. Just ran into one issue. 
RadosGW wouldn't work against it until I added "WSGIChunkedRequest On'" in the 
config. otherwise, the config as shipped with RDO worked fine. I am running giant 
radosgw, so future versions may not require that.


Thanks for the note.  Should this be bug?



Thanks,
Kevin

From: Sean Dague [s...@dague.net]
Sent: Tuesday, December 01, 2015 4:05 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Openstack-operators] [keystone] Removing 
functionality that was deprecated in Kilo and upcoming deprecated functionality 
in Mitaka

On 12/01/2015 01:57 AM, Steve Martinelli wrote:

Trying to summarize here...

- There isn't much interest in keeping eventlet around.
- Folks are OK with running keystone in a WSGI server, but feel they are
constrained by Apache.

 From an interop perspective, this concerns me a bit. My understanding is
that Apache is specifically needed for Federation. Federation is the
norm that we want for environments in the future.

I'd hate to go down a path where the reference architecture we put out
there doesn't support this. It's going to be all the pain of cells /
non-cells that Nova's or nova-net / neutron bifurcation.

Whatever the reference architecture is, it should support Federation. A
non federation capable keystone should be the exception.


- uWSGI could help to support multiple web servers.


--
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Morgan Fainberg]

On Tue, Dec 1, 2015 at 1:57 AM, Steve Martinelli 
wrote:

> Trying to summarize here...
>
> - There isn't much interest in keeping eventlet around.
> - Folks are OK with running keystone in a WSGI server, but feel they are
> constrained by Apache.
> - uWSGI could help to support multiple web servers.
>
> My opinion:
>
> - Adding support for uWSGI definitely sounds like it's worth
> investigating, but not achievable in this release (unless someone already
> has something cooked up).
>
uWSGI is trivial to use, but there are a ton of options that go along with
it. Our current wsgi file that works w/ mod_wsgi pretty much "just works"
for uWSGI. The "configure uWSGI" in a sane way is much much much more in
depth.

> - I'm tempted to let eventlet stick around another release, since it's
> causing pain on some of our operators.
> - Other folks have managed to run keystone in a web server (and hopefully
> not feel pain when doing so!), so it might be worth getting technical
> details on just how it was accomplished. If we get an OK from the operator
> community later on in mitaka, I'd still be OK with removing eventlet, but I
> don't want to break folks.
>
> stevemar
>
> From: John Dewey 
> 100% agree.
>
> We should look at uwsgi as the reference architecture. Nginx/Apache/etc
> should be interchangeable, and up to the operator which they choose to use.
> Hell, with tcp load balancing now in opensource Nginx, I could get rid of
> Apache and HAProxy by utilizing uwsgi.
>
> John
>
> On November 30, 2015 at 1:05:26 PM, Paul Czarkowski (
> *pczarkowski+openstack...@bluebox.net*
> ) wrote:
>
>I don't have a problem with eventlet itself going away, but I do feel
>   that keystone should pick a python based web server capable of running 
> WSGI
>   apps ( such as uWSGI ) for the reference implementation rather than 
> Apache
>   which can be declared appropriately in the requirements.txt of the 
> project.
>   I feel it is important to allow the operator to make choices based on 
> their
>   organization's skill sets ( i.e. apache vs nginx ) to help keep 
> complexity
>   low.
>
>   I understand there are some newer features that rely on Apache (
>   federation, etc ) but we should allow the need for those features inform
>   the operators choice of web server rather than force it for everybody.
>
>   Having a default implementation using uWSGI is also more inline
>   with the 12 factor way of writing applications and will run a lot more
>   comfortably in [application] containers than apache would which is 
> probably
>   an important consideration given how many people are focused on being 
> able
>   to run openstack projects inside containers.
>
>   On Mon, Nov 30, 2015 at 2:36 PM, Jesse Keating <*j...@bluebox.net*
>   > wrote:
>  I have an objection to eventlet going away. We have problems
>  with running Apache and mod_wsgi with multiple python virtual 
> environments.
>  In some of our stacks we're running both Horizon and Keystone. Each 
> get
>  their own virtual environment. Apache mod_wsgi doesn't really work 
> that
>  way, so we'd have to do some ugly hacks to expose the python 
> environments
>  of both to Apache at the same time.
>
>  I believe we spoke about this at Summit. Have you had time to
>  look into this scenario and have suggestions?
>
>
>  - jlk
>
>  On Mon, Nov 30, 2015 at 10:26 AM, Steve Martinelli <
>  *steve...@ca.ibm.com* > wrote:
>  This post is being sent again to the operators mailing list, and
>  i apologize if it's duplicated for some folks. The original thread 
> is here:
>  
> *http://lists.openstack.org/pipermail/openstack-dev/2015-November/080816.html*
>  
> 
>
>
>  In the Mitaka release, the keystone team will be removing
>  functionality that was marked for deprecation in Kilo, and marking 
> certain
>  functions as deprecated in Mitaka (that may be removed in at least 2
>  cycles).
>
>  removing deprecated functionality
>  =
>
>  This is not a full list, but these are by and large the most
>  contentious topics.
>
>  * Eventlet support: This was marked as deprecated back in Kilo
>  and is currently scheduled to be removed in Mitaka in favor of 
> running
>  keystone in a WSGI server. This is currently how we test keystone in 
> the
>  gate, and based on the feedback we received at the summit, a lot of 
> folks
>  have moved to running keystone under Apache since we’ve announced 
> this
>  change. OpenStack's CI is configured to mainly test using this 
> deployment
>  model. See [0] for when we started to issue warnings.
>
>  * Using LDAP to store assignment data: Like even

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Fox, Kevin M]

I just upgraded to keystone liberty for one of my production clouds, and went 
with apache since eventlet was listed as deprecated. It was pretty easy. Just 
ran into one issue. RadosGW wouldn't work against it until I added 
"WSGIChunkedRequest On'" in the config. otherwise, the config as shipped with 
RDO worked fine. I am running giant radosgw, so future versions may not require 
that.

Thanks,
Kevin

From: Sean Dague [s...@dague.net]
Sent: Tuesday, December 01, 2015 4:05 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Openstack-operators] [keystone] Removing 
functionality that was deprecated in Kilo and upcoming deprecated functionality 
in Mitaka

On 12/01/2015 01:57 AM, Steve Martinelli wrote:
> Trying to summarize here...
>
> - There isn't much interest in keeping eventlet around.
> - Folks are OK with running keystone in a WSGI server, but feel they are
> constrained by Apache.

>From an interop perspective, this concerns me a bit. My understanding is
that Apache is specifically needed for Federation. Federation is the
norm that we want for environments in the future.

I'd hate to go down a path where the reference architecture we put out
there doesn't support this. It's going to be all the pain of cells /
non-cells that Nova's or nova-net / neutron bifurcation.

Whatever the reference architecture is, it should support Federation. A
non federation capable keystone should be the exception.

> - uWSGI could help to support multiple web servers.


--
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Boris Bobrov]

On Tuesday 01 December 2015 08:50:14 Lance Bragstad wrote:
> On Tue, Dec 1, 2015 at 6:05 AM, Sean Dague  wrote:
> > 
> > From an interop perspective, this concerns me a bit. My
> > understanding is that Apache is specifically needed for
> > Federation. Federation is the norm that we want for environments
> > in the future.
> 
> (On a side note from removing eventlet, but related to what Sean
> said)
> 
> A spec has been proposed to make keystone a fully fledged saml2
> provider [0]. Depending on how we feel about implementing and
> maintaining something like this, we'd be able to use federation
> within uWSGI (we would no longer *require* Apache for federation).
> Only bringing this up because it would also solve the
> two-reference-architectures problem. A uWSGI reference architecture
> could be used for deploying keystone, regardless if you want
> federation or not.
> 
> We probably wouldn't get a uWSGI reference architecture until after
> that is all fleshed out. This is assuming the spec is accepted and
> implemented in Mitaka.
> 
> [0] https://review.openstack.org/#/c/244694/5

I don't get why we talk about uwsgi in context of federation. uwsgi is 
an application server. Apache is a web server. We can still use uwsgi 
with apache, there are several modules for that:
https://uwsgi-docs.readthedocs.org/en/latest/Apache.html

Now we require apache for federation and support mod_wsgi (which is 
tightly integrated with apache) as an app server. We can still require 
Apache and support uwsgi as an app server, without any changes to 
federation.

-- 
Best regards,
Boris Bobrov

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Lance Bragstad]

On Tue, Dec 1, 2015 at 6:05 AM, Sean Dague  wrote:

> On 12/01/2015 01:57 AM, Steve Martinelli wrote:
> > Trying to summarize here...
> >
> > - There isn't much interest in keeping eventlet around.
> > - Folks are OK with running keystone in a WSGI server, but feel they are
> > constrained by Apache.
>
> From an interop perspective, this concerns me a bit. My understanding is
> that Apache is specifically needed for Federation. Federation is the
> norm that we want for environments in the future.
>

(On a side note from removing eventlet, but related to what Sean said)

A spec has been proposed to make keystone a fully fledged saml2 provider
[0]. Depending on how we feel about implementing and maintaining something
like this, we'd be able to use federation within uWSGI (we would no longer
*require* Apache for federation). Only bringing this up because it would
also solve the two-reference-architectures problem. A uWSGI reference
architecture could be used for deploying keystone, regardless if you want
federation or not.

We probably wouldn't get a uWSGI reference architecture until after that is
all fleshed out. This is assuming the spec is accepted and implemented in
Mitaka.

Not to take away from the current thread, but it seems partially relevant.
Also, this seems like a good opportunity to gather thoughts on the idea :)

[0] https://review.openstack.org/#/c/244694/5


> I'd hate to go down a path where the reference architecture we put out
> there doesn't support this. It's going to be all the pain of cells /
> non-cells that Nova's or nova-net / neutron bifurcation.
>
> Whatever the reference architecture is, it should support Federation. A
> non federation capable keystone should be the exception.
>
> > - uWSGI could help to support multiple web servers.
>
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Sean Dague]

On 12/01/2015 01:57 AM, Steve Martinelli wrote:
> Trying to summarize here...
> 
> - There isn't much interest in keeping eventlet around.
> - Folks are OK with running keystone in a WSGI server, but feel they are
> constrained by Apache.

>From an interop perspective, this concerns me a bit. My understanding is
that Apache is specifically needed for Federation. Federation is the
norm that we want for environments in the future.

I'd hate to go down a path where the reference architecture we put out
there doesn't support this. It's going to be all the pain of cells /
non-cells that Nova's or nova-net / neutron bifurcation.

Whatever the reference architecture is, it should support Federation. A
non federation capable keystone should be the exception.

> - uWSGI could help to support multiple web servers.


-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

[Steve Martinelli]

Trying to summarize here...

  - There isn't much interest in keeping eventlet around.
  - Folks are OK with running keystone in a WSGI server, but feel they are
constrained by Apache.
  - uWSGI could help to support multiple web servers.

My opinion:

  - Adding support for uWSGI definitely sounds like it's worth
investigating, but not achievable in this release (unless someone already
has something cooked up).
  - I'm tempted to let eventlet stick around another release, since it's
causing pain on some of our operators.
  - Other folks have managed to run keystone in a web server (and hopefully
not feel pain when doing so!), so it might be worth getting technical
details on just how it was accomplished. If we get an OK from the operator
community later on in mitaka, I'd still be OK with removing eventlet, but I
don't want to break folks.

stevemar

From:   John Dewey 
100% agree.

We should look at uwsgi as the reference architecture.  Nginx/Apache/etc
should be interchangeable, and up to the operator which they choose to use.
Hell, with tcp load balancing now in opensource Nginx, I could get rid of
Apache and HAProxy by utilizing uwsgi.

John


On November 30, 2015 at 1:05:26 PM, Paul Czarkowski (pczarkowski
+openstack...@bluebox.net) wrote:


  I don't have a problem with eventlet itself going away, but I do feel
  that keystone should pick a python based web server capable of
  running WSGI apps ( such as uWSGI ) for the reference implementation
  rather than Apache which can be declared appropriately in the
  requirements.txt of the project.   I feel it is important to allow
  the operator to make choices based on their organization's skill sets
  ( i.e. apache vs nginx ) to help keep complexity low.

  I understand there are some newer features that rely on Apache
  ( federation, etc )  but we should allow the need for those features
  inform the operators choice of web server rather than force it for
  everybody.

  Having a default implementation using uWSGI is also more inline with
  the 12 factor way of writing applications and will run a lot more
  comfortably in [application] containers than apache would which is
  probably an important consideration given how many people are focused
  on being able to run openstack projects inside containers.

  On Mon, Nov 30, 2015 at 2:36 PM, Jesse Keating 
  wrote:
I have an objection to eventlet going away. We have problems with
running Apache and mod_wsgi with multiple python virtual
environments. In some of our stacks we're running both Horizon and
Keystone. Each get their own virtual environment. Apache mod_wsgi
doesn't really work that way, so we'd have to do some ugly hacks to
expose the python environments of both to Apache at the same time.

I believe we spoke about this at Summit. Have you had time to look
into this scenario and have suggestions?


- jlk

On Mon, Nov 30, 2015 at 10:26 AM, Steve Martinelli <
steve...@ca.ibm.com> wrote:
 This post is being sent again to the operators mailing list, and i
 apologize if it's duplicated for some folks. The original thread
 is here:
 
http://lists.openstack.org/pipermail/openstack-dev/2015-November/080816.html


 In the Mitaka release, the keystone team will be removing
 functionality that was marked for deprecation in Kilo, and marking
 certain functions as deprecated in Mitaka (that may be removed in
 at least 2 cycles).

 removing deprecated functionality
 =

 This is not a full list, but these are by and large the most
 contentious topics.

 * Eventlet support: This was marked as deprecated back in Kilo and
 is currently scheduled to be removed in Mitaka in favor of running
 keystone in a WSGI server. This is currently how we test keystone
 in the gate, and based on the feedback we received at the summit,
 a lot of folks have moved to running keystone under Apache since
 we’ve announced this change. OpenStack's CI is configured to
 mainly test using this deployment model. See [0] for when we
 started to issue warnings.

 * Using LDAP to store assignment data: Like eventlet support, this
 feature was also deprecated in Kilo and scheduled to be removed in
 Mitaka. To store assignment data (role assignments) we suggest
 using an SQL based backend rather than LDAP. See [1] for when we
 started to issue warnings.

 * Using LDAP to store project and domain data: The same as above,
 see [2] for when we started to issue warnings.

 * for a complete list:
 https://blueprints.launchpad.net/keystone/+spec/removed-as-of-mitaka


 functions deprecated as of mitaka
 =