Re: [Openstack] Keystone should to Apache HTTPD.
I like the recommendation. Particularly with regards to using PKI authentication. On Thu, Mar 1, 2012 at 2:05 PM, Adam Young ayo...@redhat.com wrote: I wrote up why I think that, at least for Keystone, we should move the front end over to Apache HTTPD. http://adam.younglogic.com/2012/03/keystone-should-move-to-apache-httpd/ I've reposted it below. Keystone and the other Openstack components run in an a href=http://www.eventlet.net;Eventlet/a based HTTP server. Eventlet and a href=http://http://pypi.python.org/pypi/greenlet; Greenlet/a (the project Eventlet is built on) are designed to be highly performant in networked environments due to their non-blocking nature. Everything is handled in a single thread, and scheduling is performed in user space. The one caveat is that not only must the code you write never block, the code you call must not block, either. If you are going to make a call into a third party library that performs I/O, you need to wrap that library in a thread pool. For Keystone, every call is going to be going through to a Database layer, either SQL or LDAP. Which is in turn going to call into the native driver for that Database, or the LDAP libraries. Either way, it is a native call, and it has to be wrapped in a thread pool. Keystone is an authentication hub. As such, it is literally the Keystone of the security architecture around Openstack. In order to do anything on any of the other services in Openstack, a use needs a token from Keystone. But in order to authenticate against Keystone, the user needs to provide a clear-text password. This approach may be sufficient to start development, but it is not going to be acceptable when a company needs to prove compliance with a title=Sarbanes Oxley Act of 2002 href=http://www.soxlaw.com/; target=_blankSarbanes-Oxley/a. Or a title=Health Information Privacy href=http://www.hhs.gov/ocr/privacy/hipaa/understanding/index.html; target=_blankHIPPAA/a. For these cases, we want stronger encryption and better authentication management. The Eventlet based web server does not currently support forms of authentication other than Basic-Auth. Ideally, organizations would be able to employ their Kerberos or Public Key Infrastructure (PKI) assets to support their Openstack based authentication. IPv6 is coming. The last block of IPv4 addresses has been allocated. For Cloud based deployments, people are going to need large numbers of routable IP Addresses. However, Eventlet does not currently support IPv6. Work is happening upstream, but it has not yet been commited, and will not be available for the Essex release of Openstack. There is a simple solution. Keystone is a WSGI application, and has minimal dependencies on Eventlet. Deploying Keystone in an Apache HTTPD server with mod_wsgi running in prefork mode provides the same operating environment as Eventlet does. As the de facto standard open source web server, it has received a higher degree of scrutiny than most other software products. HTTPD support for GSSAPI/Kerberos authentication, Client and Server based certificates, and IPv6. It is well supported in all the major Linux distributions. What would the drawbacks be? Probably the first thing people would look to from Eventlet is performance. I don't have the hard numbers to compare Eventlet to Apache HTTPD, but I do know that Apache is used in enough high volume sites that I would not be overly concerned. The traffic in an Openstack deployment to a Keystone server is going to be about two orders of magnitude less than any other traffic, and is highly unlikely to be the bottleneck. Second is the fact that we would be pulling in dependencies to Apache HTTPD, and that configuring it would be different and more difficult than Eventlet. However, this is a fairly well trodden path. The benefits of putting all HTTP traffic behind ports 80 and 443 overwhelm the drawbacks of configuration. Since Keystone has just gone through a major reworking, I realize that people might be reluctant to support a move like this. However, the effect on other components should be minimal or none. Apache HTTPD can be set up using the same ports that Keystone already uses, and thus replace an existing Keystone install with no changes to the configuration or code to the other services. The changes should be limited to Keystone alone. The problem that Eventlet solves does not map to Keystone. The amount of work it would take to add the features Keystone really requires to Eventlet is significant, is difficult, and is likely to be far buggier than using well established and audited libraries. The simpler path forward is for Keystone to move over to Apache HTTPD. It is also the path for greater stability, security, and growth. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe :
Re: [Openstack] Keystone should to Apache HTTPD.
This seems like it could also be done for every other WSGI endpoint right? It seems like it should be possible for all WS endpoints to be hosted in apache (or other server) without problems happening. This might be connected to extracting/abstractig out eventlet (since a pre-forked apache doesn't really care about that model) so that it is not always needed. On 3/1/12 11:05 AM, Adam Young ayo...@redhat.com wrote: I wrote up why I think that, at least for Keystone, we should move the front end over to Apache HTTPD. http://adam.younglogic.com/2012/03/keystone-should-move-to-apache-httpd/ I've reposted it below. Keystone and the other Openstack components run in an a href=http://www.eventlet.net;Eventlet/a based HTTP server. Eventlet and a href=http://http://pypi.python.org/pypi/greenlet; Greenlet/a (the project Eventlet is built on) are designed to be highly performant in networked environments due to their non-blocking nature. Everything is handled in a single thread, and scheduling is performed in user space. The one caveat is that not only must the code you write never block, the code you call must not block, either. If you are going to make a call into a third party library that performs I/O, you need to wrap that library in a thread pool. For Keystone, every call is going to be going through to a Database layer, either SQL or LDAP. Which is in turn going to call into the native driver for that Database, or the LDAP libraries. Either way, it is a native call, and it has to be wrapped in a thread pool. Keystone is an authentication hub. As such, it is literally the Keystone of the security architecture around Openstack. In order to do anything on any of the other services in Openstack, a use needs a token from Keystone. But in order to authenticate against Keystone, the user needs to provide a clear-text password. This approach may be sufficient to start development, but it is not going to be acceptable when a company needs to prove compliance with a title=Sarbanes Oxley Act of 2002 href=http://www.soxlaw.com/; target=_blankSarbanes-Oxley/a. Or a title=Health Information Privacy href=http://www.hhs.gov/ocr/privacy/hipaa/understanding/index.html; target=_blankHIPPAA/a. For these cases, we want stronger encryption and better authentication management. The Eventlet based web server does not currently support forms of authentication other than Basic-Auth. Ideally, organizations would be able to employ their Kerberos or Public Key Infrastructure (PKI) assets to support their Openstack based authentication. IPv6 is coming. The last block of IPv4 addresses has been allocated. For Cloud based deployments, people are going to need large numbers of routable IP Addresses. However, Eventlet does not currently support IPv6. Work is happening upstream, but it has not yet been commited, and will not be available for the Essex release of Openstack. There is a simple solution. Keystone is a WSGI application, and has minimal dependencies on Eventlet. Deploying Keystone in an Apache HTTPD server with mod_wsgi running in prefork mode provides the same operating environment as Eventlet does. As the de facto standard open source web server, it has received a higher degree of scrutiny than most other software products. HTTPD support for GSSAPI/Kerberos authentication, Client and Server based certificates, and IPv6. It is well supported in all the major Linux distributions. What would the drawbacks be? Probably the first thing people would look to from Eventlet is performance. I don't have the hard numbers to compare Eventlet to Apache HTTPD, but I do know that Apache is used in enough high volume sites that I would not be overly concerned. The traffic in an Openstack deployment to a Keystone server is going to be about two orders of magnitude less than any other traffic, and is highly unlikely to be the bottleneck. Second is the fact that we would be pulling in dependencies to Apache HTTPD, and that configuring it would be different and more difficult than Eventlet. However, this is a fairly well trodden path. The benefits of putting all HTTP traffic behind ports 80 and 443 overwhelm the drawbacks of configuration. Since Keystone has just gone through a major reworking, I realize that people might be reluctant to support a move like this. However, the effect on other components should be minimal or none. Apache HTTPD can be set up using the same ports that Keystone already uses, and thus replace an existing Keystone install with no changes to the configuration or code to the other services. The changes should be limited to Keystone alone. The problem that Eventlet solves does not map to Keystone. The amount of work it would take to add the features Keystone really requires to Eventlet is significant, is difficult, and is likely to be far buggier than using well established and audited libraries. The simpler path forward is for Keystone to move over to Apache HTTPD. It is also the path for greater
Re: [Openstack] Keystone should to Apache HTTPD.
On Thu, 2012-03-01 at 14:05 -0500, Adam Young wrote: The traffic in an Openstack deployment to a Keystone server is going to be about two orders of magnitude less than any other traffic, and is highly unlikely to be the bottleneck. Not quite. I wrote this up, back in November: http://etherpad.openstack.org/keystone-scalability Since then, of course, Keystone has gone through some major cleanups that have improved its efficiency, but, as Vish pointed out in the other thread, every service still has to hit Keystone to verify a given token, which makes Keystone have the highest number of hits for any given operation…which in turn makes it *the* most likely bottleneck. -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Keystone should to Apache HTTPD.
On 03/01/2012 03:52 PM, Kevin L. Mitchell wrote: On Thu, 2012-03-01 at 14:05 -0500, Adam Young wrote: The traffic in an Openstack deployment to a Keystone server is going to be about two orders of magnitude less than any other traffic, and is highly unlikely to be the bottleneck. Not quite. I wrote this up, back in November: http://etherpad.openstack.org/keystone-scalability Since then, of course, Keystone has gone through some major cleanups that have improved its efficiency, but, as Vish pointed out in the other thread, every service still has to hit Keystone to verify a given token, which makes Keystone have the highest number of hits for any given operation…which in turn makes it *the* most likely bottleneck. Good write up. My SWAG estimate was based on an efficient protocol. I am still learning Keystone, so I can't say as far as how it is deployed in practice. HMAC can be supported with a Public/Private key pair fairly easily. If the HMAC is signed with a private key, the other service can confirm them with a public key. It should really not require a round trip to verify a token. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
