Re: [Openstack] Distributed rate-limiting
Hi, Sorry to bring alive a fairly old thread, but I had a few questions on Nova's rate limiting in a distributed/ load balanced Openstack environment. My understanding is Turnstile manages the situation where, the in-memory rate limits that are configured on load balanced API servers are imposed properly on the incoming requests, so each API server is correctly updated/synced with the used rate limits. Can you please confirm this understanding? Also, I don't think this is part of the Openstack trunk code, and if so, is there any reason why it's not part of Nova, as it was meant to be a replacement? Regards, Rohit -Original Message- From: openstack-bounces+rohit.karajgi=vertex.co...@lists.launchpad.net [mailto:openstack-bounces+rohit.karajgi=vertex.co...@lists.launchpad.net] On Behalf Of Kevin L. Mitchell Sent: Saturday, March 17, 2012 3:15 AM To: openstack@lists.launchpad.net Subject: [Openstack] Distributed rate-limiting Howdy, folks. I've been working on a replacement for nova's rate-limiting middleware that will handle the multiple-node case, and I've developed a fairly generic rate-limiting package, along with a second package that adapts it to nova. (This means you could also use this rate-limiting setup with, say, glance, or with any other project that uses Python middleware.) Here is some information: * Turnstile Turnstile is a piece of WSGI middleware that performs true distributed rate-limiting. System administrators can run an API on multiple nodes, then place this middleware in the pipeline prior to the application. Turnstile uses a Redis database to track the rate at which users are hitting the API, and can then apply configured rate limits, even if each request was made against a different API node. - https://github.com/klmitch/turnstile - http://pypi.python.org/pypi/turnstile * nova_limits This package provides the ``nova_limits`` Python module, which contains the ``nova_preprocess()`` preprocessor, the ``NovaClassLimit`` limit class, and the ``NovaTurnstileMiddleware`` replacement middleware class, all for use with Turnstile. These pieces work together to provide class-based rate limiting integration with nova. - https://github.com/klmitch/nova_limits - http://pypi.python.org/pypi/nova_limits Both packages should be fairly well documented (start with README.rst), and please feel free to log issues or make pull requests. -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp __ Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Distributed rate-limiting
On Wed, 2012-12-05 at 14:12 +, Karajgi, Rohit wrote: My understanding is Turnstile manages the situation where, the in-memory rate limits that are configured on load balanced API servers are imposed properly on the incoming requests, so each API server is correctly updated/synced with the used rate limits. Can you please confirm this understanding? Yes. Turnstile uses Redis to coordinate rate limit configuration and bucket data, in order to provide rate limiting. Also, I don't think this is part of the Openstack trunk code, and if so, is there any reason why it's not part of Nova, as it was meant to be a replacement? I wrote Turnstile to be general; it can be used for Nova, Keystone, or any other system for which rate limiting is desired. (I in fact designed it with a goal of being able to use it for some personal projects which are not OpenStack-related.) This is the primary reason it's not a direct part of any OpenStack repository. That said, it is hosted on github and I welcome pull-requests…and I'm not at all adverse to the suggestion that it become an OpenStack project; I'm just not convinced that that would be generally desired, or that it would be generally beneficial… -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Distributed rate-limiting
Yep - good point Chris Kevin. I hadn't thought of it that way. -Original Message- From: openstack-bounces+philip.day=hp@lists.launchpad.net [mailto:openstack-bounces+philip.day=hp@lists.launchpad.net] On Behalf Of Chris Behrens Sent: 30 March 2012 00:30 To: Kevin L. Mitchell Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Distributed rate-limiting My issue with using the URL is someone could easily DoS any tenant. Maybe you said that below. I only have a brief moment to scan email ATM. :) On Mar 29, 2012, at 5:26 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com wrote: On Thu, 2012-03-29 at 22:58 +0100, Day, Phil wrote: - As you get the tenant id from the context I assume this module has to come after the authentication in the pipeline. Yes, I have made that assumption. It seems reasonable, given that the existing rate-limit middleware is right after authentication as well. Have you thought about using the tenant_id in the URL instead ? (I'm thinking of the case where you want rate limit requests into the authentication system as well as Nova itself). No, I haven't. I don't trust the user, which is where the tenant_id in the URL comes from. I do trust the auth system, which is why I want to use the tenant ID from the context. (And yes, you could argue that authz would prevent their access to other tenants anyway, but why make nova have to check authz if rate limiting would stop them in the first place?) As for rate limiting requests into the authentication system, I'd suggest using a Limit subclass which uses the remote IP address in place of a tenant ID, at least for the user endpoint. I don't think we want any rate limiting at all on the service side of Keystone; our current architecture means that Keystone is going to be hit a *lot*: at least once for each request that hits Nova, and more in certain cases (i.e., instance boot, where we'll have to hit quantum and glance as well). - Does this work for EC2 as well as OSAPI ? Actually, it didn't occur to me to test, given that I don't really use the EC2 API. I don't think there's anything in the basic architecture which would be incompatible with EC2; the only possible sticking point that occurs to me is the URL construction in nova_limits:NovaClassLimit.route(): if the URL specified is prefixed with '/v1.1/' or '/v2/', the version identifier is dropped (otherwise the route wouldn't match). That would be easy to work around; simply extend NovaClassLimit and override route() to do the appropriate transformation for EC2. Any EC2 experts want to weigh in? -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Distributed rate-limiting
Hi Kevin, A couple of quick questions: - As you get the tenant id from the context I assume this module has to come after the authentication in the pipeline. Have you thought about using the tenant_id in the URL instead ? (I'm thinking of the case where you want rate limit requests into the authentication system as well as Nova itself). - Does this work for EC2 as well as OSAPI ? Cheers, Phil -Original Message- From: openstack-bounces+philip.day=hp@lists.launchpad.net [mailto:openstack-bounces+philip.day=hp@lists.launchpad.net] On Behalf Of Kevin L. Mitchell Sent: 16 March 2012 21:45 To: openstack@lists.launchpad.net Subject: [Openstack] Distributed rate-limiting Howdy, folks. I've been working on a replacement for nova's rate-limiting middleware that will handle the multiple-node case, and I've developed a fairly generic rate-limiting package, along with a second package that adapts it to nova. (This means you could also use this rate-limiting setup with, say, glance, or with any other project that uses Python middleware.) Here is some information: * Turnstile Turnstile is a piece of WSGI middleware that performs true distributed rate-limiting. System administrators can run an API on multiple nodes, then place this middleware in the pipeline prior to the application. Turnstile uses a Redis database to track the rate at which users are hitting the API, and can then apply configured rate limits, even if each request was made against a different API node. - https://github.com/klmitch/turnstile - http://pypi.python.org/pypi/turnstile * nova_limits This package provides the ``nova_limits`` Python module, which contains the ``nova_preprocess()`` preprocessor, the ``NovaClassLimit`` limit class, and the ``NovaTurnstileMiddleware`` replacement middleware class, all for use with Turnstile. These pieces work together to provide class-based rate limiting integration with nova. - https://github.com/klmitch/nova_limits - http://pypi.python.org/pypi/nova_limits Both packages should be fairly well documented (start with README.rst), and please feel free to log issues or make pull requests. -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Distributed rate-limiting
On Thu, 2012-03-29 at 22:58 +0100, Day, Phil wrote: - As you get the tenant id from the context I assume this module has to come after the authentication in the pipeline. Yes, I have made that assumption. It seems reasonable, given that the existing rate-limit middleware is right after authentication as well. Have you thought about using the tenant_id in the URL instead ? (I'm thinking of the case where you want rate limit requests into the authentication system as well as Nova itself). No, I haven't. I don't trust the user, which is where the tenant_id in the URL comes from. I do trust the auth system, which is why I want to use the tenant ID from the context. (And yes, you could argue that authz would prevent their access to other tenants anyway, but why make nova have to check authz if rate limiting would stop them in the first place?) As for rate limiting requests into the authentication system, I'd suggest using a Limit subclass which uses the remote IP address in place of a tenant ID, at least for the user endpoint. I don't think we want any rate limiting at all on the service side of Keystone; our current architecture means that Keystone is going to be hit a *lot*: at least once for each request that hits Nova, and more in certain cases (i.e., instance boot, where we'll have to hit quantum and glance as well). - Does this work for EC2 as well as OSAPI ? Actually, it didn't occur to me to test, given that I don't really use the EC2 API. I don't think there's anything in the basic architecture which would be incompatible with EC2; the only possible sticking point that occurs to me is the URL construction in nova_limits:NovaClassLimit.route(): if the URL specified is prefixed with '/v1.1/' or '/v2/', the version identifier is dropped (otherwise the route wouldn't match). That would be easy to work around; simply extend NovaClassLimit and override route() to do the appropriate transformation for EC2. Any EC2 experts want to weigh in? -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Distributed rate-limiting
My issue with using the URL is someone could easily DoS any tenant. Maybe you said that below. I only have a brief moment to scan email ATM. :) On Mar 29, 2012, at 5:26 PM, Kevin L. Mitchell kevin.mitch...@rackspace.com wrote: On Thu, 2012-03-29 at 22:58 +0100, Day, Phil wrote: - As you get the tenant id from the context I assume this module has to come after the authentication in the pipeline. Yes, I have made that assumption. It seems reasonable, given that the existing rate-limit middleware is right after authentication as well. Have you thought about using the tenant_id in the URL instead ? (I'm thinking of the case where you want rate limit requests into the authentication system as well as Nova itself). No, I haven't. I don't trust the user, which is where the tenant_id in the URL comes from. I do trust the auth system, which is why I want to use the tenant ID from the context. (And yes, you could argue that authz would prevent their access to other tenants anyway, but why make nova have to check authz if rate limiting would stop them in the first place?) As for rate limiting requests into the authentication system, I'd suggest using a Limit subclass which uses the remote IP address in place of a tenant ID, at least for the user endpoint. I don't think we want any rate limiting at all on the service side of Keystone; our current architecture means that Keystone is going to be hit a *lot*: at least once for each request that hits Nova, and more in certain cases (i.e., instance boot, where we'll have to hit quantum and glance as well). - Does this work for EC2 as well as OSAPI ? Actually, it didn't occur to me to test, given that I don't really use the EC2 API. I don't think there's anything in the basic architecture which would be incompatible with EC2; the only possible sticking point that occurs to me is the URL construction in nova_limits:NovaClassLimit.route(): if the URL specified is prefixed with '/v1.1/' or '/v2/', the version identifier is dropped (otherwise the route wouldn't match). That would be easy to work around; simply extend NovaClassLimit and override route() to do the appropriate transformation for EC2. Any EC2 experts want to weigh in? -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Distributed rate-limiting
On Sat, 2012-03-17 at 12:31 -0400, Jay Pipes wrote: Kevin, you've really impressed me. Well documented, well thought-out code. Yeah, well…I got into the habit of documenting my code well when I wrote a very large project and discovered I was forgetting how to use its pieces :) I hope you won't mind if I contribute a REST-ful interface for configuration management and status reporting? Not at all; I designed the limit classes—specifically the limit attributes—to be easily introspectable for exactly that sort of purpose. -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Distributed rate-limiting
I was just thinking about why redis was chosen over memcache.. Ignoring any capability differences, service and technology sprawl was the first thing that crossed my mind. Is redis providing more than the, more then likely preexisting, memcache and rabbitmq combination? Thanks, Kiall Sent from my phone. On Mar 17, 2012 4:58 p.m., John Dickinson m...@not.mn wrote: If you're looking for wsgi rate limiting, take a look at swift's rate limiting middleware. It's distributed, simple (uses memcached to keep track of what's going on), supports multiple tiers of rate limiting (with interpolation between them), and white/black lists. While swift's rate limiting is tuned for swift, its technique could be pretty simply adapted to other applications. Swift's rate limiting is swift-specific in what it checks to rate limit on. Those checks should be able to be adapted to other use cases. source: https://github.com/openstack/swift/blob/master/swift/common/middleware/ratelimit.py docs: https://github.com/openstack/swift/blob/master/doc/source/ratelimit.rst sample config: https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L118 --John On Mar 16, 2012, at 4:45 PM, Kevin L. Mitchell wrote: Howdy, folks. I've been working on a replacement for nova's rate-limiting middleware that will handle the multiple-node case, and I've developed a fairly generic rate-limiting package, along with a second package that adapts it to nova. (This means you could also use this rate-limiting setup with, say, glance, or with any other project that uses Python middleware.) Here is some information: * Turnstile Turnstile is a piece of WSGI middleware that performs true distributed rate-limiting. System administrators can run an API on multiple nodes, then place this middleware in the pipeline prior to the application. Turnstile uses a Redis database to track the rate at which users are hitting the API, and can then apply configured rate limits, even if each request was made against a different API node. - https://github.com/klmitch/turnstile - http://pypi.python.org/pypi/turnstile * nova_limits This package provides the ``nova_limits`` Python module, which contains the ``nova_preprocess()`` preprocessor, the ``NovaClassLimit`` limit class, and the ``NovaTurnstileMiddleware`` replacement middleware class, all for use with Turnstile. These pieces work together to provide class-based rate limiting integration with nova. - https://github.com/klmitch/nova_limits - http://pypi.python.org/pypi/nova_limits Both packages should be fairly well documented (start with README.rst), and please feel free to log issues or make pull requests. -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Distributed rate-limiting
Howdy, folks. I've been working on a replacement for nova's rate-limiting middleware that will handle the multiple-node case, and I've developed a fairly generic rate-limiting package, along with a second package that adapts it to nova. (This means you could also use this rate-limiting setup with, say, glance, or with any other project that uses Python middleware.) Here is some information: * Turnstile Turnstile is a piece of WSGI middleware that performs true distributed rate-limiting. System administrators can run an API on multiple nodes, then place this middleware in the pipeline prior to the application. Turnstile uses a Redis database to track the rate at which users are hitting the API, and can then apply configured rate limits, even if each request was made against a different API node. - https://github.com/klmitch/turnstile - http://pypi.python.org/pypi/turnstile * nova_limits This package provides the ``nova_limits`` Python module, which contains the ``nova_preprocess()`` preprocessor, the ``NovaClassLimit`` limit class, and the ``NovaTurnstileMiddleware`` replacement middleware class, all for use with Turnstile. These pieces work together to provide class-based rate limiting integration with nova. - https://github.com/klmitch/nova_limits - http://pypi.python.org/pypi/nova_limits Both packages should be fairly well documented (start with README.rst), and please feel free to log issues or make pull requests. -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
