Re: [Openstack] Question on wsgi rate limiting
You make some good points about what is the appropriate level in the stack to do rate shaping, but I would just like to have a configurable, manageable and monitorable ratelimit/quota solution that doesn't seem like a giant hack :) Baby steps. -jay On 03/30/2012 01:23 PM, Caitlin Bestler wrote: Throughout the discussion on distributed rate limiting I’ve had the annoying feeling that I’ve heard this joke before. Basically, are we looking for our keys under the street lamp because the light is good rather than looking for them where they were lost? Has anyone studied the effectiveness of rate limitations implemented at this layer. From my experience with rate shaping discussions in IEEE 802.1 Data Center Bridging group I am concerned that the response time working this far up the stack will preclude effective rate shaping. Of course, if someone has studied this and shown it to be effective then this would be great news. The light is Better under the street lamp. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Question on wsgi rate limiting
Caitlin, I'm curious what were the use cases and concerns in DCB? If my memory serves me right (from rate limiting at L2 level) the main issues are guaranteeing QoS, effective bandwidth usage, fair allocation of memory buffer space. All of those goals damaged pretty badly if congestion occurs and any of the resources are exhausted (leading to indiscriminate packet loss, for lack of any other recourse). The use cases, at least the way as I conceive them for OS are very different. They're not intended to resolve resource constraints as a primary goal (though, that's definitely a secondary goal). As an example - issuing lots of Nova API calls is by itself not the problem - executing the downstream effects of what those requests trigger is whats being protected (access to DB, spawning VM's, Rabbit MQ message rates etc). For Swift, where pure bandwidth is a primary concern, and the primary resource being consumed - I imagine you're right. Some L2/L3 traffic shaping (and monitoring) would be advisable - but that's not to say that's the only resource. e.g. creating and deleting containers repeatedly will consume relatively little bandwidth, but will exert quite a lot of resource consumption on the back end. Rate limiting these API calls is probably prudent at the API layer. On Fri, Mar 30, 2012 at 1:56 PM, Jay Pipes jaypi...@gmail.com wrote: You make some good points about what is the appropriate level in the stack to do rate shaping, but I would just like to have a configurable, manageable and monitorable ratelimit/quota solution that doesn't seem like a giant hack :) Baby steps. -jay On 03/30/2012 01:23 PM, Caitlin Bestler wrote: Throughout the discussion on distributed rate limiting I’ve had the annoying feeling that I’ve heard this joke before. Basically, are we looking for our keys under the street lamp because the light is good rather than looking for them where they were lost? Has anyone studied the effectiveness of rate limitations implemented at this layer. From my experience with rate shaping discussions in IEEE 802.1 Data Center Bridging group I am concerned that the response time working this far up the stack will preclude effective rate shaping. Of course, if someone has studied this and shown it to be effective then this would be great news. The light is Better under the street lamp. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Question on wsgi rate limiting
Caitlin Replies inline /Caitlin -Original Message- From: andi abes [mailto:andi.a...@gmail.com] Sent: Friday, March 30, 2012 11:32 AM To: Jay Pipes; Caitlin Bestler Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Question on wsgi rate limiting Caitlin, I'm curious what were the use cases and concerns in DCB? If my memory serves me right (from rate limiting at L2 level) the main issues are guaranteeing QoS, effective bandwidth usage, fair allocation of memory buffer space. All of those goals damaged pretty badly if congestion occurs and any of the resources are exhausted (leading to indiscriminate packet loss, for lack of any other recourse). Caitlin Correct. The fundamental goal was to allow storage-oriented classes of service which could be effectively guaranteed to be drop-free within a Datacenter. FCoE as a specific application needed this to be a very strong guarantee, but congestion drops triggering TCP back-offs is generally not a good thing for storage traffic. And once you cut through the match, this is ultimately about good algorithms that can be implemented in a distributed fashion that allocate the buffering capacity Of the network elements somewhat intelligently and robustly. /Caitlin The use cases, at least the way as I conceive them for OS are very different. They're not intended to resolve resource constraints as a primary goal (though, that's definitely a secondary goal). As an example - issuing lots of Nova API calls is by itself not the problem - executing the downstream effects of what those requests trigger is whats being protected (access to DB, spawning VM's, Rabbit MQ message rates etc). For Swift, where pure bandwidth is a primary concern, and the primary resource being consumed - I imagine you're right. Some L2/L3 traffic shaping (and monitoring) would be advisable - but that's not to say that's the only resource. e.g. creating and deleting containers repeatedly will consume relatively little bandwidth, but will exert quite a lot of resource consumption on the back end. Rate limiting these API calls is probably prudent at the API layer. Caitlin Yes, Nova APIs calls are very unlikely to cause network congestion. Bulk payload transfers (whether Swift or Nova Volumes) is an issue. My concern is that a truly effective solution will have to be at the Quantum level, not wsgi. /Caitlin On Fri, Mar 30, 2012 at 1:56 PM, Jay Pipes jaypi...@gmail.com wrote: You make some good points about what is the appropriate level in the stack to do rate shaping, but I would just like to have a configurable, manageable and monitorable ratelimit/quota solution that doesn't seem like a giant hack :) Baby steps. -jay On 03/30/2012 01:23 PM, Caitlin Bestler wrote: Throughout the discussion on distributed rate limiting I've had the annoying feeling that I've heard this joke before. Basically, are we looking for our keys under the street lamp because the light is good rather than looking for them where they were lost? Has anyone studied the effectiveness of rate limitations implemented at this layer. From my experience with rate shaping discussions in IEEE 802.1 Data Center Bridging group I am concerned that the response time working this far up the stack will preclude effective rate shaping. Of course, if someone has studied this and shown it to be effective then this would be great news. The light is Better under the street lamp. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Question on wsgi rate limiting
Caitlin, alas, you missed my point. The intent of rate limiting API calls in OS goes well beyond limiting network traffic. It's intended to ensure that no one tenant/user consumes unduly high resources throughout the system. These resources are not just network bandwidth, but the myriad of resources involved in providing the service (DB access, replication activity (both CPU,mem and bandwidth). These have little to do with networking or quantum (though having quantum provide an API for bandwidth management would be cool). You might want to think of this functionality as API-Quota - rather than the traditional bandwidth only rate limits. On Fri, Mar 30, 2012 at 2:40 PM, Caitlin Bestler caitlin.best...@nexenta.com wrote: Caitlin Replies inline /Caitlin -Original Message- From: andi abes [mailto:andi.a...@gmail.com] Sent: Friday, March 30, 2012 11:32 AM To: Jay Pipes; Caitlin Bestler Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Question on wsgi rate limiting Caitlin, I'm curious what were the use cases and concerns in DCB? If my memory serves me right (from rate limiting at L2 level) the main issues are guaranteeing QoS, effective bandwidth usage, fair allocation of memory buffer space. All of those goals damaged pretty badly if congestion occurs and any of the resources are exhausted (leading to indiscriminate packet loss, for lack of any other recourse). Caitlin Correct. The fundamental goal was to allow storage-oriented classes of service which could be effectively guaranteed to be drop-free within a Datacenter. FCoE as a specific application needed this to be a very strong guarantee, but congestion drops triggering TCP back-offs is generally not a good thing for storage traffic. And once you cut through the match, this is ultimately about good algorithms that can be implemented in a distributed fashion that allocate the buffering capacity Of the network elements somewhat intelligently and robustly. /Caitlin The use cases, at least the way as I conceive them for OS are very different. They're not intended to resolve resource constraints as a primary goal (though, that's definitely a secondary goal). As an example - issuing lots of Nova API calls is by itself not the problem - executing the downstream effects of what those requests trigger is whats being protected (access to DB, spawning VM's, Rabbit MQ message rates etc). For Swift, where pure bandwidth is a primary concern, and the primary resource being consumed - I imagine you're right. Some L2/L3 traffic shaping (and monitoring) would be advisable - but that's not to say that's the only resource. e.g. creating and deleting containers repeatedly will consume relatively little bandwidth, but will exert quite a lot of resource consumption on the back end. Rate limiting these API calls is probably prudent at the API layer. Caitlin Yes, Nova APIs calls are very unlikely to cause network congestion. Bulk payload transfers (whether Swift or Nova Volumes) is an issue. My concern is that a truly effective solution will have to be at the Quantum level, not wsgi. /Caitlin On Fri, Mar 30, 2012 at 1:56 PM, Jay Pipes jaypi...@gmail.com wrote: You make some good points about what is the appropriate level in the stack to do rate shaping, but I would just like to have a configurable, manageable and monitorable ratelimit/quota solution that doesn't seem like a giant hack :) Baby steps. -jay On 03/30/2012 01:23 PM, Caitlin Bestler wrote: Throughout the discussion on distributed rate limiting I've had the annoying feeling that I've heard this joke before. Basically, are we looking for our keys under the street lamp because the light is good rather than looking for them where they were lost? Has anyone studied the effectiveness of rate limitations implemented at this layer. From my experience with rate shaping discussions in IEEE 802.1 Data Center Bridging group I am concerned that the response time working this far up the stack will preclude effective rate shaping. Of course, if someone has studied this and shown it to be effective then this would be great news. The light is Better under the street lamp. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Question on wsgi rate limiting
-Original Message- From: andi abes [mailto:andi.a...@gmail.com] Sent: Friday, March 30, 2012 12:12 PM To: Caitlin Bestler Cc: Jay Pipes; openstack@lists.launchpad.net Subject: Re: [Openstack] Question on wsgi rate limiting Caitlin, alas, you missed my point. The intent of rate limiting API calls in OS goes well beyond limiting network traffic. It's intended to ensure that no one tenant/user consumes unduly high resources throughout the system. These resources are not just network bandwidth, but the myriad of resources involved in providing the service (DB access, replication activity (both CPU,mem and bandwidth). These have little to do with networking or quantum (though having quantum provide an API for bandwidth management would be cool). You might want to think of this functionality as API-Quota - rather than the traditional bandwidth only rate limits. - Yes, that label does make the difference in goals clearer. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Question on wsgi rate limiting
Just FYI- swift's ratelimiting doesn't limit on bandwidth. Just requests per second to specific resources. David On Mar 30, 2012, at 1:40 PM, Caitlin Bestler wrote: Caitlin Replies inline /Caitlin -Original Message- From: andi abes [mailto:andi.a...@gmail.com] Sent: Friday, March 30, 2012 11:32 AM To: Jay Pipes; Caitlin Bestler Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Question on wsgi rate limiting Caitlin, I'm curious what were the use cases and concerns in DCB? If my memory serves me right (from rate limiting at L2 level) the main issues are guaranteeing QoS, effective bandwidth usage, fair allocation of memory buffer space. All of those goals damaged pretty badly if congestion occurs and any of the resources are exhausted (leading to indiscriminate packet loss, for lack of any other recourse). Caitlin Correct. The fundamental goal was to allow storage-oriented classes of service which could be effectively guaranteed to be drop-free within a Datacenter. FCoE as a specific application needed this to be a very strong guarantee, but congestion drops triggering TCP back-offs is generally not a good thing for storage traffic. And once you cut through the match, this is ultimately about good algorithms that can be implemented in a distributed fashion that allocate the buffering capacity Of the network elements somewhat intelligently and robustly. /Caitlin The use cases, at least the way as I conceive them for OS are very different. They're not intended to resolve resource constraints as a primary goal (though, that's definitely a secondary goal). As an example - issuing lots of Nova API calls is by itself not the problem - executing the downstream effects of what those requests trigger is whats being protected (access to DB, spawning VM's, Rabbit MQ message rates etc). For Swift, where pure bandwidth is a primary concern, and the primary resource being consumed - I imagine you're right. Some L2/L3 traffic shaping (and monitoring) would be advisable - but that's not to say that's the only resource. e.g. creating and deleting containers repeatedly will consume relatively little bandwidth, but will exert quite a lot of resource consumption on the back end. Rate limiting these API calls is probably prudent at the API layer. Caitlin Yes, Nova APIs calls are very unlikely to cause network congestion. Bulk payload transfers (whether Swift or Nova Volumes) is an issue. My concern is that a truly effective solution will have to be at the Quantum level, not wsgi. /Caitlin On Fri, Mar 30, 2012 at 1:56 PM, Jay Pipes jaypi...@gmail.com wrote: You make some good points about what is the appropriate level in the stack to do rate shaping, but I would just like to have a configurable, manageable and monitorable ratelimit/quota solution that doesn't seem like a giant hack :) Baby steps. -jay On 03/30/2012 01:23 PM, Caitlin Bestler wrote: Throughout the discussion on distributed rate limiting I've had the annoying feeling that I've heard this joke before. Basically, are we looking for our keys under the street lamp because the light is good rather than looking for them where they were lost? Has anyone studied the effectiveness of rate limitations implemented at this layer. From my experience with rate shaping discussions in IEEE 802.1 Data Center Bridging group I am concerned that the response time working this far up the stack will preclude effective rate shaping. Of course, if someone has studied this and shown it to be effective then this would be great news. The light is Better under the street lamp. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
