Re: Feature: quota control
Hi Guys, I gave this some thought. Why don't we build a system close to sendmail's milter system. An API where by clients can plug in and offer services - one can be a traffic counter, traffic limiter (as what we proposing here) and maybe a URL block, a virus scanner etc. I know that there are re-directors that do this kind of thing, but surely squid has reached a stage now where it's big enough to expand this way ? Sendmail had to end up doing it ? It might even give a solution to the single thread disk problem since you could have your disk stores register in the same way. I am just thowing ideas around - maybe they should be targeted for Squid 4, who knows :) What you guys think ? Cheers, Pieter - Original Message - From: Adrian Chadd adr...@squid-cache.org To: Amos Jeffries squ...@treenet.co.nz Cc: Robert Collins robe...@squid-cache.org; Mark Nottingham m...@yahoo-inc.com; Pieter De Wit pie...@insync.za.net; squid-dev@squid-cache.org Sent: Saturday, March 28, 2009 5:31 PM Subject: Re: Feature: quota control Just to add to this - implementing it as a delay pool inside Squid flattens traffic into one byte pool. Various places may not do this at all - there may be free versus non-free (which means one set of ACLs inside Squid); there may be cheap versus expensive (again, possibly requiring multiple delay pools and multiple ACLs to map it all together; again all inside Squid) - things get very messy, very quickly. This is why my proposal (which I hope -finally- gets approved so I can begin work on it ASAP! :) involves passing off the traffic assignment to an external daemon that implements -all- of the traffic assignment and accounting logic. Squid will then just send requests for traffic and interim updates like you've said. 2c, Adrian 2009/2/26 Amos Jeffries squ...@treenet.co.nz: Robert Collins wrote: On Fri, 2009-02-27 at 10:00 +1100, Mark Nottingham wrote: Honestly, if I wanted to do byte-based quotas today, I'd have an external ACL helper talking to an external logging helper; that way, you can just log the response sizes to a daemon and then another daemon would use that information to make a decision at access time. The only even mildly hard part about this is sharing state between the daemons, but if you don't need the decisions to be real-time, it's not that bad (especially considering that in any serious deployment, you'll have state issues between multiple boxes anyway). Sure; I think that would fit with 'ensuring enough hooks' :P -Rob The brief description of what I gave Pieter to start with was: A pool based on DelayPools in that Squid decrements live as traffic goes through. With a helper/ACL hook to retrieve the initial pool size and to call as needed to check for current quotas. How the helper operates is not relevant to Squid. Thats important. The key things being that; its always called for new visitors to assign the start quota, and when the quota is nearing empty its called again to see if they get more. Helper would need to send back UNITS AMOUNT MINIMUM where UNITS is the unit of quota (seconds, bytes, requests, misses?, other?), AMOUNT being a integer count of units the client is allowed to use, and MINIMUM is the level of units where the helper is to be asked for an update. 0 remaining units results in an Error page 'quota exceeded' or somesuch. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE13 Current Beta Squid 3.1.0.5
Re: Feature: quota control
On 03/30/2009 03:16 AM, Pieter De Wit wrote: I gave this some thought. Why don't we build a system close to sendmail's milter system. An API where by clients can plug in and offer services - one can be a traffic counter, traffic limiter (as what we proposing here) and maybe a URL block, a virus scanner etc. We already have such a plugin API for message- and content-related tasks. It is called eCAP, and it has been mentioned on this thread. Compared to eCAP, traffic shaping and quotas are different in scope as they work on multiple messages and often do not care about the messages at all (just raw bytes traffic). So, we have a few options: 1) Use eCAP nearly as is for traffic shaping and quotas, even though it is not a perfect fit for the task. 2) Significantly enhance eCAP to offer traffic shaping-specific hooks (as a standard addition or as a Squid extension), even though it may lead to eCAP API bloat. 3) Develop a different plugin API that specializes in aggregate traffic management and is unrelated to eCAP, even though it may lead to duplicating a lot of eCAP-related code. I am not sure, but I think option #2 is the worst. What do you think? Cheers, Alex. - Original Message - From: Adrian Chadd adr...@squid-cache.org To: Amos Jeffries squ...@treenet.co.nz Cc: Robert Collins robe...@squid-cache.org; Mark Nottingham m...@yahoo-inc.com; Pieter De Wit pie...@insync.za.net; squid-dev@squid-cache.org Sent: Saturday, March 28, 2009 5:31 PM Subject: Re: Feature: quota control Just to add to this - implementing it as a delay pool inside Squid flattens traffic into one byte pool. Various places may not do this at all - there may be free versus non-free (which means one set of ACLs inside Squid); there may be cheap versus expensive (again, possibly requiring multiple delay pools and multiple ACLs to map it all together; again all inside Squid) - things get very messy, very quickly. This is why my proposal (which I hope -finally- gets approved so I can begin work on it ASAP! :) involves passing off the traffic assignment to an external daemon that implements -all- of the traffic assignment and accounting logic. Squid will then just send requests for traffic and interim updates like you've said. 2c, Adrian 2009/2/26 Amos Jeffries squ...@treenet.co.nz: Robert Collins wrote: On Fri, 2009-02-27 at 10:00 +1100, Mark Nottingham wrote: Honestly, if I wanted to do byte-based quotas today, I'd have an external ACL helper talking to an external logging helper; that way, you can just log the response sizes to a daemon and then another daemon would use that information to make a decision at access time. The only even mildly hard part about this is sharing state between the daemons, but if you don't need the decisions to be real-time, it's not that bad (especially considering that in any serious deployment, you'll have state issues between multiple boxes anyway). Sure; I think that would fit with 'ensuring enough hooks' :P -Rob The brief description of what I gave Pieter to start with was: A pool based on DelayPools in that Squid decrements live as traffic goes through. With a helper/ACL hook to retrieve the initial pool size and to call as needed to check for current quotas. How the helper operates is not relevant to Squid. Thats important. The key things being that; its always called for new visitors to assign the start quota, and when the quota is nearing empty its called again to see if they get more. Helper would need to send back UNITS AMOUNT MINIMUM where UNITS is the unit of quota (seconds, bytes, requests, misses?, other?), AMOUNT being a integer count of units the client is allowed to use, and MINIMUM is the level of units where the helper is to be asked for an update. 0 remaining units results in an Error page 'quota exceeded' or somesuch. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE13 Current Beta Squid 3.1.0.5
Re: Feature: quota control
On Mon, 30 Mar 2009 11:38:31 -0600, Alex Rousskov rouss...@measurement-factory.com wrote: On 03/30/2009 03:16 AM, Pieter De Wit wrote: I gave this some thought. Why don't we build a system close to sendmail's milter system. An API where by clients can plug in and offer services - one can be a traffic counter, traffic limiter (as what we proposing here) and maybe a URL block, a virus scanner etc. We already have such a plugin API for message- and content-related tasks. It is called eCAP, and it has been mentioned on this thread. Compared to eCAP, traffic shaping and quotas are different in scope as they work on multiple messages and often do not care about the messages at all (just raw bytes traffic). So, we have a few options: 1) Use eCAP nearly as is for traffic shaping and quotas, even though it is not a perfect fit for the task. 2) Significantly enhance eCAP to offer traffic shaping-specific hooks (as a standard addition or as a Squid extension), even though it may lead to eCAP API bloat. 3) Develop a different plugin API that specializes in aggregate traffic management and is unrelated to eCAP, even though it may lead to duplicating a lot of eCAP-related code. I am not sure, but I think option #2 is the worst. What do you think? Cheers, Alex. Hey Alex and List, Sorry that I didn't read up on eCAP before starting this process, but I believe that it won't be on the TODO list if eCAP does it right. What I saw in my mind is something like this : Client/Bandwidth limiting registers with squid ... squid get a request to download object x from site y by user z at ip a.b.c.d etc squid sends that request to the client (just text not the object) client replies yes or no - if yes the client needs to track how much data is left for that user etc. (Client here is the limiting software - couldn't think of a better name...coffee) I feel this should be a persistent connection that is matched by an acl for speed etc. Now to the workings of this. I am leaning towards option 3. I am not sure how hard it is to maintain two API's. To be honest, I am not even sure how hard it is to maintain one API. I feel that this really only needs two or three commands, so is it really going to bloat the API ? Broken down - what is the most this API would need ? Cheers, Pieter snip - I am sure they can find the follow up in the archive :)
Re: Feature: quota control
On 03/30/2009 01:00 PM, Pieter De Wit wrote: On Mon, 30 Mar 2009 11:38:31 -0600, Alex Rousskov rouss...@measurement-factory.com wrote: On 03/30/2009 03:16 AM, Pieter De Wit wrote: I gave this some thought. Why don't we build a system close to sendmail's milter system. An API where by clients can plug in and offer services - one can be a traffic counter, traffic limiter (as what we proposing here) and maybe a URL block, a virus scanner etc. We already have such a plugin API for message- and content-related tasks. It is called eCAP, and it has been mentioned on this thread. Compared to eCAP, traffic shaping and quotas are different in scope as they work on multiple messages and often do not care about the messages at all (just raw bytes traffic). So, we have a few options: 1) Use eCAP nearly as is for traffic shaping and quotas, even though it is not a perfect fit for the task. 2) Significantly enhance eCAP to offer traffic shaping-specific hooks (as a standard addition or as a Squid extension), even though it may lead to eCAP API bloat. 3) Develop a different plugin API that specializes in aggregate traffic management and is unrelated to eCAP, even though it may lead to duplicating a lot of eCAP-related code. Client/Bandwidth limiting registers with squid ... squid get a request to download object x from site y by user z at ip a.b.c.d etc squid sends that request to the client (just text not the object) client replies yes or no - if yes the client needs to track how much data is left for that user etc. (Client here is the limiting software - couldn't think of a better name...coffee) I feel this should be a persistent connection that is matched by an acl for speed etc. I doubt the traffic shaping API can be that simple and still cover most useful cases: the size of the object is not always known a priori; some traffic shaping decisions need more than user ID, client IP, and a URL; there are several points at which traffic shaping may take place; etc. I also suspect it would be better to avoid mandating a connection-based interface for this API. A loadable module can use a connection-based approach to communicate with the actual traffic shaper, but the reverse is not possible: once you standardize on the connection-based API, you cannot do embedded modules (which are usually much more efficient). We probably would not invent eCAP if ICAP (which uses TCP connections) and various helpers were good enough from that point of view. Now to the workings of this. I am leaning towards option 3. I am not sure how hard it is to maintain two API's. To be honest, I am not even sure how hard it is to maintain one API. I feel that this really only needs two or three commands, so is it really going to bloat the API ? You need three commands. The person next to you needs another five. By the end of the day, you end up with a few dozen commands OR several distinct APIs. Just look at how many different helpers Squid already has and how easy/difficult it is to extend them. However, when I was talking about API bloat, I was not talking about the number of commands in the new API. I was talking about bloating the existing API (eCAP) with traffic shaping commands. Broken down - what is the most this API would need ? We probably should define the scope before counting the commands: Inbound, outbound, and peer traffic shaping. Shaping based on group membership, user credentials, IP addresses, TCP connection IDs, MIME types, HTTP headers. Shaping during message download. Reconfiguration of quotas. Etc., etc. Cheers, Alex.
Re: Feature: quota control
snip Ok - so here is how I see this : * We need to agree (maybe a strong word) on a traffic limiting standard. (Think how the limits are implemented, rather than done - if the object size is known, pass that to the limiter, if not, everytime data is RX'ed using something like a session id or so) If this has been done - sorry - and point me to it ? :) Does this justify public input ? I am pretty sure the squid dev's use squid more than the avg. Joe * How do these tasks talk to each other ? Are they modules etc ? Maybe we should open a wiki page with some proposals ?
Re: Feature: quota control
Pieter De Wit wrote: snip Ok - so here is how I see this : * We need to agree (maybe a strong word) on a traffic limiting standard. (Think how the limits are implemented, rather than done - if the object size is known, pass that to the limiter, if not, everytime data is RX'ed using something like a session id or so) If this has been done - sorry - and point me to it ? :) Does this justify public input ? I am pretty sure the squid dev's use squid more than the avg. Joe * How do these tasks talk to each other ? Are they modules etc ? Maybe we should open a wiki page with some proposals ? wiki is already open about this: http://wiki.squid-cache.org/Features/Quota any and all planning towards a solution can go in. Either to the main page or as discussions sub-context. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE13 Current Beta Squid 3.1.0.6
Re: Feature: quota control
I'm looking at implementing this as part of a contract for squid-2. I was going to take a different approach - that is, i'm not going to implement quota control or management in squid; I'm going to provide the hooks to squid to allow external controls to handle the quota. adrian 2009/2/21 Pieter De Wit pie...@insync.za.net: Hi Guys, I would like to offer my time in working on this feature - I have not done any squid dev, but since I would like to see this feature in Squid, I thought I would take it on. I have briefly contacted Amos off list and we agreed that there is no set in stone way of doing this. I would like to propose that we then start throwing around some ideas and let's see if we can get this into squid :) Some ideas that Amos quickly said : - Based on delay pools - Use of external helpers to track traffic The way I see this happening is that a Quota is like a pool that empties based on 2 classes - bytes and requests. Requests will be for things like the number of requests, i.e. a person is only allowed to download 5 exe's per day or 5 requests of 1meg or something like that (it just popped into my head :) ) Bytes is a pretty straight forward one, the user is only allowed x amount of bytes per y amount of time. Anyways - let the ideas fly :) Cheers, Pieter
Re: Feature: quota control
Isn't requests really just an external acl helper? On 27/02/2009, at 8:36 AM, Adrian Chadd wrote: I'm looking at implementing this as part of a contract for squid-2. I was going to take a different approach - that is, i'm not going to implement quota control or management in squid; I'm going to provide the hooks to squid to allow external controls to handle the quota. adrian 2009/2/21 Pieter De Wit pie...@insync.za.net: Hi Guys, I would like to offer my time in working on this feature - I have not done any squid dev, but since I would like to see this feature in Squid, I thought I would take it on. I have briefly contacted Amos off list and we agreed that there is no set in stone way of doing this. I would like to propose that we then start throwing around some ideas and let's see if we can get this into squid :) Some ideas that Amos quickly said : - Based on delay pools - Use of external helpers to track traffic The way I see this happening is that a Quota is like a pool that empties based on 2 classes - bytes and requests. Requests will be for things like the number of requests, i.e. a person is only allowed to download 5 exe's per day or 5 requests of 1meg or something like that (it just popped into my head :) ) Bytes is a pretty straight forward one, the user is only allowed x amount of bytes per y amount of time. Anyways - let the ideas fly :) Cheers, Pieter -- Mark Nottingham m...@yahoo-inc.com
Re: Feature: quota control
Key things I'd look for: - quota's shouldn't be lost when squid restarts - people should be able to use external quota systems (they may have e.g. netflow or other systems tracking direct b/w use, and squid is pulling from those allowances when downloads are caused by a given user). Both of which are nicely solved by Adrians suggestion of making sure there are appropriate hooks in squid to let other software actually do quotas. It would be nice to ship a default client for those hooks that does something simple (like monthly quotas with a simple script to report on current totals/reset). But even that is probably too much in core - it should probably be a parallel project, like the reference icap client is. (And isn't iCap enough to do quotas?, or is it too heavyweight? Perhaps look at the in-squid iCap-alike Alex has been working on?) -Rob signature.asc Description: This is a digitally signed message part
Re: Feature: quota control
Honestly, if I wanted to do byte-based quotas today, I'd have an external ACL helper talking to an external logging helper; that way, you can just log the response sizes to a daemon and then another daemon would use that information to make a decision at access time. The only even mildly hard part about this is sharing state between the daemons, but if you don't need the decisions to be real-time, it's not that bad (especially considering that in any serious deployment, you'll have state issues between multiple boxes anyway). Squid modifications that would help this (and similar use cases): 1) Allow multiple, different external loggers to be defined and used. 2) Harmonise the interfaces/configuration of *all* external helpers, so that you can pass arbitrary args to each, using the same syntax. I'm looking at you, redirectors. 3) Reduce the overhead of using external helpers, if possible. A *lot* of the customisation that I do for Squid is based on uses of helpers like this; they're actually very powerful and flexible, if you know what you're doing with them. I would very much like to see Squid turn them into more of a strength... On 27/02/2009, at 9:30 AM, Robert Collins wrote: Key things I'd look for: - quota's shouldn't be lost when squid restarts - people should be able to use external quota systems (they may have e.g. netflow or other systems tracking direct b/w use, and squid is pulling from those allowances when downloads are caused by a given user). Both of which are nicely solved by Adrians suggestion of making sure there are appropriate hooks in squid to let other software actually do quotas. It would be nice to ship a default client for those hooks that does something simple (like monthly quotas with a simple script to report on current totals/reset). But even that is probably too much in core - it should probably be a parallel project, like the reference icap client is. (And isn't iCap enough to do quotas?, or is it too heavyweight? Perhaps look at the in-squid iCap-alike Alex has been working on?) -Rob -- Mark Nottingham m...@yahoo-inc.com
Re: Feature: quota control
I was talking about request number quotas (e.g., you can make n requests in m minutes). Regarding byte quotes -- see subsequent message -- I disagree that you need eCAP :) On 27/02/2009, at 10:00 AM, Kinkie wrote: On Thu, Feb 26, 2009 at 11:23 PM, Mark Nottingham m...@yahoo- inc.com wrote: Isn't requests really just an external acl helper? Not really.. an external ACL helper would need to do real-time parsing of the logs to really know how much each client downloaded, as AFAIK both request and reply acl's are evaluated BEFORE the actual transfer takes place. Only eCAP probably has the right hooks. -- /kinkie -- Mark Nottingham m...@yahoo-inc.com
Re: Feature: quota control
On 02/26/2009 04:04 PM, Mark Nottingham wrote: I was talking about request number quotas (e.g., you can make n requests in m minutes). Regarding byte quotes -- see subsequent message -- I disagree that you need eCAP :) eCAP is one way of doing byte- and/or count- quotas, but there are, of course, other approaches as well. I am still not very excited about the slow bloat of redirectorsCo API towards duplication of *CAP functionality, but perhaps it is inevitable. If it is, a clean unified interface among all such one-line external helpers would make that duplication a lot less painful. Cheers, Alex. On 27/02/2009, at 10:00 AM, Kinkie wrote: On Thu, Feb 26, 2009 at 11:23 PM, Mark Nottingham m...@yahoo-inc.com wrote: Isn't requests really just an external acl helper? Not really.. an external ACL helper would need to do real-time parsing of the logs to really know how much each client downloaded, as AFAIK both request and reply acl's are evaluated BEFORE the actual transfer takes place. Only eCAP probably has the right hooks. -- /kinkie -- Mark Nottingham m...@yahoo-inc.com
Re: Feature: quota control
On Thu, Feb 26, 2009 at 11:23 PM, Mark Nottingham m...@yahoo-inc.com wrote: Isn't requests really just an external acl helper? Not really.. an external ACL helper would need to do real-time parsing of the logs to really know how much each client downloaded, as AFAIK both request and reply acl's are evaluated BEFORE the actual transfer takes place. Only eCAP probably has the right hooks. -- /kinkie
Re: Feature: quota control
Robert Collins wrote: On Fri, 2009-02-27 at 10:00 +1100, Mark Nottingham wrote: Honestly, if I wanted to do byte-based quotas today, I'd have an external ACL helper talking to an external logging helper; that way, you can just log the response sizes to a daemon and then another daemon would use that information to make a decision at access time. The only even mildly hard part about this is sharing state between the daemons, but if you don't need the decisions to be real-time, it's not that bad (especially considering that in any serious deployment, you'll have state issues between multiple boxes anyway). Sure; I think that would fit with 'ensuring enough hooks' :P -Rob The brief description of what I gave Pieter to start with was: A pool based on DelayPools in that Squid decrements live as traffic goes through. With a helper/ACL hook to retrieve the initial pool size and to call as needed to check for current quotas. How the helper operates is not relevant to Squid. Thats important. The key things being that; its always called for new visitors to assign the start quota, and when the quota is nearing empty its called again to see if they get more. Helper would need to send back UNITS AMOUNT MINIMUM where UNITS is the unit of quota (seconds, bytes, requests, misses?, other?), AMOUNT being a integer count of units the client is allowed to use, and MINIMUM is the level of units where the helper is to be asked for an update. 0 remaining units results in an Error page 'quota exceeded' or somesuch. Amos -- Please be using Current Stable Squid 2.7.STABLE6 or 3.0.STABLE13 Current Beta Squid 3.1.0.5
Feature: quota control
Hi Guys, I would like to offer my time in working on this feature - I have not done any squid dev, but since I would like to see this feature in Squid, I thought I would take it on. I have briefly contacted Amos off list and we agreed that there is no set in stone way of doing this. I would like to propose that we then start throwing around some ideas and let's see if we can get this into squid :) Some ideas that Amos quickly said : - Based on delay pools - Use of external helpers to track traffic The way I see this happening is that a Quota is like a pool that empties based on 2 classes - bytes and requests. Requests will be for things like the number of requests, i.e. a person is only allowed to download 5 exe's per day or 5 requests of 1meg or something like that (it just popped into my head :) ) Bytes is a pretty straight forward one, the user is only allowed x amount of bytes per y amount of time. Anyways - let the ideas fly :) Cheers, Pieter
