Re: Fair distribution of borrowed bandwidth with a lot of users
On Tue, Apr 24, 2007 at 09:49:32AM +0200, Federico Giannici wrote:
jared r r spiegel wrote:
On Tue, Apr 24, 2007 at 01:42:26AM -0400, jared r r spiegel wrote:
On Mon, Apr 23, 2007 at 10:12:56AM +0200, Federico Giannici wrote:
How can I make a single queue don't borrow ALL the traffic?
upperlimit
OK, my question was badly expressed.
I have already written the problem a few times: the question is not to
put an upper limit to a queue but to don't make it monopolize all the
bandwidth because of its great amount of traffic, leaving queues with
low traffic with ALL packets dropped (as you van see in the pftop
snapshot I sent).
I already explained it with the following example:
Both users A and B have an assigned bandwidth of 1 bps.
User A currently requires 4 bps.
User B currently requires 100 bps.
There are currently available 10 bps.
I'd like that the 10 bps would be equally distributed to both users (as
they have the same assigned bandwidth), so both had potentially 5 bps.
User A uses 4 bps and leaves 1 bps to user B that so uses 6 bps.
This is the fair bandwidth assignment I'd like to implement.
Indeed it seems that, as user B requires 25 times more bandwidth of user
A, then it is assigned almost ALL bandwidth, and user A is not able to
use more then it's committed 1 bps, and so 3 out of 4 bits are dropped!
in this case it is probably not super important to see your
whole pf.conf, but without seeing the service curves you're
applying to the queues, it's a bit of flying blind.
it seems to me that what you want to do is feasible with HFSC.
posting your altq section would be helpful in trying to
resolve the problem you're seeing.
Attached is the part of the pf.conf files that defines that part of
queues. As you can see it's pretty simple. (Only wh1_i has an upperlimit
because all the others have a limit in their router.)
how did wh10_i get to be 527KB/s if wdsl_i is capped out at 1650Kb/s?
queue wdsl_i bandwidth 1650Kb { wdsl_hi_i wdsl_low_i }
queue wdsl_hi_i bandwidth 99% { wh1_i wh2_i wh3_i wh4_i wh5_i wh6_i
wh7_i wh8_i wh9_i wh10_i wh11_i wh12_i wh13_i wh14_i wh15_i wh16_i wh17_i
wh18_i wh19_i wh20_i wh21_i wh22_i wh23_i wh24_i wh25_i wh26_i wh27_i wh28_i
wh29_i wh30_i wh31_i wh32_i wh33_i wh34_i wh35_i wh36_i wh37_i wh38_i wh39_i
wh40_i wh41_i wh42_i wh43_i }
queue wh1_i bandwidth 6400b hfsc( linkshare 6400b
upperlimit 2048Kb )
queue wh2_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh3_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh4_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh5_i bandwidth 2000b hfsc( linkshare 2000b )
snip
i've never used HFSC without explicitly defining realtime scs.
when i pass your queues through pfctl -nvvf after the following
edits:
- i took only the first 12 whX_i queues
- put in the missing 'altq on'
- added wdsl_low_i, and made it default
i see:
[/home/jrrs] $ pfctl -nvvf pf.test
altq on em0 hfsc bandwidth 8Mb tbrsize 6000 queue { wdsl_i }
queue wdsl_i bandwidth 1.65Mb { wdsl_hi_i wdsl_low_i }
queue wdsl_low_i bandwidth 1% hfsc( default )
queue wdsl_hi_i bandwidth 99% { wh1_i wh2_i wh3_i wh4_i wh5_i wh6_i wh7_i wh8_i
wh9_i wh10_i wh11_i wh12_i }
queue wh1_i bandwidth 6.40Kb hfsc( upperlimit 2.05Mb )
queue wh2_i bandwidth 6.40Kb
queue wh3_i bandwidth 6.40Kb
queue wh4_i bandwidth 2Kb
queue wh5_i bandwidth 2Kb
queue wh6_i bandwidth 2Kb
queue wh7_i bandwidth 2Kb
queue wh8_i bandwidth 2Kb
queue wh9_i bandwidth 2Kb
queue wh10_i bandwidth 6.40Kb
queue wh11_i bandwidth 6.40Kb
queue wh12_i bandwidth 2Kb
the first thing that strikes me is that there are no HFSC specific
declarations there for the subqueues, only 'bandwidth'.
in my experience with HFSC, the bandwidth keyword has very little to
do with anything other than it having to be set low enough so that
the sum of all the child queues' bandwidth doesn't exceed the parent.
i guess that the rest of the parameters are not present (in my pfctl output)
because the ones in the queue declarations in pf.conf equate to what
the defaults would be. changing the linkshare a bit seems to corroborate
this.
my first guesses would be to define a realtime explicitly (as 6400b for
one of your 6400b queues is certainly not the default, as it shows up
in pfctl -nvvf output if you explicltly say it) and/or to use an actual
sc for linkshare (define m1 and d) instead of a static number (which then
equates to having only defined m2).
from what i understand of HFSC, the service curves are where the real magic
happens -- if you're not using them in a curve fashion at all, it seems
that you end up with something not very different from regular CBQ.
it may also help, if the interface these queues hang off of has more
bandwidth (say, link of 100Mb?) than you're trying to limit
people to (in the popular
Re: Fair distribution of borrowed bandwidth with a lot of users
On Tue, Apr 24, 2007 at 01:42:26AM -0400, jared r r spiegel wrote: On Mon, Apr 23, 2007 at 10:12:56AM +0200, Federico Giannici wrote: How can I make a single queue don't borrow ALL the traffic? upperlimit in this case it is probably not super important to see your whole pf.conf, but without seeing the service curves you're applying to the queues, it's a bit of flying blind. it seems to me that what you want to do is feasible with HFSC. posting your altq section would be helpful in trying to resolve the problem you're seeing. -- jared
Re: Fair distribution of borrowed bandwidth with a lot of users
jared r r spiegel wrote:
On Tue, Apr 24, 2007 at 01:42:26AM -0400, jared r r spiegel wrote:
On Mon, Apr 23, 2007 at 10:12:56AM +0200, Federico Giannici wrote:
How can I make a single queue don't borrow ALL the traffic?
upperlimit
OK, my question was badly expressed.
I have already written the problem a few times: the question is not to
put an upper limit to a queue but to don't make it monopolize all the
bandwidth because of its great amount of traffic, leaving queues with
low traffic with ALL packets dropped (as you van see in the pftop
snapshot I sent).
I already explained it with the following example:
Both users A and B have an assigned bandwidth of 1 bps.
User A currently requires 4 bps.
User B currently requires 100 bps.
There are currently available 10 bps.
I'd like that the 10 bps would be equally distributed to both users (as
they have the same assigned bandwidth), so both had potentially 5 bps.
User A uses 4 bps and leaves 1 bps to user B that so uses 6 bps.
This is the fair bandwidth assignment I'd like to implement.
Indeed it seems that, as user B requires 25 times more bandwidth of user
A, then it is assigned almost ALL bandwidth, and user A is not able to
use more then it's committed 1 bps, and so 3 out of 4 bits are dropped!
in this case it is probably not super important to see your
whole pf.conf, but without seeing the service curves you're
applying to the queues, it's a bit of flying blind.
it seems to me that what you want to do is feasible with HFSC.
posting your altq section would be helpful in trying to
resolve the problem you're seeing.
Attached is the part of the pf.conf files that defines that part of
queues. As you can see it's pretty simple. (Only wh1_i has an upperlimit
because all the others have a limit in their router.)
Thanks.
--
___
__
|- [EMAIL PROTECTED]
|ederico Giannici http://www.neomedia.it
___
queue wdsl_i bandwidth 1650Kb { wdsl_hi_i wdsl_low_i }
queue wdsl_hi_i bandwidth 99% { wh1_i wh2_i wh3_i wh4_i wh5_i wh6_i
wh7_i wh8_i wh9_i wh10_i wh11_i wh12_i wh13_i wh14_i wh15_i wh16_i wh17_i
wh18_i wh19_i wh20_i wh21_i wh22_i wh23_i wh24_i wh25_i wh26_i wh27_i wh28_i
wh29_i wh30_i wh31_i wh32_i wh33_i wh34_i wh35_i wh36_i wh37_i wh38_i wh39_i
wh40_i wh41_i wh42_i wh43_i }
queue wh1_i bandwidth 6400b hfsc( linkshare 6400b
upperlimit 2048Kb )
queue wh2_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh3_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh4_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh5_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh6_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh7_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh8_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh9_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh10_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh11_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh12_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh13_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh14_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh15_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh16_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh17_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh18_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh19_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh20_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh21_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh22_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh23_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh24_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh25_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh26_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh27_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh28_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh29_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh30_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh31_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh32_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh33_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh34_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh35_i bandwidth 6400b hfsc( linkshare 6400b )
queue wh36_i bandwidth 2000b hfsc( linkshare 2000b )
queue wh37_i bandwidth 6400b hfsc(
Re: Fair distribution of borrowed bandwidth with a lot of users
Attached you can find a snapshot of the queue view of pftop (only part of the queue is shown). As you can see, even if all queue have similar assigned bandwidths (6400 and 2000 bps), there is a single queue with a GREAT amount of traffic (wh10_i) that monopolize ALL the bandwidth! There are 10 other queues with bytes to flow (QLEN != 0) but all the packets are dropped (B/S = 0 and DROP_P keeps increasing). To me this is a bug, or at least a bad behavior. Am I wrong? How can I make a single queue don't borrow ALL the traffic? Thanks. Federico Giannici wrote: Bob DeBolt wrote: Hi Federico Giannici Posting you pf.conf will be of considerable benefit when attempting to seek help for something that has the complexity you are currently dealing with. Additionally, the type connection you have, i.e. DSL, cable etc. as the variations each of these has throughout the day will skew the appearance of your results etc. We have a 6Mb connection (made with 3 2Mb ATM connections in IMA). Supplying your complete pf.conf correctly commented with what you are wanting each rule and queue to accomplish is a good place to start. Our pf.conf file is quite long and complex (actually it is made of 3 different files included with anchors). Anyway, at this point, I'm only making a THEORETICAL question, to see if it can explain the bad behaviour I'm seeing. It's better explined with and example: Both users A and B have an assigned bandwidth of 1 bps. User A currently requires 4 bps. User B currently requires 100 bps. There are currently available 10 bps. I'd like that the 10 bps would be equally distributed to both users (as they have the same assigned bandwidth), so both had 5 bps. User A uses 4 bps and leaves 1 bps to user B that so uses 6 bps. Indeed it seems that, as user B requires 25 times more bandwidth of user A, then it is assigned almost ALL bandwidth, and user A is not able to use more then it's committed 1 bps, and so 3 out of 4 bits are dropped! Is this true? Thanks. -- ___ __ |- [EMAIL PROTECTED] |ederico Giannici http://www.neomedia.it ___ QUEUE BW SCH PRIO PKTSBYTES DROP_P DROP_B QLEN BORROW SUSPEN P/S B/S wdsl_i1650K hfsc 000 00 0 0 wdsl_hi_i1633K hfsc 000 00 0 0 wh1_i6400 hfsc 6715 18643310 00 0 0 wh2_i6400 hfsc397422 243653K33018 1870274 50 0 0 wh3_i6400 hfsc 1890986 722889K 112854 64732960 0 0 wh4_i2000 hfsc116512 98728K 275 616704 1 106 wh5_i2000 hfsc 1033715 6029800417576 9864980 0 0 wh6_i2000 hfsc119974 62863556 2953 2045623 1.0 53 wh7_i2000 hfsc 174129850 00 0 0 wh8_i2000 hfsc 1913030 130063K81439 46565320 0 0 wh9_i2000 hfsc 82929 89439226 11 6500 0 0 wh10_i 6400 hfsc 10806525 15417M 158664 229242K 35 348 527600 wh11_i 6400 hfsc 2207 14260260 00 0 0 wh12_i 2000 hfsc 49798 620441970 00 0 0 wh13_i 2000 hfsc219271 239251K 1380 981860 0 0 wh14_i 2000 hfsc 1152602 85891883 113484 6294012 28 0 0 wh15_i 6400 hfsc 3879133 407778K 372882 22772260 50 0 0 wh16_i 2000 hfsc157744 19172673 2573 1516180 0 0 wh17_i 2000 hfsc 34590 34553732 106 65980 0 0 wh18_i 6400 hfsc133850 172387K0 00 0 0 wh19_i 6400 hfsc 39296 38332321 78 78080 0 0 wh20_i 2000 hfsc 98808 73759318 770 1642540 0 0 wh21_i
Re: Fair distribution of borrowed bandwidth with a lot of users
Bob DeBolt wrote: Hi Federico Giannici Posting you pf.conf will be of considerable benefit when attempting to seek help for something that has the complexity you are currently dealing with. Additionally, the type connection you have, i.e. DSL, cable etc. as the variations each of these has throughout the day will skew the appearance of your results etc. We have a 6Mb connection (made with 3 2Mb ATM connections in IMA). Supplying your complete pf.conf correctly commented with what you are wanting each rule and queue to accomplish is a good place to start. Our pf.conf file is quite long and complex (actually it is made of 3 different files included with anchors). Anyway, at this point, I'm only making a THEORETICAL question, to see if it can explain the bad behaviour I'm seeing. It's better explined with and example: Both users A and B have an assigned bandwidth of 1 bps. User A currently requires 4 bps. User B currently requires 100 bps. There are currently available 10 bps. I'd like that the 10 bps would be equally distributed to both users (as they have the same assigned bandwidth), so both had 5 bps. User A uses 4 bps and leaves 1 bps to user B that so uses 6 bps. Indeed it seems that, as user B requires 25 times more bandwidth of user A, then it is assigned almost ALL bandwidth, and user A is not able to use more then it's committed 1 bps, and so 3 out of 4 bits are dropped! Is this true? Thanks. -- ___ __ |- [EMAIL PROTECTED] |ederico Giannici http://www.neomedia.it ___
Re: Fair distribution of borrowed bandwidth with a lot of users
Lawrence Horvath wrote: have you considered percentages? Using percentages is equivalent to using the corresponding values. It doesn't solve my problem. or using a script to inject ips into a tables based queue setup? I already use a script to generate the pf.conf file, but I cannot understand what do you mean with tables based queue. How can it change the way the bandwidth in excess is distributed between queues? Bye. On 16/04/07, Federico Giannici [EMAIL PROTECTED] wrote: As there was no reply to this email of mine, anybody can tell me if there is some other place where I can submit this question? At least, anybody can tell me if my theory about bandwidth assignments is wrong? Thanks. Federico Giannici wrote: I have a problem creating a policy to make a fair use of available bandwidth in a situation with a lot of potential users. We have a certain number of users, each one with a committed bandwidth of X or Y. We setup a queue system with an HFSC queue for each user, with a bandwidth (linkshare parameter) proportional to X or Y. This system doesn't seem to work in a fair way: when an user does a HUGE download with a very large number of packets and bytes per second, it steals almost all bandwidth to the other users requiring only a little of bandwidth. It seems that the borrowed bandwidth is distributed to the queues based on the amount of required packets (or bytes). Indeed I'd like that the bandwidth is distributed strictly proportional to the assigned bandwidth. Let's make and example: Both users A and B have an assigned bandwidth of 1 bps. User A currently requires 4 bps. User B currently requires 100 bps. There are currently available 10 bps. I'd like that the 10 bps would be equally distributed to both users (as they have the same assigned bandwidth), so both had 5 bps. User A uses 4 bps and leaves 1 bps to user B that so uses 6 bps. Indeed it seems that, as user B requires 25 times more bandwidth of user A, then it is assigned almost ALL bandwidth, and user A is not able to use more then it's committed 1 bps, and so 3 out of 4 bits are dropped! Is this true? Is there a way to obtain my desired behavior? Please note that I cannot simply increase the commited bandwidth of both users to 5 bps because there are a lot of other potential users (that currently are not using bandwidth) so the sum of bandwidths would exceeds the bandwidth of the parent queue. Thanks. -- ___ __ |- [EMAIL PROTECTED] |ederico Giannici http://www.neomedia.it ___
Re: Fair distribution of borrowed bandwidth with a lot of users
Federico Giannici [EMAIL PROTECTED] writes: or using a script to inject ips into a tables based queue setup? I already use a script to generate the pf.conf file, but I cannot understand what do you mean with tables based queue. I think what he means is, you use pass from tablename queue foo constructs -- Peter N. M. Hansteen, member of the first RFC 1149 implementation team http://www.blug.linux.no/rfc1149/ http://www.datadok.no/ http://www.nuug.no/ First, we kill all the spammers The Usenet Bard, Twice-forwarded tales delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
Re: Fair distribution of borrowed bandwidth with a lot of users
Hi Federico Giannici Posting you pf.conf will be of considerable benefit when attempting to seek help for something that has the complexity you are currently dealing with. Additionally, the type connection you have, i.e. DSL, cable etc. as the variations each of these has throughout the day will skew the appearance of your results etc. Supplying your complete pf.conf correctly commented with what you are wanting each rule and queue to accomplish is a good place to start. You will consistently find that without adequate information it will be difficult for people to help you. Bob signature.asc Description: OpenPGP digital signature
