Re: CoDel Flows

[Daniel Melameth]

On Fri, Oct 13, 2017 at 7:47 PM, Mike Belopuhov  wrote:
> On Fri, Oct 13, 2017 at 05:33 +, Daniel Melameth wrote:
>> In playing around with the new CoDel/fair traffic sharing, it's not
>> clear to me the best way to work with this when also using the
>> previous queuing.
>
> It's not clear to me either at least not in the generic case :-)
> I guess it depends on what you're trying to achieve.
>
>> Will CoDel still work as expected if all my child queues have flows,
>> but my root queue is using "fifo" (revealed with systat queues)?
>
> Depends on what you expect CoDel to do.  Normally the idea here is
> to set an upper bound on latency that all outgoing packets experience.
> For example if you have 10 connections and 2 are uploading data and
> other 8 are a mix of ACKs and SSH keystrokes, with FIFO you'd normally
> see bulk connections saturating the link and not leaving other 8
> connections a chance to send a packet.
>
> So you go and create those HFSC queues and try to reserve the bandwidth
> for your ACKs, SSH and whatnot.  The approach that FQ-CoDel takes is
> different.  You no longer need to reserve bandwidth as FQ-CoDel attempts
> to make the bandwidth "available" when needed -- this is what is fair
> sharing essentially.  Which in practice means that those 8 connections
> are able to send their small packets "practically" whenever they want
> without disrupting your uploads.
>
> This means that if all you want is to be able for your outgoing
> connections to fair share the bandwidth you don't need to reserve the
> bandwidth at all.
>
>> Assuming it does, if one of my child queues is just for TCP ACKs, does
>> it make sense to have a small quantum for this queue, but a larger
>> quantum for a child queue that focuses on bulk file transfers?
>
> Quantum of service just tilts the balance at the expense of extra CPU
> cycles and potentially extra overall latency.  I think you need to
> figure out the big picture first and then fine tune.
>
>> Or is
>> CoDel orthogonal of child queues and it only really works well with a
>> single root flow queue (and requires me to give up bandwidth control
>> with child queues)?
>
> "Works well this way or that way" would imply that we have enough data
> to make such a judgement.  At the moment we don't.  Last week we had it
> running with 8192 flows feeding into an LTE connection with a rather
> flaky 50Mbit/s downlink (150Mbit/s up) for about a hundred of users.
> With a few HFSC tweaks we had almost no observable SSH latency with
> ping times to 8.8.8.8 of about 25ms with fairly low variation.  This
> setup used two root queues: one on the uplink, one on the downlink.
>
>> Also, the pf.conf man page says the default qlimit is 1024, but, if I
>> don't specify a qlimit, pfctl –vsq shows a qlength of 50 when I was
>> expecting it to be 1024.  What am I missing?
>
> I've updated the man page today to address some of the concerns since
> the same question was also brought up on reddit yesterday:
>
> https://www.reddit.com/r/openbsd/comments/75ps6h/fqcodel_and_pf/
>
> The gist of it is that 1024 is not the HFSC default.  When you're
> specifying both "flows" and "bandwidth" thus requesting an FQ-CoDel
> queue manager for your HFSC queue, the HFSC default qlimit (50) is
> still applied.  It's a bit counter-intuitive I guess, so I've removed
> mention of this from the man page.

Thanks for taking the time for a detailed reply Mike.  From your
Reddit post, it seems, for those queues that use both flows and
bandwidth, it makes sense to always override the HFSC qlimit default,
but will this increase latency (in the same way a queue with no flows
will increase latency with a higher qlimit)?  I'll see what I can dig
up on CoDel and quantum outside of OpenBSD circles.

That said, I've been piloting various queuing scenarios in a Hyper-V
environment, but I haven't been able to make much progress here as, it
appears, there's some timing issue with HFSC and/or hvn(4) (thank you
for terminating my use of de(4), which was horrible under Hyper-V!); I
can never seem to reach my modest bandwidth specifications with
something like tcpbench, but perhaps this is better left for another
thread or I should just get on the vmd(8) bandwagon.

Cheers.

Re: CoDel Flows

[Mike Belopuhov]

On Mon, Oct 16, 2017 at 03:24 +, Glenn Faustino wrote:
> Thanks Mike!
> 
> When I was using newqueue/hfsc I used to assign queues to certain traffic
> like below:
> 
> match inet proto tcp from any to any port ssh set queue (ssh_bulkq,
> ssh_prioq)
> match inet proto {tcp,udp} from any to any port {domain,ntp} set queue
> (dnsq, ackq)
> match inet proto {tcp,udp} from any to any port {www,https} set queue
> (webq, ackq)
> match inet proto tcp from any to any port ftp set queue (webq, ackq)
> 
> And I was looking how to do that when using flow queues but it seems that
> it is not needed anymore (if I'm not mistaken) ,
> all you need to do is define a flow queue and that's it.
>

I see.  Indeed, I'd start with a single flow queue, but that doesn't
necessarily mean that this is going to work in 100% of cases.  Please
take a look at the approach I've outlined here:

https://www.reddit.com/r/openbsd/comments/75ps6h/fqcodel_and_pf/doemlgi/

I don't have enough feedback and experience with this exact setup so
I'm treading carefully here until there's a clear understanding what
works and what doesn't.

And please let me remind you again, that first of all you need to
identify whether or not the problem actually exists.  Once you figured
that your setup is prone to exposing the bufferbloat on the uplink,
setup the flow queue as I've suggested here:

https://www.reddit.com/r/openbsd/comments/75ps6h/fqcodel_and_pf/doemlgi/

Test and then re-evaluate the situation and only if you see the problem
with downloads, attempt to fix it.

Cheers,
Mike

> 
> Regards,
> Glenn
> 
> 
> 
> On Mon, Oct 16, 2017 at 2:06 AM, Mike Belopuhov  wrote:
> 
> > On Sat, Oct 14, 2017 at 02:51 +, Glenn Faustino wrote:
> > > Hi Mike,
> > >
> > > I'm using HFSC with two root queues (1 for uplink and 1 for downlink),
> > can
> > > you please share your config for FQ-CoDel with HFSC with two queues if
> > you
> > > don't mind?
> > >
> > > Regards,
> > > Glenn
> > >
> >
> > I'm not certain which config you're talking about but there's nothing
> > particularly different about any of those that I've used. I'd like to
> > know instead what might be causing trouble for you to figure out what
> > piece of documentation can be improved.
> >
> > Having said that I tend to set quantum a bit lower when CPU resources
> > are abundant and latency is not increased as a result. For example,
> > OpenWRT sets it to 300 by default which might mean that they optimize
> > for an average packet size on the internet (or it might not, I don't
> > know for sure why do they do it). My limited testing hasn't shown any
> > quantifiable gain when doing that.
> >
> > Regards,
> > Mike
> >

Re: CoDel Flows

[Glenn Faustino]

Thanks Mike!

When I was using newqueue/hfsc I used to assign queues to certain traffic
like below:

match inet proto tcp from any to any port ssh set queue (ssh_bulkq,
ssh_prioq)
match inet proto {tcp,udp} from any to any port {domain,ntp} set queue
(dnsq, ackq)
match inet proto {tcp,udp} from any to any port {www,https} set queue
(webq, ackq)
match inet proto tcp from any to any port ftp set queue (webq, ackq)

And I was looking how to do that when using flow queues but it seems that
it is not needed anymore (if I'm not mistaken) ,
all you need to do is define a flow queue and that's it.


Regards,
Glenn



On Mon, Oct 16, 2017 at 2:06 AM, Mike Belopuhov  wrote:

> On Sat, Oct 14, 2017 at 02:51 +, Glenn Faustino wrote:
> > Hi Mike,
> >
> > I'm using HFSC with two root queues (1 for uplink and 1 for downlink),
> can
> > you please share your config for FQ-CoDel with HFSC with two queues if
> you
> > don't mind?
> >
> > Regards,
> > Glenn
> >
>
> I'm not certain which config you're talking about but there's nothing
> particularly different about any of those that I've used. I'd like to
> know instead what might be causing trouble for you to figure out what
> piece of documentation can be improved.
>
> Having said that I tend to set quantum a bit lower when CPU resources
> are abundant and latency is not increased as a result. For example,
> OpenWRT sets it to 300 by default which might mean that they optimize
> for an average packet size on the internet (or it might not, I don't
> know for sure why do they do it). My limited testing hasn't shown any
> quantifiable gain when doing that.
>
> Regards,
> Mike
>

Re: CoDel Flows

[Mike Belopuhov]

On Sat, Oct 14, 2017 at 02:51 +, Glenn Faustino wrote:
> Hi Mike,
> 
> I'm using HFSC with two root queues (1 for uplink and 1 for downlink), can
> you please share your config for FQ-CoDel with HFSC with two queues if you
> don't mind?
> 
> Regards,
> Glenn
> 

I'm not certain which config you're talking about but there's nothing
particularly different about any of those that I've used. I'd like to
know instead what might be causing trouble for you to figure out what
piece of documentation can be improved.

Having said that I tend to set quantum a bit lower when CPU resources
are abundant and latency is not increased as a result. For example,
OpenWRT sets it to 300 by default which might mean that they optimize
for an average packet size on the internet (or it might not, I don't
know for sure why do they do it). My limited testing hasn't shown any
quantifiable gain when doing that.

Regards,
Mike

Re: CoDel Flows

[Glenn Faustino]

Hi Mike,

I'm using HFSC with two root queues (1 for uplink and 1 for downlink), can
you please share your config for FQ-CoDel with HFSC with two queues if you
don't mind?

Regards,
Glenn



On Sat, Oct 14, 2017 at 9:47 AM, Mike Belopuhov  wrote:

> On Fri, Oct 13, 2017 at 05:33 +, Daniel Melameth wrote:
> > In playing around with the new CoDel/fair traffic sharing, it's not
> > clear to me the best way to work with this when also using the
> > previous queuing.
>
> It's not clear to me either at least not in the generic case :-)
> I guess it depends on what you're trying to achieve.
>
> > pfctl balks when I specify a flow on the root queue
> > with child queues present, so I only specify flows on child queues.
>
> Only leaf queues can be used to assign traffic, so if your root queue
> isn't a leaf queue, then it's actual packet queue (sorry for overusing
> the term) is not being used and therefore it doesn't make sense for
> codel to be specified there at all.
>
> > Will CoDel still work as expected if all my child queues have flows,
> > but my root queue is using "fifo" (revealed with systat queues)?
>
> Depends on what you expect CoDel to do.  Normally the idea here is
> to set an upper bound on latency that all outgoing packets experience.
> For example if you have 10 connections and 2 are uploading data and
> other 8 are a mix of ACKs and SSH keystrokes, with FIFO you'd normally
> see bulk connections saturating the link and not leaving other 8
> connections a chance to send a packet.
>
> So you go and create those HFSC queues and try to reserve the bandwidth
> for your ACKs, SSH and whatnot.  The approach that FQ-CoDel takes is
> different.  You no longer need to reserve bandwidth as FQ-CoDel attempts
> to make the bandwidth "available" when needed -- this is what is fair
> sharing essentially.  Which in practice means that those 8 connections
> are able to send their small packets "practically" whenever they want
> without disrupting your uploads.
>
> This means that if all you want is to be able for your outgoing
> connections to fair share the bandwidth you don't need to reserve the
> bandwidth at all.
>
> > Assuming it does, if one of my child queues is just for TCP ACKs, does
> > it make sense to have a small quantum for this queue, but a larger
> > quantum for a child queue that focuses on bulk file transfers?
>
> Quantum of service just tilts the balance at the expense of extra CPU
> cycles and potentially extra overall latency.  I think you need to
> figure out the big picture first and then fine tune.
>
> > Or is
> > CoDel orthogonal of child queues and it only really works well with a
> > single root flow queue (and requires me to give up bandwidth control
> > with child queues)?
> >
>
> "Works well this way or that way" would imply that we have enough data
> to make such a judgement.  At the moment we don't.  Last week we had it
> running with 8192 flows feeding into an LTE connection with a rather
> flaky 50Mbit/s downlink (150Mbit/s up) for about a hundred of users.
> With a few HFSC tweaks we had almost no observable SSH latency with
> ping times to 8.8.8.8 of about 25ms with fairly low variation.  This
> setup used two root queues: one on the uplink, one on the downlink.
>
> > Also, the pf.conf man page says the default qlimit is 1024, but, if I
> > don't specify a qlimit, pfctl –vsq shows a qlength of 50 when I was
> > expecting it to be 1024.  What am I missing?
> >
>
> I've updated the man page today to address some of the concerns since
> the same question was also brought up on reddit yesterday:
>
> https://www.reddit.com/r/openbsd/comments/75ps6h/fqcodel_and_pf/
>
> The gist of it is that 1024 is not the HFSC default.  When you're
> specifying both "flows" and "bandwidth" thus requesting an FQ-CoDel
> queue manager for your HFSC queue, the HFSC default qlimit (50) is
> still applied.  It's a bit counter-intuitive I guess, so I've removed
> mention of this from the man page.
>
> > Thank you!
> >
>
>

Re: CoDel Flows

[Mike Belopuhov]

On Fri, Oct 13, 2017 at 05:33 +, Daniel Melameth wrote:
> In playing around with the new CoDel/fair traffic sharing, it's not
> clear to me the best way to work with this when also using the
> previous queuing.

It's not clear to me either at least not in the generic case :-)
I guess it depends on what you're trying to achieve.

> pfctl balks when I specify a flow on the root queue
> with child queues present, so I only specify flows on child queues.

Only leaf queues can be used to assign traffic, so if your root queue
isn't a leaf queue, then it's actual packet queue (sorry for overusing
the term) is not being used and therefore it doesn't make sense for
codel to be specified there at all.

> Will CoDel still work as expected if all my child queues have flows,
> but my root queue is using "fifo" (revealed with systat queues)?

Depends on what you expect CoDel to do.  Normally the idea here is
to set an upper bound on latency that all outgoing packets experience.
For example if you have 10 connections and 2 are uploading data and
other 8 are a mix of ACKs and SSH keystrokes, with FIFO you'd normally
see bulk connections saturating the link and not leaving other 8
connections a chance to send a packet.

So you go and create those HFSC queues and try to reserve the bandwidth
for your ACKs, SSH and whatnot.  The approach that FQ-CoDel takes is
different.  You no longer need to reserve bandwidth as FQ-CoDel attempts
to make the bandwidth "available" when needed -- this is what is fair
sharing essentially.  Which in practice means that those 8 connections
are able to send their small packets "practically" whenever they want
without disrupting your uploads.

This means that if all you want is to be able for your outgoing
connections to fair share the bandwidth you don't need to reserve the
bandwidth at all.

> Assuming it does, if one of my child queues is just for TCP ACKs, does
> it make sense to have a small quantum for this queue, but a larger
> quantum for a child queue that focuses on bulk file transfers?

Quantum of service just tilts the balance at the expense of extra CPU
cycles and potentially extra overall latency.  I think you need to
figure out the big picture first and then fine tune.

> Or is
> CoDel orthogonal of child queues and it only really works well with a
> single root flow queue (and requires me to give up bandwidth control
> with child queues)?
>

"Works well this way or that way" would imply that we have enough data
to make such a judgement.  At the moment we don't.  Last week we had it
running with 8192 flows feeding into an LTE connection with a rather
flaky 50Mbit/s downlink (150Mbit/s up) for about a hundred of users.
With a few HFSC tweaks we had almost no observable SSH latency with
ping times to 8.8.8.8 of about 25ms with fairly low variation.  This
setup used two root queues: one on the uplink, one on the downlink.

> Also, the pf.conf man page says the default qlimit is 1024, but, if I
> don't specify a qlimit, pfctl –vsq shows a qlength of 50 when I was
> expecting it to be 1024.  What am I missing?
>

I've updated the man page today to address some of the concerns since
the same question was also brought up on reddit yesterday:

https://www.reddit.com/r/openbsd/comments/75ps6h/fqcodel_and_pf/

The gist of it is that 1024 is not the HFSC default.  When you're
specifying both "flows" and "bandwidth" thus requesting an FQ-CoDel
queue manager for your HFSC queue, the HFSC default qlimit (50) is
still applied.  It's a bit counter-intuitive I guess, so I've removed
mention of this from the man page.

> Thank you!
> 

Re: CoDel Flows

[Daniel Melameth]

On Thu, Oct 12, 2017 at 11:48 PM, Daniel Ouellet  wrote:
>> Also, the pf.conf man page says the default qlimit is 1024, but, if I
>> don't specify a qlimit, pfctl –vsq shows a qlength of 50 when I was
>> expecting it to be 1024.  What am I missing?
>
> Why would you want to have a pool of 1024 oppose to the default of 50
> slots for your queue?

Because CoDel works differently.

> You will increase latency when you have congestion. It's not because you
> can have a 1024 limits that it make sense to use it by default.
>
> The default is 50 and that's plenty good for most usage.
>
> As Peter Hansteen would write, Quote
>
> "Cranking up queue sizes here means we’re a little less likely to drop
> packets when the traffic approaches the set limits, but it also means
> that when the traffic shaping kicks in, we’ll see increased latency for
> connections that end up in these larger than default pools."
>
> It's a trade off, just know the impact of your choice.

While I agree with you (and Peter!), this is no longer the case with
CoDel--see the author's guidance at
https://www.reddit.com/r/openbsd/comments/6ttuhn/fq_codel_scheduling/
and send him beer to convince him to write the article he mentions.

Re: CoDel Flows

[Daniel Ouellet]

> Also, the pf.conf man page says the default qlimit is 1024, but, if I
> don't specify a qlimit, pfctl –vsq shows a qlength of 50 when I was
> expecting it to be 1024.  What am I missing?

Why would you want to have a pool of 1024 oppose to the default of 50
slots for your queue?

You will increase latency when you have congestion. It's not because you
can have a 1024 limits that it make sense to use it by default.

The default is 50 and that's plenty good for most usage.

As Peter Hansteen would write, Quote

"Cranking up queue sizes here means we’re a little less likely to drop
packets when the traffic approaches the set limits, but it also means
that when the traffic shaping kicks in, we’ll see increased latency for
connections that end up in these larger than default pools."

It's a trade off, just know the impact of your choice.

As for the CoDel I will let someone that actually use it explain it
better then I could.

Hope this help some.

Daniel