Re: PF: can't make queueing and priority work as expected

[lists]

Sat, 9 Jan 2016 17:45:01 +0200 Mihai Popescu 
> Where is the enlightment gone?

In proofreading the queuing FAQ and updating the best current practice,
thank Marko for bravely trying to understand and apply it with the risk
of being insulted unjustly.

Line 59
[http://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/www/faq/pf/index.html?=text/html]


Line 63
[http://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/www/faq/pf/queueing.html?=text/html]
WARNING: This document is currently out-of-date and refers to a
queueing method that will be removed in OpenBSD 5.6.

For homework, try to see if this still applies:

http://www.benzedrine.ch/ackpri.html

Re: PF: can't make queueing and priority work as expected

[Craig Skinner]

On 2016-01-15 Fri 12:53 PM |, David Gwynne wrote:
> > On 13 Jan 2016, at 19:19, Marko Cupa??  wrote:
> >
> > Have we come to conclusion that currently prio makes no sense at all?
> 
> it wont have the effect you want. that doesn't mean it doesn't make sense
> somewhere else.
> 

Such as an ADSL PPPoE bridge?
-- 
The older a man gets, the farther he had to walk to school as a boy.

Re: PF: can't make queueing and priority work as expected

[Marko Cupać]

On Fri, 15 Jan 2016 21:57:15 +1000
David Gwynne  wrote:

> the other thing to note is that loading a ruleset resets the
> assignment of existing states to queues.
>
> states are assigned to queues via rules, but if the rules go away
> (which is what happens when you load a new ruleset) the intermediary
> between rules and queues has gone.

I noticed states in systat states which have '*' as RULE, and noticed
behaviour you explained (traffic being in different queue than stated
in loaded ruleset) in systat queues.

> it kind of sucks, especially for testing.

I don't mind having to flush states or even reboot complete system.
Would be better if not, but definitely not a show stopper.

I just gave another look to earlier thread Atanas posted link to:
http://marc.info/?l=openbsd-misc=141085207225887=2

I have not yet tried i386, but I'll give it a try today and post
results back.

--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Re: PF: can't make queueing and priority work as expected

[David Gwynne]

> On 15 Jan 2016, at 9:07 PM, Craig Skinner  wrote:
>
> On 2016-01-15 Fri 12:53 PM |, David Gwynne wrote:
>>> On 13 Jan 2016, at 19:19, Marko Cupa??  wrote:
>>>
>>> Have we come to conclusion that currently prio makes no sense at all?
>>
>> it wont have the effect you want. that doesn't mean it doesn't make sense
>> somewhere else.
>>
>
> Such as an ADSL PPPoE bridge?

yeah.

the other thing to note is that loading a ruleset resets the assignment of
existing states to queues.

states are assigned to queues via rules, but if the rules go away (which is
what happens when you load a new ruleset) the intermediary between rules and
queues has gone.

it kind of sucks, especially for testing.

dlg

Re: PF: can't make queueing and priority work as expected

[Mihai Popescu]

> Ok, let's start insult each other.

No insult intended from my side and no one commited.

> I've setup my first PF on OpenBSD in 2006.

As Master Fu said once, if you can't setup it by yourself, maybe you
should not use it.

> Stuck your advice up your ass and fuck off.

I'm curious who will dear to answer your question on this forum anymore.

Bye.

Re: PF: can't make queueing and priority work as expected

[David Gwynne]

> On 13 Jan 2016, at 19:19, Marko Cupać  wrote:
>
> On Tue, 12 Jan 2016 16:40:58 +0100
> Claudio Jeker  wrote:
>
>> On Tue, Jan 12, 2016 at 05:33:06AM -0700, Daniel Melameth wrote:
>>> On Mon, Jan 11, 2016 at 9:37 PM, David Gwynne 
>>> wrote:
> On 11 Jan 2016, at 22:43, Daniel Melameth 
> wrote: On Sun, Jan 10, 2016 at 7:58 AM, Marko Cupa??
> 
>>> wrote:
>> On Sat, 9 Jan 2016 11:11:27 -0700
>> Daniel Melameth  wrote:
>>> You NEED to set a max on your ROOT queues.
>> I came to this conclusion as well. But not only on root queues.
>> For example, when max is set on root queue but only bandwidth
>> on child queues, no shaping takes place...
> This works for me.
>> Or, to cut the long story short, if someone can paste queue
>> definition which accomplishes 'give both queues max bandwidth,
>> but throttle traffic from first queue when traffic from the
>> second one arrives', I will be more than happy to quit
>> bothering misc@ list readers with my rants and observations.
> I would expect this to be possible with prio alone, but I've
> never been able to get it to work.  Perhaps I'm misunderstanding
> how prio works.
 prio is basically an array of lists of packets to be transmitted.
 high
>>> priority packets go on a different list to low priority packets.

 the problem is the way packets go on and off these lists.
 basically as soon
>>> as a packet is queued on one of these lists for transmission, we
>>> call the driver immediately to send it. generally as soon as a
>>> packet is queued on the interface, it immediately gets dequeued by
>>> the driver and transmitted on the hardware.

 it is only when you build up a backlog of packets that priq can
 come into
>>> effect. the only way you can build up a backlog of packets is if
>>> your hardware is slower at transmitting packets than the thing that
>>> generates these packets to send.

 in your case you're probably getting packets from a relatively
 slow internet
>>> connection and transmitting them on a high speed local network. the
>>> transmit hardware is almost certainly going to be faster than your
>>> source of packets, so you'll never build up a queue of backlogged
>>> packets, so prio is effectively a nop.

 dlg
>>>
>>> Thanks for taking the time to chime in guys.  Prior to implementing
>>> any queueing, I tested this stuff out on a LAN--so no slower
>>> connectionswere involved--and I was unable to see prio in action, at
>>> least not with any observable similarity to ALTQ's PRIQ.
>>>
>>> A simple rule set:
>>>
>>> match out on egress proto tcp to port 12345 set prio 7
>>> match out on egress proto tcp to port 12346 set prio 0
>>> pass
>>>
>>> Using tcpbench to push packets into both queues, I would have
>>> expected the packets destined for port 12346 to get throttled, but
>>> both flows simply reached an equilibrium, which I would have
>>> expected without prio.  Under PRIQ, I would have seen the flow to
>>> port 12346 get almost completely starved of bandwidth.  When doing
>>> non-prio queuing with a similarly simple ruleset, both flows
>>> properly matched their target bandwidth.
>>
>> This assumes that you manage to fill the TX interface queue to a level
>> that it always fills the tx DMA rings before being empty. On high
>> speed interfaces this most of the time not the case and so both
>> sessions are able to reach the maximum bandwidth.
>> To be honest prio queue only make sense when you have a slow interface
>> (10Mbps) or a shaper in place that causes the queue to fill up.
>> There is currently no shaper you can use together with the prio
>> queues so only option one remains.
>>
>
> Have we come to conclusion that currently prio makes no sense at all?

it wont have the effect you want. that doesn't mean it doesn't make sense
somewhere else.

>
> Can I hope that saying 'currently' means this is not the intended
> design? Or should I come to peace with the fact that with OpenBSD and
> PF I can forget about shaping inbound TCP traffic in a way that
> child queues can expand to max link bandwidth unless there is a
> congestion, while in congestion admin can choose which child queues to
> throttle and in which order?

hfsc might need some work at the code level, it might just suck to configure.

>
> --
> Before enlightenment - chop wood, draw water.
> After  enlightenment - chop wood, draw water.
>
> Marko Cupać
> https://www.mimar.rs/

Re: PF: can't make queueing and priority work as expected

[Marko Cupać]

On Tue, 12 Jan 2016 16:40:58 +0100
Claudio Jeker  wrote:

> On Tue, Jan 12, 2016 at 05:33:06AM -0700, Daniel Melameth wrote:
> > On Mon, Jan 11, 2016 at 9:37 PM, David Gwynne 
> > wrote:
> > >> On 11 Jan 2016, at 22:43, Daniel Melameth 
> > >> wrote: On Sun, Jan 10, 2016 at 7:58 AM, Marko Cupa??
> > >> 
> > wrote:
> > >>> On Sat, 9 Jan 2016 11:11:27 -0700
> > >>> Daniel Melameth  wrote:
> >  You NEED to set a max on your ROOT queues.
> > >>> I came to this conclusion as well. But not only on root queues.
> > >>> For example, when max is set on root queue but only bandwidth
> > >>> on child queues, no shaping takes place...
> > >> This works for me.
> > >>> Or, to cut the long story short, if someone can paste queue
> > >>> definition which accomplishes 'give both queues max bandwidth,
> > >>> but throttle traffic from first queue when traffic from the
> > >>> second one arrives', I will be more than happy to quit
> > >>> bothering misc@ list readers with my rants and observations.
> > >> I would expect this to be possible with prio alone, but I've
> > >> never been able to get it to work.  Perhaps I'm misunderstanding
> > >> how prio works.
> > > prio is basically an array of lists of packets to be transmitted.
> > > high
> > priority packets go on a different list to low priority packets.
> > >
> > > the problem is the way packets go on and off these lists.
> > > basically as soon
> > as a packet is queued on one of these lists for transmission, we
> > call the driver immediately to send it. generally as soon as a
> > packet is queued on the interface, it immediately gets dequeued by
> > the driver and transmitted on the hardware.
> > >
> > > it is only when you build up a backlog of packets that priq can
> > > come into
> > effect. the only way you can build up a backlog of packets is if
> > your hardware is slower at transmitting packets than the thing that
> > generates these packets to send.
> > >
> > > in your case you're probably getting packets from a relatively
> > > slow internet
> > connection and transmitting them on a high speed local network. the
> > transmit hardware is almost certainly going to be faster than your
> > source of packets, so you'll never build up a queue of backlogged
> > packets, so prio is effectively a nop.
> > >
> > > dlg
> >
> > Thanks for taking the time to chime in guys.  Prior to implementing
> > any queueing, I tested this stuff out on a LAN--so no slower
> > connectionswere involved--and I was unable to see prio in action, at
> > least not with any observable similarity to ALTQ's PRIQ.
> >
> > A simple rule set:
> >
> > match out on egress proto tcp to port 12345 set prio 7
> > match out on egress proto tcp to port 12346 set prio 0
> > pass
> >
> > Using tcpbench to push packets into both queues, I would have
> > expected the packets destined for port 12346 to get throttled, but
> > both flows simply reached an equilibrium, which I would have
> > expected without prio.  Under PRIQ, I would have seen the flow to
> > port 12346 get almost completely starved of bandwidth.  When doing
> > non-prio queuing with a similarly simple ruleset, both flows
> > properly matched their target bandwidth.
>
> This assumes that you manage to fill the TX interface queue to a level
> that it always fills the tx DMA rings before being empty. On high
> speed interfaces this most of the time not the case and so both
> sessions are able to reach the maximum bandwidth.
> To be honest prio queue only make sense when you have a slow interface
> (10Mbps) or a shaper in place that causes the queue to fill up.
> There is currently no shaper you can use together with the prio
> queues so only option one remains.
>

Have we come to conclusion that currently prio makes no sense at all?

Can I hope that saying 'currently' means this is not the intended
design? Or should I come to peace with the fact that with OpenBSD and
PF I can forget about shaping inbound TCP traffic in a way that
child queues can expand to max link bandwidth unless there is a
congestion, while in congestion admin can choose which child queues to
throttle and in which order?

--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Re: PF: can't make queueing and priority work as expected

[Marko Cupać]

On Wed, 13 Jan 2016 10:19:22 +0100
Marko Cupać  wrote:


> Can I hope that saying 'currently' means this is not the intended
> design? Or should I come to peace with the fact that with OpenBSD and
> PF I can forget about shaping inbound TCP traffic in a way that
> child queues can expand to max link bandwidth unless there is a
> congestion, while in congestion admin can choose which child queues to
> throttle and in which order?

Of course I didn't mean 'inbound' as 'incoming to NIC', but 'incoming
from the Internet'. Actual shaping would take place outbound on internal
interface.

I'm writing this so I don't get another set of mails which warn me I
can't shape inbound, but need to shape outbound traffic.

--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Re: PF: can't make queueing and priority work as expected

[Mihai Popescu]

> I'm writing this so I don't get another set of mails which warn me I
> can't shape inbound, but need to shape outbound traffic.

Hello Marko,

Don't you think you are out of subject with this thread already?
Now, seriously, do you expect someone to teach you queues online?

Maybe the PF implementation part is not the swiss army knife of
traffic shapping, but your emails have no actual question.

My advice is to start a new thread, state your desires for traffic
shapping in a gateway handling p2p and http(s) and see if someone can
help.

Is is very interesting that you started learning traffic rules a few
days ago but you are up to question the implementation.

Regards.

Re: PF: can't make queueing and priority work as expected

[Marko Cupać]

On Wed, 13 Jan 2016 16:36:23 +0200
Mihai Popescu  wrote:

> > I'm writing this so I don't get another set of mails which warn me I
> > can't shape inbound, but need to shape outbound traffic.
>
> Hello Marko,
>
> Don't you think you are out of subject with this thread already?
> Now, seriously, do you expect someone to teach you queues online?
>
> Maybe the PF implementation part is not the swiss army knife of
> traffic shapping, but your emails have no actual question.
>
> My advice is to start a new thread, state your desires for traffic
> shapping in a gateway handling p2p and http(s) and see if someone can
> help.
>
> Is is very interesting that you started learning traffic rules a few
> days ago but you are up to question the implementation.
>
> Regards.

Ok, let's start insult each other.

I've setup my first PF on OpenBSD in 2006.

Stuck your advice up your ass and fuck off.
--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Re: PF: can't make queueing and priority work as expected

[Marko Cupać]

On Tue, 12 Jan 2016 14:37:49 +1000
David Gwynne  wrote:

> prio is basically an array of lists of packets to be transmitted. high
> priority packets go on a different list to low priority packets.


> the problem is the way packets go on and off these lists. basically
> as soon as a packet is queued on one of these lists for transmission,
> we call the driver immediately to send it. generally as soon as a
> packet is queued on the interface, it immediately gets dequeued by
> the driver and transmitted on the hardware.
>
> it is only when you build up a backlog of packets that priq can come
> into effect. the only way you can build up a backlog of packets is if
> your hardware is slower at transmitting packets than the thing that
> generates these packets to send.
>
> in your case you're probably getting packets from a relatively slow
> internet connection and transmitting them on a high speed local
> network. the transmit hardware is almost certainly going to be faster
> than your source of packets, so you'll never build up a queue of
> backlogged packets, so prio is effectively a nop.
>
> dlg
>

David,

Thank you for the explanation of the mechanism. As you say, I am
getting packets from a relatively slow internet connection (10M/1M),
and transmitting them on a high speed (1G) local network. NICs are 1G. I
guess this is standard for majority of smaller deployments (homes,
smaller businesses).

Besides what you explained now, I don't have the knowledge of
underlying queueing mechanisms of PF. But from what you said, it seems
logical that it should be possible to build up a queue of backlogged
packets not only by hitting physical limit of the hardware, but also
by setting logical limit. Otherwise PF prio is only useful to ISPs who
can saturate 1G or 10G links.

This may sound stupid, but perhaps it would be possible to keep prio
values on packets in child queues and let all those queues transfer
packets to additional 'outbound' queue before letting driver send it.
Something like frame-relay, but not only flagging packets with discard
bits and drop them first in cases of congestion, but something more
intelligent, which takes into account target, min and max bandwiths,
bursts and their timeperiods, priorities...

...which describes the idea of ALTQ HFSC. Which people who know
what they are talking say wasn't working. But current queueing
mechanism doesn't work for a lot of setups either. Now what?
--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Re: PF: can't make queueing and priority work as expected

[Daniel Melameth]

On Mon, Jan 11, 2016 at 9:37 PM, David Gwynne  wrote:
>> On 11 Jan 2016, at 22:43, Daniel Melameth  wrote:
>> On Sun, Jan 10, 2016 at 7:58 AM, Marko Cupać 
wrote:
>>> On Sat, 9 Jan 2016 11:11:27 -0700
>>> Daniel Melameth  wrote:
 You NEED to set a max on your ROOT queues.
>>> I came to this conclusion as well. But not only on root queues. For
>>> example, when max is set on root queue but only bandwidth on child
>>> queues, no shaping takes place...
>> This works for me.
>>> Or, to cut the long story short, if someone can paste queue definition
>>> which accomplishes 'give both queues max bandwidth, but throttle
>>> traffic from first queue when traffic from the second one arrives', I
>>> will be more than happy to quit bothering misc@ list readers with my
>>> rants and observations.
>> I would expect this to be possible with prio alone, but I've never
>> been able to get it to work.  Perhaps I'm misunderstanding how prio
>> works.
> prio is basically an array of lists of packets to be transmitted. high
priority packets go on a different list to low priority packets.
>
> the problem is the way packets go on and off these lists. basically as soon
as a packet is queued on one of these lists for transmission, we call the
driver immediately to send it. generally as soon as a packet is queued on the
interface, it immediately gets dequeued by the driver and transmitted on the
hardware.
>
> it is only when you build up a backlog of packets that priq can come into
effect. the only way you can build up a backlog of packets is if your hardware
is slower at transmitting packets than the thing that generates these packets
to send.
>
> in your case you're probably getting packets from a relatively slow internet
connection and transmitting them on a high speed local network. the transmit
hardware is almost certainly going to be faster than your source of packets,
so you'll never build up a queue of backlogged packets, so prio is effectively
a nop.
>
> dlg

Thanks for taking the time to chime in guys.  Prior to implementing
any queueing, I tested this stuff out on a LAN--so no slower
connectionswere involved--and I was unable to see prio in action, at
least not with any observable similarity to ALTQ's PRIQ.

A simple rule set:

match out on egress proto tcp to port 12345 set prio 7
match out on egress proto tcp to port 12346 set prio 0
pass

Using tcpbench to push packets into both queues, I would have expected
the packets destined for port 12346 to get throttled, but both flows
simply reached an equilibrium, which I would have expected without
prio.  Under PRIQ, I would have seen the flow to port 12346 get almost
completely starved of bandwidth.  When doing non-prio queuing with a
similarly simple ruleset, both flows properly matched their target
bandwidth.

Re: PF: can't make queueing and priority work as expected

[Claudio Jeker]

On Tue, Jan 12, 2016 at 05:33:06AM -0700, Daniel Melameth wrote:
> On Mon, Jan 11, 2016 at 9:37 PM, David Gwynne  wrote:
> >> On 11 Jan 2016, at 22:43, Daniel Melameth  wrote:
> >> On Sun, Jan 10, 2016 at 7:58 AM, Marko Cupa?? 
> wrote:
> >>> On Sat, 9 Jan 2016 11:11:27 -0700
> >>> Daniel Melameth  wrote:
>  You NEED to set a max on your ROOT queues.
> >>> I came to this conclusion as well. But not only on root queues. For
> >>> example, when max is set on root queue but only bandwidth on child
> >>> queues, no shaping takes place...
> >> This works for me.
> >>> Or, to cut the long story short, if someone can paste queue definition
> >>> which accomplishes 'give both queues max bandwidth, but throttle
> >>> traffic from first queue when traffic from the second one arrives', I
> >>> will be more than happy to quit bothering misc@ list readers with my
> >>> rants and observations.
> >> I would expect this to be possible with prio alone, but I've never
> >> been able to get it to work.  Perhaps I'm misunderstanding how prio
> >> works.
> > prio is basically an array of lists of packets to be transmitted. high
> priority packets go on a different list to low priority packets.
> >
> > the problem is the way packets go on and off these lists. basically as soon
> as a packet is queued on one of these lists for transmission, we call the
> driver immediately to send it. generally as soon as a packet is queued on the
> interface, it immediately gets dequeued by the driver and transmitted on the
> hardware.
> >
> > it is only when you build up a backlog of packets that priq can come into
> effect. the only way you can build up a backlog of packets is if your hardware
> is slower at transmitting packets than the thing that generates these packets
> to send.
> >
> > in your case you're probably getting packets from a relatively slow internet
> connection and transmitting them on a high speed local network. the transmit
> hardware is almost certainly going to be faster than your source of packets,
> so you'll never build up a queue of backlogged packets, so prio is effectively
> a nop.
> >
> > dlg
> 
> Thanks for taking the time to chime in guys.  Prior to implementing
> any queueing, I tested this stuff out on a LAN--so no slower
> connectionswere involved--and I was unable to see prio in action, at
> least not with any observable similarity to ALTQ's PRIQ.
> 
> A simple rule set:
> 
> match out on egress proto tcp to port 12345 set prio 7
> match out on egress proto tcp to port 12346 set prio 0
> pass
> 
> Using tcpbench to push packets into both queues, I would have expected
> the packets destined for port 12346 to get throttled, but both flows
> simply reached an equilibrium, which I would have expected without
> prio.  Under PRIQ, I would have seen the flow to port 12346 get almost
> completely starved of bandwidth.  When doing non-prio queuing with a
> similarly simple ruleset, both flows properly matched their target
> bandwidth.

This assumes that you manage to fill the TX interface queue to a level
that it always fills the tx DMA rings before being empty. On high speed
interfaces this most of the time not the case and so both sessions are
able to reach the maximum bandwidth. 
To be honest prio queue only make sense when you have a slow interface
(10Mbps) or a shaper in place that causes the queue to fill up.
There is currently no shaper you can use together with the prio queues so
only option one remains.

-- 
:wq Claudio

Re: PF: can't make queueing and priority work as expected

[David Gwynne]

> On 11 Jan 2016, at 22:43, Daniel Melameth  wrote:
>
> On Sun, Jan 10, 2016 at 7:58 AM, Marko Cupać  wrote:
>> On Sat, 9 Jan 2016 11:11:27 -0700
>> Daniel Melameth  wrote:
>>> You NEED to set a max on your ROOT queues.
>> I came to this conclusion as well. But not only on root queues. For
>> example, when max is set on root queue but only bandwidth on child
>> queues, no shaping takes place...
>
> This works for me.
>
>> Or, to cut the long story short, if someone can paste queue definition
>> which accomplishes 'give both queues max bandwidth, but throttle
>> traffic from first queue when traffic from the second one arrives', I
>> will be more than happy to quit bothering misc@ list readers with my
>> rants and observations.
>
> I would expect this to be possible with prio alone, but I've never
> been able to get it to work.  Perhaps I'm misunderstanding how prio
> works.

prio is basically an array of lists of packets to be transmitted. high
priority packets go on a different list to low priority packets.

the problem is the way packets go on and off these lists. basically as soon as
a packet is queued on one of these lists for transmission, we call the driver
immediately to send it. generally as soon as a packet is queued on the
interface, it immediately gets dequeued by the driver and transmitted on the
hardware.

it is only when you build up a backlog of packets that priq can come into
effect. the only way you can build up a backlog of packets is if your hardware
is slower at transmitting packets than the thing that generates these packets
to send.

in your case you're probably getting packets from a relatively slow internet
connection and transmitting them on a high speed local network. the transmit
hardware is almost certainly going to be faster than your source of packets,
so you'll never build up a queue of backlogged packets, so prio is effectively
a nop.

dlg

Re: PF: can't make queueing and priority work as expected

[Daniel Melameth]

On Sun, Jan 10, 2016 at 7:58 AM, Marko Cupać  wrote:
> On Sat, 9 Jan 2016 11:11:27 -0700
> Daniel Melameth  wrote:
>> You NEED to set a max on your ROOT queues.
> I came to this conclusion as well. But not only on root queues. For
> example, when max is set on root queue but only bandwidth on child
> queues, no shaping takes place...

This works for me.

> Or, to cut the long story short, if someone can paste queue definition
> which accomplishes 'give both queues max bandwidth, but throttle
> traffic from first queue when traffic from the second one arrives', I
> will be more than happy to quit bothering misc@ list readers with my
> rants and observations.

I would expect this to be possible with prio alone, but I've never
been able to get it to work.  Perhaps I'm misunderstanding how prio
works.

Re: PF: can't make queueing and priority work as expected

[Marko Cupać]

On Sat, 9 Jan 2016 11:11:27 -0700
Daniel Melameth  wrote:

> >> You NEED to set a max on your ROOT queues.
> > I came to this conclusion as well. But not only on root queues. For
> > example, when max is set on root queue but only bandwidth on child
> > queues, no shaping takes place...
>
> This works for me.

Thank you for "small bandwidth issue" information. Can someone give
more precise measurement of "small bandwidth"? What is the smallest
bandwidth of a queue which is known to work reliably? Did I understand
correctly there is resolution to this issue, and if so, which is it?
Where do I start looking why this works for Daniel but doesn't for me?

> > Or, to cut the long story short, if someone can paste queue
> > definition which accomplishes 'give both queues max bandwidth, but
> > throttle traffic from first queue when traffic from the second one
> > arrives', I will be more than happy to quit bothering misc@ list
> > readers with my rants and observations.
>
> I would expect this to be possible with prio alone, but I've never
> been able to get it to work.  Perhaps I'm misunderstanding how prio
> works.

I would expect this as well, and I've been able to get it to work on
old ALTQ queueing system but not on current queueing system. Either I'm
misunderstanding how prio works as well, or we are coming to the point
where multiple users aren't able to get it to work.

Either way I hope some authoritative person from OpenBSD who knows
how prio works will give us the possibility to understand it by means of
direct advice, correct documentation, or confirm it doesn't work. Some
of us poor illiterates don't understand pf code and depend on manpages.
--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Re: PF: can't make queueing and priority work as expected

[Marko Cupać]

On Sat, 9 Jan 2016 11:11:27 -0700
Daniel Melameth  wrote:

> You NEED to set a max on your ROOT queues.

I came to this conclusion as well. But not only on root queues. For
example, when max is set on root queue but only bandwidth on child
queues, no shaping takes place:

queue upload   on $if_ext   bandwidth  850K max  850K
 queue bulk parent upload   bandwidth  350K default
 queue slow parent upload   bandwidth  500K
queue download on $if_int   bandwidth 8500K max 8500K
 queue bulk parent download bandwidth 3500K default
 queue slow parent download bandwidth 5000K

The only way I can shape traffic to set values is by setting both
bandwidth and max on all the queues. Bandwidth and max values must be
equal per queue, and sum of child queue bandwidths must be equal to
parent queue bandwidth:

queue upload   on $if_ext   bandwidth  850K max  850K
 queue bulk parent upload   bandwidth  350K max  350K default
 queue slow parent upload   bandwidth  500K max  500K
queue download on $if_int   bandwidth 8500K max 8500K
 queue bulk parent download bandwidth 3500K max 3500K default
 queue slow parent download bandwidth 5000K max 5000K

So, while fixed-bandwidth queues seem to work, they have the
disadvantage of wasted bandwidth reserved for traffic which is not
currently on wire.

I have also noticed that min value setting does not reserve bandwidth.
No shaping takes place with the following queue definition:

queue upload   on $if_ext   bandwidth  850K max  850K
 queue bulk parent upload   bandwidth  350K min  350K default
 queue slow parent upload   bandwidth  500K min  500K
queue download on $if_int   bandwidth 8500K max 8500K
 queue bulk parent download bandwidth 3500K min 3500K default
 queue slow parent download bandwidth 5000K min 5000K

Let's take a look at this queue definition:
queue upload   on $if_ext   bandwidth  850K min  850K max  850K
 queue bulk parent upload   bandwidth  350K min  150K max  800K default
 queue slow parent upload   bandwidth  500K min  100K max  800K
queue download on $if_int   bandwidth 8500K min 8500K max 8500K
 queue bulk parent download bandwidth 3500K min 1500K max 8000K default
 queue slow parent download bandwidth 5000K min 1000K max 8000K

I guess it is clear what was intended with it:
- limit outbound traffic to 850K, inbound to 8.5M
- both child queues can take up to 800K/8M if there is no traffic from
  the other queue on the wire
- if traffic from both queues is on the wire, requesting maximum
  possible bandwidth, shape bulk to 350K/3.5M and slow to 500K/5M.
  Under no circumstances let slow queue starve bulk queue under
  150K/1.5M, and under no circumstances let bulk queue starve slow
  queue under 100K/1M

However, it does no shaping at all. Bulk queue becomes completely
starved with slow queue. Not only bulk queue doesn't get shaped to
350K/3.5M - it doesn't get its min 150K/1.5M either.

> prio is useless when shaping--see
> http://marc.info/?l=openbsd-misc=140127924031145=2.

Thank you for reminding me I've been trying to make this work more than
year and half ago. Back in time I thought "it's completely new, perhaps
I'll wait a release or two, it'll get better". But obviously it didn't.

Could it be that removing the ability to prioritize traffic already in
queues prevents throttling?

Or, to cut the long story short, if someone can paste queue definition
which accomplishes 'give both queues max bandwidth, but throttle
traffic from first queue when traffic from the second one arrives', I
will be more than happy to quit bothering misc@ list readers with my
rants and observations.
--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Re: PF: can't make queueing and priority work as expected

[Marko Cupać]

Thanks to everyone who contributed to this thread. Replies were not
what I hoped for, but I still appreciate I didn't get just silence.

My claims about the issue were:
- setting prio as per manpage instructions does not throttle
  lower-priority traffic
- setting bandwidth on queues as manpage instructs does not shape
  traffic to declared bandwidth, unless queues are fixed with
  undocumented mandatory min and max values

I hoped for something like:
- observe output of this and that, post it here so we can check facts
  and give you clear instruction on how to achieve your goal
- (or at least) it is not possible with contemporary OpenBSD

I got:
- try what you already have
- I have also noticed it
- recompile the kernel with unsupported option
- I guess I made it work for me by doing undocumented thing

I think that the fact that a network admin with 10 years worth of
experience with OpenBSD and PF has to spend more than 50 hours of
reading bits and pieces on the Internet in order to classify
low-priority traffic and throttle it down when higher-priority traffic
is on the wire, and still not succeed in it, proves bad state of either
relevant official documentation, or software capabilities.

New queueing mechanism might be cool from developer's point of view,
but if people who actually configure production firewalls need 50 hours
not to accomplish the same goal which they used to accomplish in 20
minutes, I'm sure they would go for uncool queueing mechanism of old.

So, when I post "your shiny new queueing mechanism doesn't do its job,
and your famous documentation is incorrect" on openbsd-misc mailing
list, developers are hush?

Hope this is due to Christmas holidays...

--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Re: PF: can't make queueing and priority work as expected

[Mihai Popescu]

> Situation is still the same: torrents being downloaded at full speed
> (~8Mbit/s), simultaneous download of install59.fs from ftp.openbsd.org
> averages at ~6Kbit/s.

I'm not a PF consultant, but be aware that p2p can be a real beast to
setup. I was asked by someone to handle a double NAT situation for
uTorrent and I was searching for some details about uTorrent port
handling. I found something not so clear like OpenBSD man pages, and I
found out that uTorrent is able to work in 2 modes: with ports blocked
and with open ports (aka 'port forwarding'). Be sure you are in one of
those 2 modes and your PF is seeing the traffic accordingly.

Re: PF: can't make queueing and priority work as expected

[Daniel Melameth]

On Thu, Jan 7, 2016 at 11:28 AM, Marko Cupać  wrote:
> I am setting up gateway for a small network which has two main types of
> traffic: p2p and http(s). The idea is to give p2p traffic all the
> available bandwidth until there is http(s) traffic, in which case p2p
> should be throttled down and http(s) should be given all the available
> bandwidth.
>
> The problem is that p2p does not get throttled down when http(s) is on
> wire. I spent days re-reading QUEUEING section of pf.conf and
> chapter #7 of 3rd edition of "Book of PF" but I still couldn't make it
> work. From 'systat queues' I can conclude that traffic seem to be
> assigned to appropriate queues, but queue bandwidth does not seem to be
> respected. What am I doing wrong?
>
> Snapshot of 'systat queues' and active pf.conf below:
>
> QUEUE BW SCH P  PKTS BYTES DROP_P DROP_B QLEN P/S  B/S
> upload on pppoe 860K   0 0  0  00   00
>  ack 10K228K   12M  0  00 364  20K
>  fast20K  60  5397  0  00   00
>  bulk   800K 263  110K  0  00 0.6  417
>  slow30K   34234   35M  0  00  59  67K
> download on re2   8M   0 0  0  00   00
>  ack100K   18314 1205K  0  00  34 2388
>  fast   200K  51 15491  0  00   00
>  bulk 8M   29014   41M  0  00  57  85K
>  slow   500K317K  457M  0  00 523 771K

> # PORT MACROS
> p2p = "{ 1:65535 }"
> prpi= "{ 4:40100 }"
> ipsec   = "{ 500 4500 }"
> web = "{ 80 443 }"
> mail= "{ 25 110 143 587 993 995 }"
> xmpp= "{ 5222 }"

> # QUEUES
> queue upload  on $if_ext bandwidth  860K
>queue ack  parent upload   qlimit 50  bandwidth   10K
>queue fast parent upload   qlimit 50  bandwidth   20K
>queue bulk parent upload   qlimit 50  bandwidth  800K default
>queue slow parent upload   qlimit 50  bandwidth   30K
> queue download on $if_intbandwidth 8800K
>queue ack  parent download qlimit 50  bandwidth  100K
>queue fast parent download qlimit 50  bandwidth  200K
>queue bulk parent download qlimit 50  bandwidth 8000K default
>queue slow parent download qlimit 50  bandwidth  500K

You NEED to set a max on your ROOT queues.  That said, as mentioned
previously, there is a known issue with resolution with small
bandwidth values.

> # SHAPING
> match proto icmp   set ( queue   fast   prio   4 )
> match proto tcp  to port 22set ( queue ( fast ack ) prio ( 4 5 ) )
> match proto tcp  to port 53set ( queue ( fast ack ) prio ( 4 5 ) )
> match proto udp  to port 53set ( queue   fast   prio   4 )
> match proto tcp  to port $web  set ( queue ( bulk ack ) prio ( 3 5 ) )
> match proto tcp  to port $mail set ( queue ( bulk ack ) prio ( 3 5 ) )
> match proto tcp  to port $xmpp set ( queue ( bulk ack ) prio ( 3 5 ) )
> match proto tcp  to port $p2p  set ( queue ( slow ack ) prio ( 0 5 ) )
> match proto udp  to port $p2p  set ( queue   slow   prio   0 )

prio is useless when shaping--see
http://marc.info/?l=openbsd-misc=140127924031145=2.

Re: PF: can't make queueing and priority work as expected

[Mihai Popescu]

--
Before queue - p2p high priority, http(s) low priority, no bad self
image on the misc@.
After queue - p2p high priority, http(s) low priority, bad self image
on the misc@.

Where is the enlightment gone?

Re: PF: can't make queueing and priority work as expected

[sven falempin]

On Fri, Jan 8, 2016 at 6:35 PM, Marko Cupać  wrote:

> On Fri, 8 Jan 2016 11:13:08 -0500
> sven falempin  wrote:
>
> > You will need to forward the all rule set i think, maybe the set prio
> > 0 is erased by a further rules, try to pass in quick those p2p
> > traffic before maybe ?
>
> I had the luxury of ditching the complete ruleset for very simple one:
>
> ---pf.conf-start---
> # RUNTIME OPTIONS
> set skip on lo0
>
> # INTERFACES
> if_int  = "re2"
> if_ext  = "pppoe0"
>
> # HOSTS & NETWORKS
> localnet   = "{ 192.168.33.0/24 }"
>
> # PORTS
> both_p2p = "{ 1000:65535 }"
>
> # NAT
> match in  all scrub ( no-df random-id max-mss 1440 )
> match out on $if_ext inet from $localnet to any nat-to ($if_ext:0)
>
> # RULES
> block drop log all
> pass inet from ($if_ext:0)   to any
> pass inet from $localnet to any
> pass inet proto tcp from ($if_ext:0) to any port $both_p2p \
>  set ( prio 0 )
> pass inet proto tcp from $localnet   to any port $both_p2p \
>  set ( prio 0 )
> pass inet proto udp from ($if_ext:0) to any port $both_p2p \
>  set ( prio 0 )
> pass inet proto udp from $localnet   to any port $both_p2p \
>  set ( prio 0 )
> ---pf.conf-end---
>
> Actual ruleset (as seen by pfctl -sr output) is as follows:
>
> ---pfctl-sr-start---
> match in all scrub (no-df random-id max-mss 1440)
> match out on pppoe0 inet from 192.168.33.0/24 to any nat-to (pppoe0:0)
> block drop log all
> pass inet from (pppoe0:0) to any flags S/SA
> pass inet from 192.168.33.0/24 to any flags S/SA
> pass inet proto tcp from (pppoe0:0) to any port 1000:65535 \
>  flags S/SA set ( prio 0 )
> pass inet proto tcp from 192.168.33.0/24 to any port 1000:65535 \
>  flags S/SA set ( prio 0 )
> pass inet proto udp from (pppoe0:0) to any port 1000:65535 \
>  set ( prio 0 )
> pass inet proto udp from 192.168.33.0/24 to any port 1000:65535 \
>  set ( prio 0 )
> ---pfctl-sr-end---
>
> Situation is still the same: torrents being downloaded at full speed
> (~8Mbit/s), simultaneous download of install59.fs from ftp.openbsd.org
> averages at ~6Kbit/s.
>
> Can anyone reproduce this?
> --
> Before enlightenment - chop wood, draw water.
> After  enlightenment - chop wood, draw water.
>
> Marko Cupać
> https://www.mimar.rs/
>

First of all, you cannot stop p2p outside world to flood you in, whatever
you do in udp.
If the software say "hey world DDOS me on my IP" , the world will ddos you
and packet will load the other
side of your pppoe.

So dont start ranting for the 50 hours, it s frustrating but there is hope,
but you may have to rethink your position
about bandwith shapping magic.


What is was proposing is you to quick out the rules first , something LIKE
this,
but it wont work dude, afaik, and it will be the same whatever the
algorithm.

---pf.conf-start---
# RUNTIME OPTIONS
set skip on lo0

# INTERFACES
if_int  = "re2"
if_ext  = "pppoe0"

# HOSTS & NETWORKS
localnet   = "{ 192.168.33.0/24 }"

# PORTS
both_p2p = "{ 1000:65535 }"


match in  all scrub ( no-df random-id max-mss 1440 )

# NAT

pass quick out on $if_ext inet from ($if_int:network) to any port
 1000:65535 set prio 0 nat-to ($if_ext:0)

match out on $if_ext inet from $localnet to any nat-to ($if_ext:0)

# RULES
block drop log all
pass inet from ($if_ext:0)   to any
pass inet from $localnet to any
pass inet proto tcp from ($if_ext:0) to any port $both_p2p \
 set ( prio 0 )
pass inet proto tcp from $localnet   to any port $both_p2p \
 set ( prio 0 )
pass inet proto udp from ($if_ext:0) to any port $both_p2p \
 set ( prio 0 )
pass inet proto udp from $localnet   to any port $both_p2p \
 set ( prio 0 )
---pf.conf-end---

--
() ascii ribbon campaign - against html e-mail
/\

Re: PF: can't make queueing and priority work as expected

[Atanas Vladimirov]

On 8 януари 2016 г. 17:51:21 Marko Cupać  wrote:


I am completely confused. It seems that everything I've known about
queueing in PF does not apply any more, while at the same time there are
no reliable sources to learn new stuff.

Let's follow this paragraph from 'Book of PF':

---quote---
Shaping by Setting Traffic Priorities
If you’re mainly interested in pushing certain kinds of traffic ahead
of others, you may be able to achieve what you want by simply setting
priorities: assigning a higher priority to some items so that they
receive attention before others.
---quote---

This is _exactly_ what I'm mainly interested in. As I want to throttle
just p2p traffic, I should be able to accomplish my goal with the
following relevant lines:

p2p = "{ 1:65535 }"
match proto { tcp udp } to port $p2p set prio 0
pass in  on $if_int inet proto { tcp udp }  from $if_int:network \
 to any port $p2p
pass out on $if_ext inet proto { tcp udp }  from $if_int:network \
 to any port $p2p

But, at least on my hardware, this does nothing. Torrents are still
being downloaded at max speed (~8Mbit/s), while simultaneous
download of install59.fs from ftp.obenbsd.org goes at at ~6Kbit/s.

Any comment? Suggestion? Instruction how to troubleshoot?

Thank you in advance.


http://marc.info/?l=openbsd-misc=141085207225887=2

Please read the whole thread. You have to set both min and max bw for every 
queue.

I think that this is a bug in amd64 - i386 works as it should.


Sent with AquaMail for Android
http://www.aqua-mail.com

Re: PF: can't make queueing and priority work as expected

[Marko Cupać]

On Fri, 8 Jan 2016 11:13:08 -0500
sven falempin  wrote:

> You will need to forward the all rule set i think, maybe the set prio
> 0 is erased by a further rules, try to pass in quick those p2p
> traffic before maybe ?

I had the luxury of ditching the complete ruleset for very simple one:

---pf.conf-start---
# RUNTIME OPTIONS
set skip on lo0

# INTERFACES
if_int  = "re2"
if_ext  = "pppoe0"

# HOSTS & NETWORKS
localnet   = "{ 192.168.33.0/24 }"

# PORTS
both_p2p = "{ 1000:65535 }"

# NAT
match in  all scrub ( no-df random-id max-mss 1440 )
match out on $if_ext inet from $localnet to any nat-to ($if_ext:0)

# RULES
block drop log all
pass inet from ($if_ext:0)   to any
pass inet from $localnet to any
pass inet proto tcp from ($if_ext:0) to any port $both_p2p \
 set ( prio 0 )
pass inet proto tcp from $localnet   to any port $both_p2p \
 set ( prio 0 )
pass inet proto udp from ($if_ext:0) to any port $both_p2p \
 set ( prio 0 )
pass inet proto udp from $localnet   to any port $both_p2p \
 set ( prio 0 )
---pf.conf-end---

Actual ruleset (as seen by pfctl -sr output) is as follows:

---pfctl-sr-start---
match in all scrub (no-df random-id max-mss 1440)
match out on pppoe0 inet from 192.168.33.0/24 to any nat-to (pppoe0:0)
block drop log all
pass inet from (pppoe0:0) to any flags S/SA
pass inet from 192.168.33.0/24 to any flags S/SA
pass inet proto tcp from (pppoe0:0) to any port 1000:65535 \
 flags S/SA set ( prio 0 )
pass inet proto tcp from 192.168.33.0/24 to any port 1000:65535 \
 flags S/SA set ( prio 0 )
pass inet proto udp from (pppoe0:0) to any port 1000:65535 \
 set ( prio 0 )
pass inet proto udp from 192.168.33.0/24 to any port 1000:65535 \
 set ( prio 0 )
---pfctl-sr-end---

Situation is still the same: torrents being downloaded at full speed
(~8Mbit/s), simultaneous download of install59.fs from ftp.openbsd.org
averages at ~6Kbit/s.

Can anyone reproduce this?
--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Re: PF: can't make queueing and priority work as expected

[Marko Cupać]

On Thu, 7 Jan 2016 22:41:47 + (UTC)
Stuart Henderson  wrote:

> On 2016-01-07, Marko Cupać  wrote:
> > # QUEUES
> > queue upload  on $if_ext bandwidth  860K
> >queue ack  parent upload   qlimit 50  bandwidth   10K
> >queue fast parent upload   qlimit 50  bandwidth   20K
> >queue bulk parent upload   qlimit 50  bandwidth  800K default
> >queue slow parent upload   qlimit 50  bandwidth   30K
> > queue download on $if_intbandwidth 8800K
> >queue ack  parent download qlimit 50  bandwidth  100K
> >queue fast parent download qlimit 50  bandwidth  200K
> >queue bulk parent download qlimit 50  bandwidth 8000K default
> >queue slow parent download qlimit 50  bandwidth  500K
>
> While the manual suggests it works like this, I've only got it working
> close to how I expect when I set "max" on the queues. I don't know
> whether that's a bug or simply lack of fully understanding it on my
> part, though.
>

I changed my configuration, it has 5 queues now. I also don't put ACKs
into separate queue, and I don't give them higher priority. I've set
"max" value on each queue. But http(s) traffic still doesn't get
priority over p2p.

Here's relevant part of pf.conf:

# QUEUES
queue upload  on $if_ext   bandwidth  860K max  860K
 queue fast parent upload   qlimit 50  bandwidth   40K max   40K
 queue web  parent upload   qlimit 50  bandwidth  600K max  600K default
 queue bulk parent upload   qlimit 50  bandwidth   20K max   20K
 queue slow parent upload   qlimit 50  bandwidth  100K max  600K
 queue p2p  parent upload   qlimit 50  bandwidth  100K max  600K
qeue download on $if_int   bandwidth 8600K max 8600K
 queue fast parent download qlimit 50  bandwidth  400K max  400K
 queue web  parent download qlimit 50  bandwidth 6000K max 6000K default
 queue bulk parent download qlimit 50  bandwidth  200K max  200K
 queue slow parent download qlimit 50  bandwidth 1000K max 6000K
 queue p2p  parent download qlimit 50  bandwidth 1000K max 6000K

# SHAPING
match proto icmp   set ( queue fast prio 6 )
match proto tcp  to port 22set ( queue fast prio 5 )
match proto tcp  to port 53set ( queue fast prio 5 )
match proto udp  to port 53set ( queue fast prio 5 )
match proto tcp  to port $xmpp set ( queue fast prio 5 )
match proto tcp  to port $web  set ( queue web  prio 4 )
match proto tcp  to port $mail set ( queue slow prio 2 )
match proto tcp  to port $p2p  set ( queue p2p  prio 0 )
match proto udp  to port $p2p  set ( queue p2p  prio 0 )

Should I conclude my goal of throttling smaller priority traffic to
minimum when higher priority traffic arrives can't be achieved with
current PF? If I haven't gone senile, I did this successfully on dozens
of firewalls back in altq/HFSC age.

Any good soul out there to point me in the right direction to achieve
my goal? Or at least confirm it is not possible? Could the problem be
related to hardware and not software? This is 5.8 with all errata
patches on pcengines' apu1d.

dmesg:
OpenBSD 5.8 (GENERIC.MP) #2: Thu Nov 26 10:23:47 CET 2015
pacija@kerber.mimar.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP
RTC BIOS diagnostic error
ff
real mem = 2098511872 (2001MB)
avail mem = 2031079424 (1936MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0x7e16d820 (7 entries)
bios0: vendor coreboot version "4.0" date 09/08/2014
bios0: PC Engines APU
acpi0 at bios0: rev 0
acpi0: sleep states S0 S1 S3 S4 S5
acpi0: tables DSDT FACP SPCR HPET APIC HEST SSDT SSDT SSDT
acpi0: wakeup devices AGPB(S4) HDMI(S4) PBR4(S4) PBR5(S4) PBR6(S4)
PBR7(S4) PE20(S4) PE21(S4) PE22(S4) PE23(S4) PIBR(S4) UOH1(S3) UOH2(S3)
UOH3(S3) UOH4(S3) UOH5(S3) [...]
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpihpet0 at acpi0: 14318180 Hz
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD G-T40E Processor, 1000.15 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,SSSE3,CX16,POPCNT,NXE,MMXX,FFXSR,PAGE1GB,L
ONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,IBS,SKINIT,ITSC
cpu0: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 512KB
64b/line 16-way L2 cache
cpu0: 8 4MB entries fully associative
cpu0: DTLB 40 4KB entries fully associative, 8 4MB entries fully
associative
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 199MHz
cpu0: mwait min=64, max=64, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD G-T40E Processor, 1000.00 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,SSSE3,CX16,POPCNT,NXE,MMXX,FFXSR,PAGE1GB,L

Re: PF: can't make queueing and priority work as expected

[David Coppa]

On Fri, Jan 8, 2016 at 12:44 PM, Marko Cupać  wrote:

> Should I conclude my goal of throttling smaller priority traffic to
> minimum when higher priority traffic arrives can't be achieved with
> current PF? If I haven't gone senile, I did this successfully on dozens
> of firewalls back in altq/HFSC age.
>
> Any good soul out there to point me in the right direction to achieve
> my goal? Or at least confirm it is not possible? Could the problem be
> related to hardware and not software? This is 5.8 with all errata
> patches on pcengines' apu1d.

Maybe is the "old" problem of queues with the default HZ value of 100?

See: http://marc.info/?l=openbsd-misc=140863695214420

Ciao!
David

> dmesg:
> OpenBSD 5.8 (GENERIC.MP) #2: Thu Nov 26 10:23:47 CET 2015
> pacija@kerber.mimar.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> RTC BIOS diagnostic error
> ff
> real mem = 2098511872 (2001MB)
> avail mem = 2031079424 (1936MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.7 @ 0x7e16d820 (7 entries)
> bios0: vendor coreboot version "4.0" date 09/08/2014
> bios0: PC Engines APU
> acpi0 at bios0: rev 0
> acpi0: sleep states S0 S1 S3 S4 S5
> acpi0: tables DSDT FACP SPCR HPET APIC HEST SSDT SSDT SSDT
> acpi0: wakeup devices AGPB(S4) HDMI(S4) PBR4(S4) PBR5(S4) PBR6(S4)
> PBR7(S4) PE20(S4) PE21(S4) PE22(S4) PE23(S4) PIBR(S4) UOH1(S3) UOH2(S3)
> UOH3(S3) UOH4(S3) UOH5(S3) [...]
> acpitimer0 at acpi0: 3579545 Hz, 32 bits
> acpihpet0 at acpi0: 14318180 Hz
> acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: AMD G-T40E Processor, 1000.15 MHz
> cpu0:
>
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
>
H,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,SSSE3,CX16,POPCNT,NXE,MMXX,FFXSR,PAGE1GB,L
> ONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,IBS,SKINIT,ITSC
> cpu0: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 512KB
> 64b/line 16-way L2 cache
> cpu0: 8 4MB entries fully associative
> cpu0: DTLB 40 4KB entries fully associative, 8 4MB entries fully
> associative
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 199MHz
> cpu0: mwait min=64, max=64, IBE
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: AMD G-T40E Processor, 1000.00 MHz
> cpu1:
>
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
>
H,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,SSSE3,CX16,POPCNT,NXE,MMXX,FFXSR,PAGE1GB,L
> ONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,IBS,SKINIT,ITSC
> cpu1: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 512KB
> 64b/line 16-way L2 cache
> cpu1: 8 4MB entries fully associative
> cpu1: DTLB 40 4KB entries fully associative, 8 4MB entries fully
> associative
> cpu1: smt 0, core 1, package 0
> ioapic0 at mainbus0: apid 2 pa 0xfec0, version 21, 24 pins
> acpiprt0 at acpi0: bus -1 (AGPB)
> acpiprt1 at acpi0: bus -1 (HDMI)
> acpiprt2 at acpi0: bus 1 (PBR4)
> acpiprt3 at acpi0: bus 2 (PBR5)
> acpiprt4 at acpi0: bus 3 (PBR6)
> acpiprt5 at acpi0: bus -1 (PBR7)
> acpiprt6 at acpi0: bus 5 (PE20)
> acpiprt7 at acpi0: bus -1 (PE21)
> acpiprt8 at acpi0: bus -1 (PE22)
> acpiprt9 at acpi0: bus -1 (PE23)
> acpiprt10 at acpi0: bus 0 (PCI0)
> acpiprt11 at acpi0: bus 4 (PIBR)
> acpicpu0 at acpi0: !C2(0@100 io@0x841), C1(@1 halt!), PSS
> acpicpu1 at acpi0: !C2(0@100 io@0x841), C1(@1 halt!), PSS
> acpibtn0 at acpi0: PWRB
> cpu0: 1000 MHz: speeds: 1000 800 MHz
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "AMD AMD64 14h Host" rev 0x00
> ppb0 at pci0 dev 4 function 0 "AMD AMD64 14h PCIE" rev 0x00: msi
> pci1 at ppb0 bus 1
> re0 at pci1 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E
> (0x2c00), msi, address 00:0d:b9:3e:84:9c
> rgephy0 at re0 phy 7: RTL8169S/8110S/8211 PHY, rev. 4
> ppb1 at pci0 dev 5 function 0 "AMD AMD64 14h PCIE" rev 0x00: msi
> pci2 at ppb1 bus 2
> re1 at pci2 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E
> (0x2c00), msi, address 00:0d:b9:3e:84:9d
> rgephy1 at re1 phy 7: RTL8169S/8110S/8211 PHY, rev. 4
> ppb2 at pci0 dev 6 function 0 "AMD AMD64 14h PCIE" rev 0x00: msi
> pci3 at ppb2 bus 3
> re2 at pci3 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E
> (0x2c00), msi, address 00:0d:b9:3e:84:9e
> rgephy2 at re2 phy 7: RTL8169S/8110S/8211 PHY, rev. 4
> ahci0 at pci0 dev 17 function 0 "ATI SBx00 SATA" rev 0x40: apic 2 int
> 19, AHCI 1.2
> scsibus1 at ahci0: 32 targets
> ohci0 at pci0 dev 18 function 0 "ATI SB700 USB" rev 0x00: apic 2 int
> 18, version 1.0, legacy support
> ehci0 at pci0 dev 18 function 2 "ATI SB700 USB2" rev 0x00: apic 2 int 17
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 "ATI EHCI root hub" rev 2.00/1.00 addr 1
> ohci1 at pci0 dev 19 function 0 "ATI SB700 USB" rev 0x00: apic 2 int
> 18, version 1.0, legacy support

Re: PF: can't make queueing and priority work as expected

[Marko Cupać]

I am completely confused. It seems that everything I've known about
queueing in PF does not apply any more, while at the same time there are
no reliable sources to learn new stuff.

Let's follow this paragraph from 'Book of PF':

---quote---
Shaping by Setting Traffic Priorities
If you’re mainly interested in pushing certain kinds of traffic ahead
of others, you may be able to achieve what you want by simply setting
priorities: assigning a higher priority to some items so that they
receive attention before others.
---quote---

This is _exactly_ what I'm mainly interested in. As I want to throttle
just p2p traffic, I should be able to accomplish my goal with the
following relevant lines:

p2p = "{ 1:65535 }"
match proto { tcp udp } to port $p2p set prio 0
pass in  on $if_int inet proto { tcp udp }  from $if_int:network \
 to any port $p2p
pass out on $if_ext inet proto { tcp udp }  from $if_int:network \
 to any port $p2p

But, at least on my hardware, this does nothing. Torrents are still
being downloaded at max speed (~8Mbit/s), while simultaneous
download of install59.fs from ftp.obenbsd.org goes at at ~6Kbit/s.

Any comment? Suggestion? Instruction how to troubleshoot?

Thank you in advance.
--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/

Re: PF: can't make queueing and priority work as expected

[Stuart Henderson]

On 2016-01-07, Marko Cupać  wrote:
> # QUEUES
> queue upload  on $if_ext bandwidth  860K
>queue ack  parent upload   qlimit 50  bandwidth   10K
>queue fast parent upload   qlimit 50  bandwidth   20K
>queue bulk parent upload   qlimit 50  bandwidth  800K default
>queue slow parent upload   qlimit 50  bandwidth   30K
> queue download on $if_intbandwidth 8800K
>queue ack  parent download qlimit 50  bandwidth  100K
>queue fast parent download qlimit 50  bandwidth  200K
>queue bulk parent download qlimit 50  bandwidth 8000K default
>queue slow parent download qlimit 50  bandwidth  500K

While the manual suggests it works like this, I've only got it working
close to how I expect when I set "max" on the queues. I don't know whether
that's a bug or simply lack of fully understanding it on my part, though.

Re: PF: can't make queueing and priority work as expected

[sven falempin]

On Thu, Jan 7, 2016 at 1:28 PM, Marko Cupać  wrote:

> Hi,
>
> I am setting up gateway for a small network which has two main types of
> traffic: p2p and http(s). The idea is to give p2p traffic all the
> available bandwidth until there is http(s) traffic, in which case p2p
> should be throttled down and http(s) should be given all the available
> bandwidth.
>
> The problem is that p2p does not get throttled down when http(s) is on
> wire. I spent days re-reading QUEUEING section of pf.conf and
> chapter #7 of 3rd edition of "Book of PF" but I still couldn't make it
> work. From 'systat queues' I can conclude that traffic seem to be
> assigned to appropriate queues, but queue bandwidth does not seem to be
> respected. What am I doing wrong?
>
> Snapshot of 'systat queues' and active pf.conf below:
>
> QUEUE BW SCH P  PKTS BYTES DROP_P DROP_B QLEN P/S  B/S
> upload on pppoe 860K   0 0  0  00   00
>  ack 10K228K   12M  0  00 364  20K
>  fast20K  60  5397  0  00   00
>  bulk   800K 263  110K  0  00 0.6  417
>  slow30K   34234   35M  0  00  59  67K
> download on re2   8M   0 0  0  00   00
>  ack100K   18314 1205K  0  00  34 2388
>  fast   200K  51 15491  0  00   00
>  bulk 8M   29014   41M  0  00  57  85K
>  slow   500K317K  457M  0  00 523 771K
>
>
> # INTERFACE MACROS
> if_int  = "re2"
> if_ext  = "pppoe0"
>
> # HOST MACROS
> efreet= "{ 192.168.33.20 }"
> rpi   = "{ 192.168.33.22 }"
>
> # PORT MACROS
> p2p = "{ 1:65535 }"
> prpi= "{ 4:40100 }"
> ipsec   = "{ 500 4500 }"
> web = "{ 80 443 }"
> mail= "{ 25 110 143 587 993 995 }"
> xmpp= "{ 5222 }"
>
> # TABLES
> tablepersist
>
> # RUNTIME OPTIONS
> set ruleset-optimization none
> set loginterface $if_ext
>
> # QUEUES
> queue upload  on $if_ext bandwidth  860K
>queue ack  parent upload   qlimit 50  bandwidth   10K
>queue fast parent upload   qlimit 50  bandwidth   20K
>queue bulk parent upload   qlimit 50  bandwidth  800K default
>queue slow parent upload   qlimit 50  bandwidth   30K
> queue download on $if_intbandwidth 8800K
>queue ack  parent download qlimit 50  bandwidth  100K
>queue fast parent download qlimit 50  bandwidth  200K
>queue bulk parent download qlimit 50  bandwidth 8000K default
>queue slow parent download qlimit 50  bandwidth  500K
>
> # QUICKS AND BLOCKS
> block log quick inet6
> block log quick from 
> antispoof for $if_int
> antispoof for $if_ext
>
> # SCRUB & NAT
> match in all scrub ( no-df random-id max-mss 1440 )
> match out on egress inet from $if_int:network to any nat-to ($if_ext:0)
>
> # SHAPING
> match proto icmp   set ( queue   fast   prio   4 )
> match proto tcp  to port 22set ( queue ( fast ack ) prio ( 4 5 ) )
> match proto tcp  to port 53set ( queue ( fast ack ) prio ( 4 5 ) )
> match proto udp  to port 53set ( queue   fast   prio   4 )
> match proto tcp  to port $web  set ( queue ( bulk ack ) prio ( 3 5 ) )
> match proto tcp  to port $mail set ( queue ( bulk ack ) prio ( 3 5 ) )
> match proto tcp  to port $xmpp set ( queue ( bulk ack ) prio ( 3 5 ) )
> match proto tcp  to port $p2p  set ( queue ( slow ack ) prio ( 0 5 ) )
> match proto udp  to port $p2p  set ( queue   slow   prio   0 )
>
> # RULES
> block log all
>
> pass in  on $if_int inet proto icmp from $if_int:network to any
> pass in  on $if_int inet proto tcp  from $if_int:network to any \
>  port 22
> pass in  on $if_int inet proto tcp  from $if_int:network to any \
>  port 53
> pass in  on $if_int inet proto udp  from $if_int:network to any \
>  port 53
> pass in  on $if_int inet proto tcp  from $if_int:network to any \
>  port $web
> pass in  on $if_int inet proto tcp  from $if_int:network to any \
>  port $mail
> pass in  on $if_int inet proto tcp  from $if_int:network to any \
>  port $xmpp
> pass in  on $if_int inet proto tcp  from $if_int:network to any \
>  port $p2p
> pass in  on $if_int inet proto udp  from $if_int:network to any \
>  port $p2p
>
> pass out on $if_ext inet proto icmp from ($if_ext:0) to any
> pass out on $if_ext inet proto tcp  from ($if_ext:0) to any \
>  port 22
> pass out on $if_ext inet proto tcp  from ($if_ext:0) to any \
>  port 53
> pass out on $if_ext inet proto udp  from ($if_ext:0) to any \
>  port 53
> pass out on $if_ext inet proto tcp  from ($if_ext:0) to any \
>  port $web
> pass out on $if_ext inet proto tcp  from ($if_ext:0) to any \
>  port $mail
> pass out on $if_ext inet proto tcp  from ($if_ext:0) to any \
>  port $xmpp
> pass out on