Hi Stuart,
The ftp server is on em0 side and the initiator is 192.168.110.2 and
is on em1 side:
86.55.8.30(ftp sever) ---- em0(192.168.100.233)-OpenBSD
box-(192.168.110.1)em1 ---- 192.168.110.2(initiator).
I have modified my rules according to Giancarlo's recomandation and
removed the "flags any" statement :
~# grep -v \# /etc/pf.conf | grep -v ^$
altq on em1 cbq bandwidth 100Mb queue { ftp,other }
queue ftp on em1 bandwidth 1Mb priority 0 cbq(ecn)
queue other on em1 bandwidth 99Mb priority 1 cbq(ecn,default)
nat on em0 from 192.168.110.2 -> 192.168.100.233
pass in quick on em0 from 86.55.8.30 queue ftp
pass log all
I started a new ftp sesion from 192.168.110.2 and this is what pf logged :
1227628112.272991 rule 1/(match) pass in on em1: 192.168.110.2.3061 >
86.55.8.30.21: [|tcp] (DF)
1227628112.273000 rule 1/(match) pass out on em0:
192.168.100.233.50909 > 86.55.8.30.21: [|tcp] (DF)
1227628112.297853 rule 1/(match) pass in on em1: 192.168.110.2.3063 >
86.55.8.30.49723: [|tcp] (DF)
1227628112.297861 rule 1/(match) pass out on em0:
192.168.100.233.62166 > 86.55.8.30.49723: [|tcp] (DF)
1227628112.307175 rule 1/(match) pass in on em1: 192.168.110.2.3065 >
86.55.8.30.18917: [|tcp] (DF)
1227628112.307183 rule 1/(match) pass out on em0:
192.168.100.233.53378 > 86.55.8.30.18917: [|tcp] (DF)
1227628136.780448 rule 1/(match) pass in on em1: 192.168.110.2.3069 >
86.55.8.30.14919: [|tcp] (DF)
1227628136.780457 rule 1/(match) pass out on em0:
192.168.100.233.53489 > 86.55.8.30.14919: [|tcp] (DF)
1227628136.807671 rule 1/(match) pass in on em1: 192.168.110.2.3071 >
86.55.8.30.2344: [|tcp] (DF)
1227628136.807678 rule 1/(match) pass out on em0:
192.168.100.233.62091 > 86.55.8.30.2344: [|tcp] (DF)
1227628139.661602 rule 1/(match) pass in on em1: 192.168.110.2.3073 >
86.55.8.30.30207: [|tcp] (DF)
1227628139.661610 rule 1/(match) pass out on em0:
192.168.100.233.56393 > 86.55.8.30.30207: [|tcp] (DF)
1227628139.675897 rule 1/(match) pass in on em1: 192.168.110.2.3075 >
86.55.8.30.47420: [|tcp] (DF)
1227628139.675904 rule 1/(match) pass out on em0:
192.168.100.233.54381 > 86.55.8.30.47420: [|tcp] (DF)
1227628144.398885 rule 1/(match) pass in on em1: 192.168.110.2.3077 >
86.55.8.30.22614: [|tcp] (DF)
1227628144.398893 rule 1/(match) pass out on em0:
192.168.100.233.56068 > 86.55.8.30.22614: [|tcp] (DF)
0 packets hit the first rule :
~# pfctl -sr -vv
@0 pass in quick on em0 inet from 86.55.8.30 to any flags S/SA keep
state queue ftp
[ Evaluations: 2074 Packets: 0 Bytes: 0 States: 0 ]
[ Inserted: uid 0 pid 19345 State Creations: 0 ]
@1 pass log all flags S/SA keep state
[ Evaluations: 2076 Packets: 56520 Bytes: 52131049 States: 63 ]
[ Inserted: uid 0 pid 19345 State Creations: 1390 ]
I replaced "pass in quick on em0 from 86.55.8.30 queue ftp" with "pass
out quick on em1 from 86.55.8.30 queue ftp" and made
the test again :
tcpdump -nettipflog0 host 86.55.8.30 :
1227630608.794664 rule 1/(match) pass in on em1: 192.168.110.2.3433 >
86.55.8.30.21: [|tcp] (DF)
1227630608.794672 rule 1/(match) pass out on em0:
192.168.100.233.57968 > 86.55.8.30.21: [|tcp] (DF)
1227630608.923694 rule 1/(match) pass in on em1: 192.168.110.2.3435 >
86.55.8.30.63973: [|tcp] (DF)
1227630608.923701 rule 1/(match) pass out on em0:
192.168.100.233.55330 > 86.55.8.30.63973: [|tcp] (DF)
Again, 0 packets hit the first rule :
:~# pfctl -sr -vv
@0 pass out quick on em1 inet from 86.55.8.30 to any flags S/SA keep
state queue ftp
[ Evaluations: 267 Packets: 0 Bytes: 0 States: 0 ]
[ Inserted: uid 0 pid 10640 State Creations: 0 ]
@1 pass log all flags S/SA keep state
[ Evaluations: 269 Packets: 37149 Bytes: 41268847 States: 96 ]
[ Inserted: uid 0 pid 10640 State Creations: 96 ]
According to what pf logged, it permits traffic incoming on em1 to
86.55.8.30 and outgoing trafic on em0 to 86.55.8.30. So I modified
my designation rule to :
"pass out quick on em0 to 86.55.8.30 queue ftp"
And traffic started hitting the first rule :
~# pfctl -sr -vv
@0 pass out quick on em0 inet from any to 86.55.8.30 flags S/SA keep
state queue ftp
[ Evaluations: 374 Packets: 3452 Bytes: 3658907 States: 2 ]
[ Inserted: uid 0 pid 753 State Creations: 2 ]
@1 pass log all flags S/SA keep state
[ Evaluations: 374 Packets: 3564 Bytes: 3694526 States: 28 ]
[ Inserted: uid 0 pid 753 State Creations: 32 ]
I made another test and designated the queue on the same interface
where ALTQ policy was aplied :
"pass in quick on em1 to 86.55.8.30 queue ftp"
Traffic also got matched by this rule succesfully :
~# pfctl -sr -vv
@0 pass in quick on em1 inet from any to 86.55.8.30 flags S/SA keep
state queue ftp
[ Evaluations: 806 Packets: 2186 Bytes: 2294367 States: 2 ]
[ Inserted: uid 0 pid 27436 State Creations: 2 ]
@1 pass log all flags S/SA keep state
[ Evaluations: 806 Packets: 2223 Bytes: 2298770 States: 22 ]
[ Inserted: uid 0 pid 27436 State Creations: 22 ]
So it seems http://www.openbsd.org/faq/pf/queueing.html is outdated
since it says queue designation should take place on a different
interface than the one where queueing policy is applied.
The result of the above queue designation rules was a constantly 1mbit
download rate for the initiator (192.168.110.2). So my understanding
is that ALTQ is per flow/session connection oriented rather than per
source/destination.
In fact it seems the ALTQ queueing policy is a per-session per-initiator since:
"pass in quick on em0 from 86.55.8.30 queue ftp" and "pass out quick
on em1 from 86.55.8.30 queue ftp" matched nothing
"pass out quick on em0 to 86.55.8.30 queue ftp" , "pass in quick on
em1 to 86.55.8.30 queue ftp" (which looks more like ingress rate
limiting to me) matched trafic and it is the trafic back to the
initiator that got queued.
So let's say I want to implement two different queueing policies : one
from 192.168.110.2 to 86.55.8.30 (upload) and one from 86.55.8.30 to
192.168.110.2 (download).
In order to test asymmetric queueing and confirm my understanding of
ALTQ's per session/per initiator behavior, I installed a ftp service
on 192.168.110.2 and used another machine from the lan as a session
initiator (192.168.6.216) and generated bidirectional traffic between
them. 192.168.6.216 is two hops away from the openbsd box :
192.168.6.216(initiator) ---- eth0(192.168.6.1)Linux
Router(192.168.100.1)eth1 ---- em0(192.168.100.233)-OpenBSD
box-(192.168.110.1)em1 ---- 192.168.110.2(ftp test server)
So, according to the results I got from the previous tests, in order
to apply downstream queueing to particular destination for a
particular host, I had to :
match inbound traffic to the destination on the interface on the host side
or
match outbound trafic to the destination on the interface on the
destination side
So for 192.168.6.216(initiator)'s downstream queueing I set up the
following pf rules :
~# grep -v \# /etc/pf.conf | grep -v ^$
altq on em0 cbq bandwidth 100Mb queue { ftp,other }
queue ftp on em0 bandwidth 2Mb priority 0 cbq(ecn)
queue other on em0 bandwidth 98Mb priority 1 cbq(ecn,default)
altq on em1 cbq bandwidth 100Mb queue { ftp,other }
queue ftp on em1 bandwidth 1Mb priority 0 cbq(ecn)
queue other on em1 bandwidth 99Mb priority 1 cbq(ecn,default)
nat on em0 from 192.168.110.2 to <test> -> 192.168.100.233
table <test> { any , !192.168.0.0/16 }
pass in quick on em0 to 192.168.110.2 queue ftp
pass log all
Then started a ftp session from 192.168.6.216 with 192.168.110.2 on
port 2121 but nothing got matched, it was only a few packets at
beginning of the session :
~# pfctl -sr -vv
@0 pass in quick on em0 inet from any to 192.168.110.2 flags S/SA keep
state queue ftp
[ Evaluations: 7777 Packets: 39 Bytes: 2693 States: 1 ]
[ Inserted: uid 0 pid 25740 State Creations: 1 ]
@1 pass log all flags S/SA keep state
[ Evaluations: 7776 Packets: 270861 Bytes: 232036862 States: 93 ]
[ Inserted: uid 0 pid 25740 State Creations: 117 ]
pf log caught nothing :
~# tcpdump -nettipflog0 | grep -E 192.168.110.2\|192.168.6.216
tcpdump: listening on pflog0, link-type PFLOG
^C
44 packets received by filter
0 packets dropped by kernel
tcpdump on em0 caught all the traffic, this is a snip from the session :
tcpdump -vv -n -i em0 host 192.168.110.2 and host 192.168.6.216
22:09:50.278141 192.168.6.216.39464 > 192.168.110.2.2120: . [tcp sum
ok] 1:1(0) ack 133947393 win 65535 (DF) [tos 0x8] (ttl 63, id 416, len
40)
22:09:50.278240 192.168.110.2.2120 > 192.168.6.216.39464: P
133954409:133955585(1176) ack 1 win 65535 (DF) (ttl 127, id 43540, len
1216)
22:09:50.278365 192.168.110.2.2120 > 192.168.6.216.39464: .
133955585:133957045(1460) ack 1 win 65535 (DF) (ttl 127, id 43541, len
1500)
22:09:50.278389 192.168.6.216.39464 > 192.168.110.2.2120: . [tcp sum
ok] 1:1(0) ack 133950313 win 65535 (DF) [tos 0x8] (ttl 63, id 417, len
40)
22:09:50.278391 192.168.6.216.39464 > 192.168.110.2.2120: . [tcp sum
ok] 1:1(0) ack 133951489 win 65535 (DF) [tos 0x8] (ttl 63, id 418, len
40)
22:09:50.278494 192.168.110.2.2120 > 192.168.6.216.39464: .
133957045:133958505(1460) ack 1 win 65535 (DF) (ttl 127, id 43542, len
1500)
22:09:50.278496 192.168.110.2.2120 > 192.168.6.216.39464: P
133958505:133959681(1176) ack 1 win 65535 (DF) (ttl 127, id 43543, len
1216)
22:09:50.278613 192.168.110.2.2120 > 192.168.6.216.39464: .
133959681:133961141(1460) ack 1 win 65535 (DF) (ttl 127, id 43544, len
1500)
22:09:50.278736 192.168.110.2.2120 > 192.168.6.216.39464: .
133961141:133962601(1460) ack 1 win 65535 (DF) (ttl 127, id 43545, len
1500)
22:09:50.278762 192.168.6.216.39464 > 192.168.110.2.2120: . [tcp sum
ok] 1:1(0) ack 133954409 win 65535 (DF) [tos 0x8] (ttl 63, id 419, len
40)
I modified the queue designation rule to "pass out quick on em1 to
192.168.110.2 queue ftp" but got the same result; nothing got
matched/queued.
So then I modified it to "pass out quick on em0 to 192.168.6.216
queue ftp" and traffic got matched and queued at a 2mbits rate.
"pass out quick on em0 from 192.168.110.2 queue ftp", "pass in quick
on em1 from 192.168.110.2 queue ftp" , "pass in quick on em1 to
192.168.6.216" queue ftp also worked. PF log caught nothing for these
rules. Now it seems ALTQ does per-source/per-destination queue
policing.
Now for the downstream of 192.168.110.2, I set up a ftp server on
192.168.6.216 and applied the following rules :
"pass out quick on em1 to 192.168.110.2 queue ftp" - nothing got matched
"pass out quick on em1 from 192.168.6.216 queue ftp" - nothing got matched
"pass in quick on em1 to 192.168.6.216 queue ftp" - traffic got
matched, got a download rate of 1mbits on 192.168.110.2
"pass out quick on em0 to 192.168.6.216 queue ftp" - traffic got
matched, got a download rate of 1mbits on 192.168.110.2
"pass in quick on em1 from 192.168.110.2 queue ftp" - traffic got
matched, got a download rate of 1mbits on 192.168.110.2
"pass out quick on em0 from 192.168.110.2 queue ftp" - traffic got
matched, got a download rate of 1mbits on 192.168.110.2
Well, at this point I can't really get much sense of this results...
hopefully someone will :)
Thank you and regards.
On Mon, Nov 24, 2008 at 10:46 PM, Stuart Henderson <[EMAIL PROTECTED]> wrote:
> On 2008-11-24, Damian Higgins <[EMAIL PROTECTED]> wrote:
>> However, I did try to apply the queue designation on em1 interface
>> with the statement "pass out quick on em1 from 86.55.8.30 flags any
>> queue ftp" but got the same result. Nothing got matched. I'll paste
>> the outputs tomorrow because I don't have access to that particular
>> machine right now.
>
> try just "pass log all" and "tcpdump -nettipflog0". it will show
> the interface and direction of all the packets which create state.
> nice and easy to convert this to rules which will get matched.
