Ohh, Forgot to mention.. PF by default sets good ToS values on its CARP
heartbeats, but we use HP Procurve switches with DiffServ enabled.
By default with HP, HP maps the ToS value that PF uses for CARP by default,
into a low priority CoS queue! Yes really ;) Don't you just love HP. And on
many HP switches, you cannot modify this DiffServ <-> CoS mapping.
So the suggestion at the bottom is just to set a ToS that HP switches will
prioritise..
Have fun, all the best.
Andy Lemin
On Wed, Jun 15, 2016 at 8:18 PM, Andy Lemin <a...@brandwatch.com> wrote:
> Peter is quite right, to add some examples to his suggestion;
>
> tcpdump -nettti pflog0 <- Shows only dropped packets
> tcpdump -nettti em0 <- Shows all packets on the interface, including ToS
> values and VLAN ID etc.
> tcpdump -nettti vlanX <- Shows only packets on the VLAN without the extra
> info.
>
> Sure you can figure out the rest..
>
> There are also a few caveats to writing good PF QoS rules that some are
> not aware off. For example the PRIO value is copied into the VLAN header as
> the CoS value, but if it is an untagged VLAN the frame wont have a value as
> their is no VLAN header to store it in. I.e. PRIO is only transitive for
> connected VLAN subnets, beyond the nexthop you cannot control layer 2 CoS,
> only layer 3 QoS is transitive.
>
> Also PRIO is strictly speaking internal to the firewall, and it works for
> both ingress and egress, whereas queuing/shaping is egress only. Best to
> think of it as a priority picker or scheduler. I.e. packets get selected
> from the buffers for processing based on their priority whether they are
> input or output buffers (I am only 90% sure of this, so please correct me
> if I am wrong).
>
> Also common good practice assumes that you would normally want to use two
> prio values; E.g.
>
> pass quick on { $if_ext, $if_DMZ } proto { tcp, udp } from any to {
> $int_ip_dns0 } port { 53 } queue (wan_web,wan_pri) set prio (2,4)
> The first prio (2) is used for the payload packets in the session (ToS not
> set), and the second prio (4) is used for the control packets (ACKs etc
> because they have the ToS set). This again also sets the VLAN CoS correctly
> for each packet type in the same session.
>
> Another thing to be careful of is setting ToS yourself and using PRIO (and
> if using queues too). For example;
>
> match in proto tcp all scrub (no-df max-mss 1460)
>
> match in proto { udp, icmp } all scrub (no-df max-mss 1472)
>
> match out on { $if_ext } proto { tcp, udp } from any to {
> <ext_voipgateways> } scrub (no-df max-mss 1420) set (tos ef, prio 7)
>
> The first two lines are just housekeeping. But the third line will set the
> ToS value EF on every single packet in the session (payload and ACKs) for
> the VoIP traffic. This means that the later pass rules will place all
> voip traffic into 'second' "queue" and second "priority".
>
> And if you didn't spot the clue in the first example, yes, I believe state
> does match returning traffic and does apply the prio to return traffic
> also. But you wont see it with tcpdump unless you are using VLANs to
> inspect the CoS field.
>
>
> In my first example you will also notice I have only one rule that matches
> traffic on both the inside and outside interfaces, so you need to make sure
> you are using the same queue names on both the inside and outside
> interfaces. This is done by adding the "on $if_ext" directive to your
> queues. E.g.
>
> queue ext_root on $if_ext bandwidth 800M
>
> queue qlocal on $if_ext parent ext_root bandwidth 600M
>
> queue local_kern on $if_ext parent qlocal bandwidth 6M min 6M
> burst 10M for 1000ms
>
> queue local_pri on $if_ext parent qlocal bandwidth 60M min 60M
>
> queue local_data on $if_ext parent qlocal bandwidth 510M min 100M
>
> queue qwan on $if_ext parent ext_root bandwidth 190M
>
> queue wan_rt on $if_ext parent qwan bandwidth 38M min 19M burst
> 38M for 5000ms
>
> queue wan_int on $if_ext parent qwan bandwidth 19M min 9M
>
> queue wan_pri on $if_ext parent qwan bandwidth 19M min 10M burst
> 25M for 2000ms
>
> queue wan_vpn on $if_ext parent qwan bandwidth 50M min 25M
>
> queue wan_web on $if_ext parent qwan bandwidth 19M min 10M burst
> 19M for 3000ms
>
> queue wan_dflt on $if_ext parent qwan bandwidth 19M min 10M burst
> 19M for 5000ms
>
> queue wan_bulk on $if_ext parent qwan bandwidth 20M max 50M
> default
>
>
> queue trunk_root on $if_trunk bandwidth 4294M
>
> queue qlocal on $if_trunk parent trunk_root bandwidth 4.1G
>
> queue local_kern on $if_trunk parent qlocal bandwidth 8M min 8M
> burst 8M for 1000ms
>
> queue local_pri on $if_trunk parent qlocal bandwidth 150M min
> 150M burst 200M for 2500ms
>
> queue local_data on $if_trunk parent qlocal bandwidth 4G min 1G
>
> queue qwan on $if_trunk parent trunk_root bandwidth 190M
>
> queue wan_rt on $if_trunk parent qwan bandwidth 38M min 19M
> burst 38M for 5000ms
>
> queue wan_int on $if_trunk parent qwan bandwidth 19M min 9M
>
> queue wan_pri on $if_trunk parent qwan bandwidth 19M min 10M
> burst 25M for 2000ms
>
> queue wan_vpn on $if_trunk parent qwan bandwidth 50M min 25M
>
> queue wan_web on $if_trunk parent qwan bandwidth 19M min 10M
> burst 19M for 3000ms
>
> queue wan_dflt on $if_trunk parent qwan bandwidth 19M min 10M
> burst 19M for 5000ms
>
> queue wan_bulk on $if_trunk parent qwan bandwidth 20M max 50M
> default
>
>
> Another hint, if you are using VLANs to gain the benefit of using the
> priority field in the VLAN header, to maintain your CoS on your layer 2 all
> the way to the client, you should apply your queues to the trunk interface,
> and not each VLAN.
>
> This way you can share your total download transit bandwidth across all
> internal VLANs instead of having to divide it out across the VLANs. Hope
> that makes sense.
>
>
> Just for some extra reading I have attached a paper that explains HFSC
> (the underlying queuing algorithm).
>
> PF Queue "bandwidth" = Link Sharing Service Curve
>
> PF Queue "min" = Realtime Service Curve
>
> PF Queue "burst" = Short-term realtime service curve for delay target
>
> PF queue "for" = time 't' for burst (delay target to be achieved within
> this initial period).
>
> And I have also attached a PF Packet Flow diagram I made (it's not 100%
> accurate, I need to update it but it should help).
>
>
> And lastly just for fun if you are using CARP with busy layer 2 networks
> where you are having problems with heartbeat delay, this solves that, again
> not perfect, just ideas for you;
>
> pass out quick proto carp keep state (no-sync) queue local_kern set (prio
> 7, tos ef)
>
> pass in quick proto carp keep state (no-sync) set (prio 7, tos ef)
>
> pass out quick inet6 proto carp from { fe80::/10 } to { ff00::/8 } keep
> state (no-sync) queue local_kern set (prio 7, tos ef)
>
> pass in quick inet6 proto carp from { fe80::/10 } to { ff00::/8 } keep
> state (no-sync) set (prio 7, tos ef)
>
> pass quick on { $if_pfsync_dev } proto pfsync keep state (no-sync) set
> (prio 7, tos ef)
>
>
> Kind regards, Andy.
>
>
>
>
> On Wed, Jun 15, 2016 at 11:02 AM, Peter N. M. Hansteen <pe...@bsdly.net>
> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> This is one of the cases where the best possible answer is, "tcpdump
>> is your friend".
>>
>> You have outlined a number of test cases. It would be really useful if
>> you try each one of them, and use tcpdump to record and identify the
>> effects of each one. It's worth noting that tcpdump with the right
>> options is able to display information such as the packets's ToS and
>> which rule in the loaded PF rule set the packet matched.
>>
>> If you run those tests properly and report your findings, I'm sure it
>> will be appreciated.
>>
>> - --
>> Peter N. M. Hansteen, member of the first RFC 1149 implementation team
>> http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/
>> "Remember to set the evil bit on all malicious network traffic"
>> delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.
>> iQIcBAEBCAAGBQJXYSecAAoJELJiGF9h4DyeghcP/RZQeJ/4P8cj6DUoBhSw7HuZ
>> q0t8fgnnyfw7ItkWGP6WayW9aT7oMfR9XdgX3jn/jFBLj8aW55K1i/v4PbXFJTkB
>> yjnJ1WJN7fohVSYOYyfnjxCxw2RdGbcVUZpYkFCfIzsKPTxsuJynJyR7i6Ke8dYE
>> 5FiF68oqhKq0yAiHcE91UlMVFH/v8NAy3crzkeK1yjgYK3sU5dVs0H7D/qR8Zlfe
>> fmOO9SqDDcvMMn/7c6bQ9sHKBXSsHizZcf//yuQseSXv9ttsl/3XZyUEhS3fyqNt
>> WKw80vNwQ7MJShOFqjn12G+j72s0kaSkiDEi93rXUZJxsoD28Vn6dyBJhcrWFtfr
>> eEOwuyp82FiNabAvn3StzkKE+cAQ01Kag0hFhgwx/u1sD/9K31B9J8IiMpSIplFV
>> tx4jfWBh1MjadAu3DIvHINYzEPoaju4zUY1mh840l5Wz7SpaBUyeJce0eNtA3n6Z
>> pbpZQsi9mHCP7MOR2b+RvzcjFc4m5XoiLz29aMQDzeLj4GzroY9H0ramWchqbj1y
>> BXKtFNgOglKIkjickdlSnzahFAf53r5T6vv1KY7Ea4Z5PP88e8OiXdcJqiuJlo0T
>> c9VXE5cCy37i21ZPV4YK3LsuiCxMVuGtQ63B/OnP1kX34NVoatpZz6gcx5Y62MWA
>> rsxLSEMFHSJuoJzgGF7j
>> =mgmr
>> -----END PGP SIGNATURE-----
>>
>>
>
--
*Download our latest free guide here
<https://www.brandwatch.com/competitive-intelligence-guide/>*
