Hello misc!
I am with a very non expected behaviour. With this simple pf.conf
# pfctl -vnf /etc/pf.conf
set skip on { lo }
queue root on alc0 bandwidth 600M default
pass out on alc0 all flags S/SA set ( queue root )
I got this queue output when running tcpbench in client mode
# pfctl -vvvsq
[ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0
]
[ qlength: 0/ 50 ]
queue root on alc0 bandwidth 600M default qlimit 50
[ pkts: 6099167 bytes: 9233990662 dropped pkts: 0 bytes: 0
]
[ qlength: 0/ 50 ]
[ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0
]
[ qlength: 0/ 50 ]
[ measured: 0.0 packets/s, 0 b/s ]
queue root on alc0 bandwidth 600M default qlimit 50
[ pkts: 6500911 bytes: 9842225822 dropped pkts: 0 bytes: 0
]
[ qlength: 0/ 50 ]
[ measured: 80348.8 packets/s, 973.18Mb/s ]
[ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0
]
[ qlength: 0/ 50 ]
[ measured: 0.0 packets/s, 0 b/s ]
queue root on alc0 bandwidth 600M default qlimit 50
[ pkts: 6902593 bytes: 10450369962 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 ]
[ measured: 80342.6 packets/s, 973.10Mb/s ]
# pfctl -vsr
pass out on alc0 all flags S/SA set ( queue root )
[ Evaluations: 493 Packets: 14082601 Bytes: 13949048492 States: 1
]
[ Inserted: uid 0 pid 3493 State Creations: 1 ]
I've tried with 100M, 200M and 400M, all not shaping.
I've also tried to setup a root queue with 200M and two child: a default
with 1M and the other, referred in the rule, with 100M, also not working.
I am playing with tcpbench and this is the only traffic I really care about
on this machine. I restarted the tcpbench client on this machine every time
I reloaded the testing rule and queue, and even deleted the related states
(or states, in cases that I run tcpbench -b <some alias>), but nothing
leads me to the desired bandwidth shaping.
I am experiencing the same behaviour in a virtual machine under KVM with
PCI Passthrough of an Intel NIC. These are the conf and results from the
virtual machine:
# pfctl -vf /etc/pf.conf
set skip on { lo }
queue std on em0 bandwidth 100M default
pass out on em0 all flags S/SA set ( queue std )
# pfctl -vvvsq
[ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0
]
[ qlength: 0/ 50 ]
queue std on em0 bandwidth 100M default qlimit 50
[ pkts: 1195513815 bytes: 87858084628 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 ]
[ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0
]
[ qlength: 0/ 50 ]
[ measured: 0.0 packets/s, 0 b/s ]
queue std on em0 bandwidth 100M default qlimit 50
[ pkts: 1195734870 bytes: 88192747866 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 ]
[ measured: 44211.0 packets/s, 535.46Mb/s ]
[ pkts: 0 bytes: 0 dropped pkts: 0 bytes: 0
]
[ qlength: 0/ 50 ]
[ measured: 0.0 packets/s, 0 b/s ]
queue std on em0 bandwidth 100M default qlimit 50
[ pkts: 1195960995 bytes: 88535089028 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 ]
[ measured: 44718.0 packets/s, 541.60Mb/s ]
# pfctl -vsr
pass out on em0 all flags S/SA set ( queue std )
[ Evaluations: 2 Packets: 1853414 Bytes: 1708817040 States: 2
]
[ Inserted: uid 0 pid 19622 State Creations: 2 ]
The traffic passes through a Linux box where I have per ip bandwitdh
control (justifying tcpbench -b <alias>), an in house bandwidth controller
(poor man's 'net equalizer'). My intent was to not put a very high load
over this machine by getting close to my real pps and bps and so make my
capacity planing.
What am I doing wrong with these queues?
Thank you all,
Raimundo Santos
Here is my dmesgs, first from the physical machine and after from the
virtual machine:
OpenBSD 5.5 (GENERIC.MP) #315: Wed Mar 5 09:37:46 MST 2014
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8538095616 (8142MB)
avail mem = 8302202880 (7917MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.5 @ 0x9f400 (53 entries)
bios0: vendor American Megatrends Inc. version "0803" date 07/23/2012
bios0: ASUSTeK Computer INC. M4A78LT-M-LE
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S3 S4 S5
acpi0: tables DSDT FACP APIC MCFG OEMB SRAT HPET SSDT
acpi0: wakeup devices PCE2(S4) PCE3(S4) PCE4(S4) PCE5(S4) PCE6(S4) ALAN(S4)
PCE7(S4) PCE9(S4) PCEA(S4) PCEB(S4) PCEC(S4) SBAZ(S4) PS2M(S4) PS2K(S4)
UAR1(S4) P0PC(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD Phenom(tm) II X2 565 Processor, 3415.72 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,ITSC
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache, 6MB 64b/line 48-way L3 cache
cpu0: ITLB 32 4KB entries fully associative, 16 4MB entries fully
associative
cpu0: DTLB 48 4KB entries fully associative, 48 4MB entries fully
associative
cpu0: AMD erratum 721 detected and fixed
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 200MHz
cpu0: mwait min=64, max=64, C-substates=0.0.0.0.0, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD Phenom(tm) II X2 565 Processor, 3415.24 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,ITSC
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache, 6MB 64b/line 48-way L3 cache
cpu1: ITLB 32 4KB entries fully associative, 16 4MB entries fully
associative
cpu1: DTLB 48 4KB entries fully associative, 48 4MB entries fully
associative
cpu1: AMD erratum 721 detected and fixed
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 21, 24 pins
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 14318180 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (P0P1)
acpiprt2 at acpi0: bus -1 (PCE2)
acpiprt3 at acpi0: bus -1 (PCE3)
acpiprt4 at acpi0: bus 2 (PCE4)
acpiprt5 at acpi0: bus 3 (PCE6)
acpiprt6 at acpi0: bus 4 (P0PC)
acpicpu0 at acpi0: PSS
acpicpu1 at acpi0: PSS
aibs0 at acpi0 RTMP RVLT RFAN GGRP GITM SITM
acpibtn0 at acpi0: PWRB
cpu0: 3415 MHz: speeds: 3400 2700 2300 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "AMD RS780 Host" rev 0x00
ppb0 at pci0 dev 1 function 0 vendor "Asustek", unknown product 0x9602 rev
0x00
pci1 at ppb0 bus 1
radeondrm0 at pci1 dev 5 function 0 "ATI Radeon HD 3000" rev 0x00
drm0 at radeondrm0
radeondrm0: apic 2 int 18
ppb1 at pci0 dev 4 function 0 "AMD RS780 PCIE" rev 0x00: msi
pci2 at ppb1 bus 2
re0 at pci2 dev 0 function 0 "Realtek 8168" rev 0x03: RTL8168D/8111D
(0x2800), msi, address 00:e0:4c:68:26:38
rgephy0 at re0 phy 7: RTL8169S/8110S PHY, rev. 2
ppb2 at pci0 dev 6 function 0 "AMD RS780 PCIE" rev 0x00: msi
pci3 at ppb2 bus 3
alc0 at pci3 dev 0 function 0 "Attansic Technology L1C" rev 0xc0: msi,
address f4:6d:04:00:8f:a1
atphy0 at alc0 phy 0: F1 10/100/1000 PHY, rev. 11
ahci0 at pci0 dev 17 function 0 "ATI SBx00 SATA" rev 0x00: apic 2 int 22,
AHCI 1.1
scsibus0 at ahci0: 32 targets
sd0 at scsibus0 targ 0 lun 0: <ATA, ST3500418AS, CC34> SCSI3 0/direct fixed
naa.5000c500120cd875
sd0: 476940MB, 512 bytes/sector, 976773168 sectors
ohci0 at pci0 dev 18 function 0 "ATI SB700 USB" rev 0x00: apic 2 int 16,
version 1.0, legacy support
ohci1 at pci0 dev 18 function 1 "ATI SB700 USB" rev 0x00: apic 2 int 16,
version 1.0, legacy support
ehci0 at pci0 dev 18 function 2 "ATI SB700 USB2" rev 0x00: apic 2 int 17
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "ATI EHCI root hub" rev 2.00/1.00 addr 1
ohci2 at pci0 dev 19 function 0 "ATI SB700 USB" rev 0x00: apic 2 int 18,
version 1.0, legacy support
ohci3 at pci0 dev 19 function 1 "ATI SB700 USB" rev 0x00: apic 2 int 18,
version 1.0, legacy support
ehci1 at pci0 dev 19 function 2 "ATI SB700 USB2" rev 0x00: apic 2 int 19
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "ATI EHCI root hub" rev 2.00/1.00 addr 1
piixpm0 at pci0 dev 20 function 0 "ATI SBx00 SMBus" rev 0x3c: SMI
iic0 at piixpm0
iic0: addr 0x20 01=11 02=28 04=00 05=00 06=00 07=00 09=00 0a=0f 0b=0f 0c=0f
0d=0f 0e=19 0f=8a 10=00 11=00 12=00 13=00 14=00 15=0e 16=3a 17=9c 18=80
19=fb 1a=af 1b=a0 1c=b6 1d=80 1e=0c 1f=01 20=09 21=09 22=09 23=09 24=03
3e=bf words 00=ff11 01=1128 02=28ff 03=ff00 04=0000 05=0000 06=0000 07=00ff
spdmem0 at iic0 addr 0x50: 4GB DDR3 SDRAM PC3-10600
spdmem1 at iic0 addr 0x51: 4GB DDR3 SDRAM PC3-10600
pciide0 at pci0 dev 20 function 1 "ATI SB700 IDE" rev 0x00: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
pcib0 at pci0 dev 20 function 3 "ATI SB700 ISA" rev 0x00
ppb3 at pci0 dev 20 function 4 "ATI SB600 PCI" rev 0x00
pci4 at ppb3 bus 4
ohci4 at pci0 dev 20 function 5 "ATI SB700 USB" rev 0x00: apic 2 int 18,
version 1.0, legacy support
pchb1 at pci0 dev 24 function 0 "AMD AMD64 10h HyperTransport" rev 0x00
pchb2 at pci0 dev 24 function 1 "AMD AMD64 10h Address Map" rev 0x00
pchb3 at pci0 dev 24 function 2 "AMD AMD64 10h DRAM Cfg" rev 0x00
km0 at pci0 dev 24 function 3 "AMD AMD64 10h Misc Cfg" rev 0x00
pchb4 at pci0 dev 24 function 4 "AMD AMD64 10h Link Cfg" rev 0x00
usb2 at ohci0: USB revision 1.0
uhub2 at usb2 "ATI OHCI root hub" rev 1.00/1.00 addr 1
usb3 at ohci1: USB revision 1.0
uhub3 at usb3 "ATI OHCI root hub" rev 1.00/1.00 addr 1
usb4 at ohci2: USB revision 1.0
uhub4 at usb4 "ATI OHCI root hub" rev 1.00/1.00 addr 1
usb5 at ohci3: USB revision 1.0
uhub5 at usb5 "ATI OHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
it0 at isa0 port 0x2e/2: IT8712F rev 8, EC port 0x290
usb6 at ohci4: USB revision 1.0
uhub6 at usb6 "ATI OHCI root hub" rev 1.00/1.00 addr 1
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (5a32fb5b12fd1447.a) swap on sd0b dump on sd0b
drm: initializing kernel modesetting (RS780 0x1002:0x9616 0x1043:0x8388).
radeondrm0: VRAM: 32M 0x00000000C0000000 - 0x00000000C1FFFFFF (32M used)
radeondrm0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
error: [drm:pid0:r600_init_microcode] *ERROR* r600_cp: Failed to load
firmware "radeon-rs780_pfp"
error: [drm:pid0:r600_startup] *ERROR* Failed to load firmware!
error: [drm:pid0:r600_init] *ERROR* disabling GPU acceleration
error: [drm:pid0:radeon_bo_unpin] *ERROR* 0xfffffe821d3ae2b0 unpin not
necessary
error: [drm:pid0:radeon_bo_unpin] *ERROR* 0xfffffe821d3ae2b0 unpin not
necessary
radeondrm0: 1920x1080
wsdisplay0 at radeondrm0 mux 1: console (std, vt100 emulation), using wskbd0
wsdisplay0: screen 1-5 added (std, vt100 emulation)
Virtual machine:
OpenBSD 5.5 (GENERIC) #271: Wed Mar 5 09:31:16 MST 2014
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 1056956416 (1007MB)
avail mem = 1020309504 (973MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfd9a0 (10 entries)
bios0: vendor Bochs version "Bochs" date 01/01/2007
bios0: Bochs Bochs
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC HPET SSDT
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0
mpbios at bios0 not configured
cpu0 at mainbus0: (uniprocessor)
cpu0: QEMU Virtual CPU version 1.1.2, 3003.29 MHz
cpu0:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,POPCNT,NXE,LONG,LAHF
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel
0 wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: irq 11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: irq 10
iic0 at piixpm0
iic0: addr 0x4c 48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000
06=0000 07=0000
iic0: addr 0x4e 48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000
06=0000 07=0000
vga1 at pci0 dev 2 function 0 unknown vendor 0x1234 product 0x1111 rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00: Virtio
Network Device
vio0 at virtio0: RingEventIdx disabled by UKC: address 52:54:00:7a:d5:c2
virtio0: irq 11
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00: Virtio
Block Device
vioblk0 at virtio1
scsibus0 at vioblk0: 2 targets
sd0 at scsibus0 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
sd0: 1024MB, 512 bytes/sector, 2097152 sectors
virtio1: irq 11
em0 at pci0 dev 5 function 0 "Intel 82576" rev 0x01: irq 10, address
00:1b:21:53:51:38
em1 at pci0 dev 6 function 0 "Intel 82576" rev 0x01: irq 11, address
00:1b:21:53:51:39
virtio2 at pci0 dev 7 function 0 "Qumranet Virtio Memory" rev 0x00: Virtio
Memory Balloon Device
viomb0 at virtio2
virtio2: irq 11
virtio3 at pci0 dev 8 function 0 "Qumranet Virtio Network" rev 0x00: Virtio
Network Device
vio1 at virtio3: RingEventIdx disabled by UKC: address 52:54:00:57:b8:fa
virtio3: irq 11
virtio4 at pci0 dev 9 function 0 "Qumranet Virtio Network" rev 0x00: Virtio
Network Device
vio2 at virtio4: RingEventIdx disabled by UKC: address 52:54:00:d0:35:7a
virtio4: irq 10
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: density unknown
fd1 at fdc0 drive 1: density unknown
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
nvram: invalid checksum
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (f03547feaad421a4.a) swap on sd0b dump on sd0b
clock: unknown CMOS layout
