I followed your suggestions regarding nc.
Also this time I had 4 nics - 2 compex nics (the same model as
previously - RL100-ATX, running under wb driver) and my two old
trusty cards - 3c905 and 3c905b.
First - both 3com cards solved any anomalies I've had - everything
worked beautifully and as intended. Unfortunately, changing
nics is not an option atm.
But compex cards had similar problems as previously.
I tested with ftp, sftp, http, and nc (reading from
/dev/zero as well as from a prepared file).
Simple test setup was as follows:
#-----------------------------
test="192.168.100.75"
int_if="wb0"
#int_if="xl0"
set skip on lo0
scrub in on $int_if
altq on $int_if bandwidth 8mb cbq queue {std, ftp, pri, ack}
queue std on $int_if bandwidth 2mb cbq(default)
queue ftp on $int_if bandwidth 2mb cbq
queue pri on $int_if bandwidth 2mb cbq
queue ack on $int_if bandwidth 2mb cbq
block drop all
pass in on $int_if inet from $test to any keep state queue (std, ack)
pass out on $int_if inet from any to any keep state queue (std, ack)
#12345 - nc ; 40000:40050 - vsftpd
pass in on $int_if inet proto tcp from $test to any port \
{21,80,12345,40000:40050} keep state queue (ftp, ack)
pass out on $int_if inet proto tcp from any to $test port 12345 \
keep state queue (ftp, ack)
pass in on $int_if inet proto tcp from $test to any port \
{22} keep state queue (pri, ack)
#-----------------------------
While using ftp / sftp I've never managed to achieve more
than 15-20 kilobytes/s - exactly as in my previous post. NC reading
from regular file, to /dev/null on the other machine - in *most*
cases behave in the same improper way, but sometimes after simple
ifconfig operation (setting ip, up, down) it worked properly. Directly
after tests with other programs (ftp, ..) it never worked properly.
NC reading from /dev/null worked *usually* well after mentioned
ifconfing, but always poorly after ftp, sftp, etc.
Here are few pfctl -vvsq outputs (pf config as above in each case):
ftp:
queue root_wb0 bandwidth 8Mb priority 0 cbq( wrr root ) {std, ftp, pri,
ack}
[ pkts: 1255 bytes: 1629170 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 0 ]
[ measured: 10.4 packets/s, 122.72Kb/s ]
queue std bandwidth 2Mb cbq( default )
[ pkts: 84 bytes: 7211 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 0 ]
[ measured: 0.0 packets/s, 10.50 b/s ]
queue ftp bandwidth 2Mb
[ pkts: 1066 bytes: 1612720 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 2 ]
[ measured: 10.1 packets/s, 122.52Kb/s ]
queue pri bandwidth 2Mb
[ pkts: 16 bytes: 1385 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 0 ]
[ measured: 0.0 packets/s, 0 b/s ]
queue ack bandwidth 2Mb
[ pkts: 89 bytes: 7854 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 0 ]
[ measured: 0.3 packets/s, 191.25 b/s ]
sftp quite similar:
queue root_wb0 bandwidth 8Mb priority 0 cbq( wrr root ) {std, ftp, pri,
ack}
[ pkts: 5158 bytes: 7271163 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 0 ]
[ measured: 15.7 packets/s, 184.32Kb/s ]
queue std bandwidth 2Mb cbq( default )
[ pkts: 92 bytes: 7691 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 0 ]
[ measured: 0.0 packets/s, 0 b/s ]
queue ftp bandwidth 2Mb
[ pkts: 1351 bytes: 2044210 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 2 ]
[ measured: 0.0 packets/s, 0 b/s ]
queue pri bandwidth 2Mb
[ pkts: 3526 bytes: 5203444 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 76 ]
[ measured: 15.3 packets/s, 184.08Kb/s ]
queue ack bandwidth 2Mb
[ pkts: 189 bytes: 15818 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 0 ]
[ measured: 0.3 packets/s, 237.60 b/s ]
nc from /dev/zero, after the above:
queue root_wb0 bandwidth 8Mb priority 0 cbq( wrr root ) {std, ftp, pri,
ack}
[ pkts: 6049 bytes: 8563746 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 0 ]
[ measured: 12.9 packets/s, 153.29Kb/s ]
queue std bandwidth 2Mb cbq( default )
[ pkts: 96 bytes: 7931 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 0 ]
[ measured: 0.0 packets/s, 19.20 b/s ]
queue ftp bandwidth 2Mb
[ pkts: 1935 bytes: 2926526 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 2 ]
[ measured: 12.6 packets/s, 153.10Kb/s ]
queue pri bandwidth 2Mb
[ pkts: 3799 bytes: 5610975 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 84 ]
[ measured: 0.0 packets/s, 0 b/s ]
queue ack bandwidth 2Mb
[ pkts: 219 bytes: 18314 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 0 ]
[ measured: 0.2 packets/s, 172.80 b/s ]
But then the only thing I did was: ifconfig wb0 down followed by up,
and I got:
queue root_wb0 bandwidth 8Mb priority 0 cbq( wrr root ) {std, ftp, pri,
ack}
[ pkts: 14900 bytes: 21908759 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 0 ]
[ measured: 164.0 packets/s, 1.99Mb/s ]
queue std bandwidth 2Mb cbq( default )
[ pkts: 103 bytes: 8297 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 0 ]
[ measured: 0.0 packets/s, 0 b/s ]
queue ftp bandwidth 2Mb
[ pkts: 10752 bytes: 16268874 dropped pkts: 0 bytes:
0 ]
[ qlength: 5/ 50 borrows: 0 suspends: 2723 ]
[ measured: 163.9 packets/s, 1.99Mb/s ]
queue pri bandwidth 2Mb
[ pkts: 3800 bytes: 5611066 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 84 ]
[ measured: 0.0 packets/s, 0 b/s ]
queue ack bandwidth 2Mb
[ pkts: 245 bytes: 20522 dropped pkts: 0 bytes:
0 ]
[ qlength: 0/ 50 borrows: 0 suspends: 0 ]
[ measured: 0.1 packets/s, 36 b/s ]
In short - sftp, ftp, and others except certain nc uses - always break
the throughput. Ifconfig can help - but only with nc and generally only
if it reads /dev/zero. But if regular file is used, it's like in
ftp & etc. cases (but excpetions happend few times, and I couldn't find
a way to repeat the steps leading to them). .... It doesn't really make
any sense.
Everything was ran with debug level set to loud - no issues here
either.
As noted previously - 3c905 cards had no problems of whatsoever
in any case. But this compex model have some (unless queueing
is disabled - then it works perfectly too). And in a way that is kinda
hard to explain logically. I didn't even manage to find the steps to
make it always fail or always work, despite many tests. Also,
considering that two different compex cards were tested, it seems
unlikely that they were damaged in some unusual way.
Any other suggestions what could be the problem here - wb driver
problem
(but I guess it's unlikely), incompatiblity of this compex nic with it,
some friction between pf and wb driver maybe ?
(dmesg is the same as previously)
