[Iperf-users] Difference between InF and InP

[Bob McMahon via Iperf-users]

Hi All,

There is a subtle but important distinction between iperf 2's in
progress stats. One is called InF for in-flight and computed by taking the
stats from the stack. The second is called InP fo in-progress and taken by
computing the queue depth per Little's Law. The latter is e2e.  The
following two runs illustrate the differences.

First a run where there is a very large send side buffer and
TCP_NOTSENT_LOWAT is not used. In this case the send side socket buffer
should build up, which it does, and the e2e delay is seen per InP but not
InF. That's because InF is from the stack's view of the network while InP
is from the application view. Notice the InP is near the 100Mbytes socket
buffer size while the InF is about 1.5Mbytes

The second run sets TCP_NOTSENT_LOWAT and no the InF and InP are aligned at
about 1.5Mbytes

Run 1: No TCP_NOTSENT_LOWAT

rjmcmahon@fedora:~/Code/inflight/iperf2-code$ iperf -c 192.168.1.35
--trip-times -e -i 1 --sync-transfer-id -w 50m --tcp-write-prefetch 0

Client connecting to 192.168.1.35, TCP port 5001 with pid 240696 (1/0
flows/load)
Write buffer size: 131072 Byte
TCP congestion control using cubic
TOS set to 0x0 (dscp=0,ecn=0) (Nagle on)
TCP window size: 95.4 MByte (WARNING: requested 47.7 MByte)

[  1] local 192.168.1.103%enp4s0 port 44754 connected with 192.168.1.35
port 5001 (trip-times) (sock=3) (icwnd/mss/irtt=14/1448/176) (ct=0.24 ms)
on 2024-03-25 16:50:54.655 (PDT)
[ ID] IntervalTransferBandwidth   Write/Err  Rtry
InF(pkts)/Cwnd/RTT(var)NetPwr
[  1] 0.00-1.00 sec   192 MBytes  1.61 Gbits/sec  1533/061
1538K(1088)/1555K/13234(110) us  15183
[  1] 1.00-2.00 sec  96.4 MBytes   808 Mbits/sec  771/0 0
1667K(1179)/1674K/14273(81) us  7080
[  1] 2.00-3.00 sec   128 MBytes  1.08 Gbits/sec  1026/0 0
1790K(1266)/1801K/15451(109) us  8704
[  1] 3.00-4.00 sec  96.2 MBytes   807 Mbits/sec  770/0 0
1859K(1315)/1875K/16008(84) us  6305
[  1] 4.00-5.00 sec   128 MBytes  1.08 Gbits/sec  1026/0 2
1340K(948)/1365K/11658(133) us  11535
[  1] 5.00-6.00 sec  96.2 MBytes   807 Mbits/sec  770/0 0
1409K(997)/1448K/12301(98) us  8205
[  1] 6.00-7.00 sec   128 MBytes  1.08 Gbits/sec  1026/0 0
1474K(1043)/1530K/13059(123) us  10298
[  1] 7.00-8.00 sec  96.2 MBytes   807 Mbits/sec  770/0 0
1534K(1085)/1575K/13494(111) us  7479
[  1] 8.00-9.00 sec   128 MBytes  1.08 Gbits/sec  1027/0 0
1602K(1133)/1614K/13807(132) us  9749
[  1] 9.00-10.00 sec  96.2 MBytes   807 Mbits/sec  770/0 0
1593K(1127)/1636K/13980(130) us  7219
[  1] 10.00-10.57 sec   128 KBytes  1.83 Mbits/sec  1/0 0
 0K(0)/1665K/14358(111) us  15.90
[  1] 0.00-10.57 sec  1.16 GBytes   941 Mbits/sec  9490/063
 0K(0)/1665K/14358(111) us  8193

root@rpi5-35:~# iperf -s -i 1 -e

Server listening on TCP port 5001 with pid 35250
Read buffer size:  128 KByte (Dist bin width=16.0 KByte)
TCP congestion control default cubic
TCP window size:  128 KByte (default)

[  1] local 192.168.1.35%eth0 port 5001 connected with 192.168.1.103 port
44754 (trip-times) (sock=4) (peer 2.2.0-rc) (icwnd/mss/irtt=14/1448/182) on
2024-03-25 16:50:54.655 (PDT)
[ ID] IntervalTransferBandwidthBurst Latency
avg/min/max/stdev (cnt/size) inP NetPwr  Reads=Dist
[  1] 0.00-1.00 sec   112 MBytes   941 Mbits/sec
 451.474/1.878/839.327/236.465 ms (897/131093)  171 MByte 260
 22019=21972:11:11:8:0:2:2:13
[  1] 1.00-2.00 sec   112 MBytes   941 Mbits/sec
 709.032/560.182/839.576/80.886 ms (898/131052) 69.9 MByte 166
 22841=22836:3:2:0:0:0:0:0
[  1] 2.00-3.00 sec   112 MBytes   942 Mbits/sec
 690.533/559.994/839.511/79.692 ms (897/131201) 90.0 MByte 170
 18997=18993:4:0:0:0:0:0:0
[  1] 3.00-4.00 sec   112 MBytes   942 Mbits/sec
 708.975/559.939/839.014/80.667 ms (898/131063) 69.8 MByte 166
 19325=19325:0:0:0:0:0:0:0
[  1] 4.00-5.00 sec   112 MBytes   941 Mbits/sec
 689.814/559.899/838.858/79.718 ms (898/131053) 90.0 MByte 171
 19552=19552:0:0:0:0:0:0:0
[  1] 5.00-6.00 sec   112 MBytes   941 Mbits/sec
 709.082/559.756/838.974/80.714 ms (898/131053) 69.8 MByte 166
 19050=19030:4:0:0:1:0:1:14
[  1] 6.00-7.00 sec   112 MBytes   941 Mbits/sec
 690.027/559.817/839.295/79.776 ms (898/131050) 90.0 MByte 171
 19573=19570:2:1:0:0:0:0:0
[  1] 7.00-8.00 sec   112 MBytes   942 Mbits/sec
 709.150/560.197/839.131/80.593 ms (898/131056) 69.8 MByte 166
 20616=20608:4:3:1:0:0:0:0
[  1] 8.00-9.00 sec   112 MBytes   941 Mbits/sec
 690.031/560.092/839.122/79.848 ms (898/131050) 90.0 MByte 171
 18866=18864:2:0:0:0:0:0:0
[  1] 9.00-10.00 sec   112 MBytes   942 Mbits/sec
 709.217/560.065/838.999/80.484 ms (898/131056) 69.8 MByte 166
 18885=18882:2:1:0:0:0:0:0
[  1] 10.00-10.57 sec  64.0 MBytes   941 Mbits/sec
 

[Iperf-users] iperf 2-2-0-rc (release candidate)

[Bob McMahon via Iperf-users]

Hi All,

iperf 2 has a release candidate ready for users to try. It's on sourceforge
 as iperf 2-2-0-rc. Please file
tickets on sourceforge  as well.

Release notes:

2.2.0 (as of March 14th, 2024
--
o new ./configure --enable-summing-debug option to help with summing debug
o select ahead of writes slow down UDP performance. support ./configure
--disable-write-select
o support fo -b 0 with UDP, unlimited load or no delay between writes
o support for --sync-transfer-id so client and server will match the ids
and give a remap message
o support --dscp command line option
o support for application level retries and minimum retry interval of the
TCP connect() syscall via --connect-retry-time and --connect-retry-timer,
repsectively
o support for --ignore-shutdown so test will end on writes vs the BDP drain
and TCP close/shutdown, recommended not to use this but in rare cases
o support for --fq-rate-step and --fq-rate-step-interval
o CCAs per --tcp-cca, --tcp-congestion, etc neeed to be case sensitive
o support for both packets and bytes inflight taken from tcp_info struct
amd pkt calc of (tcp_info_buf.tcpi_unacked - tcp_info_buf.tcpi_sacked -
tcp_info_buf.tcpi_lost + tcp_info_buf.tcpi_retrans)
o man page updates and -h to reflect new options, better descriptions
o lots of work around summing with parallel threads, new implementation
based on interval or slot counters, hopefully should work reliably
o --bounceback tests are much more reliable and robust
o Improve event handling around select timeouts, helps with larger -P
values and summing
o use the getsockopt IP_TOS for the displayed output, warn when set and get
don't match
o better tos byte output, include dscp and ecn fields individually
o better tos setting code for both v6 and v4, so they behave the same
around checks and warnings
o much better NULL events to help with reporter processing even when
traffic is not flowing
o support for a new string report
o python flows work around CDF based tests
o rate limit fflush calls to a max of one every millisecond or 1000 per sec
o remove superfulous fflush calls
o reports when P = 1 and --sum-only need sum outputs
o enable summing with --incr-dstip
o add macro TIME_GET_NOW to set a struct timeval in a portable manner
o code readability improvements with enums, bools, etc.
o fix for TCP rate limited and -l less than min burst size
o only use linux/tcp.h when absolutely needed, otherwise use netinet/tcp.h
o print bounceback OWD tx/rx in interval reports
o add flows Makefiles for tarball or make dist-all
o support interval reports for bounceback histograms
o support for TCP working loads and UDP primary flows, including UDP
isochronous, per ticket 283
o fix working-load with isoch so working-load streams are capacity seeking
o exit when CCA not supported or read of the current CCA doesn't match
requested CCA
o add more make check tests
o add support for omit string (omit code not ready for this release)
o pyflows qdisc settings and outputs
o add first send pacing with --tx-starttime so listener threads udp_accept
has time to perform udp_accept() between the client threads
o adjust the sender time per the client delay and the client first write,
i.e. subtract out this delay in the calculations
o fixes for small packets and --tx-starttime
o use more modern multicast socket options (now in
src/iperf_multicast_api.c)
o warn on bind port not sent with --incr-srcport
o display fq-rate values in outputs when --fq-rate is used
o add support for --test-exchange-timeout
o fixes around wait_tick
o add support for TCP_TX_DELAY via --tcp-tx-delay  option on both
client and server
o pass the CCA from client to server
o support burst-size with different write sizes and don't require
--burst-period
o output traffic thread send scheduling error stats in final ouput
o output clock unsync stats with --bounceback
o add warn message on MSG_CTRUNC
o UDP select fixes
o enable TCP_NOTSENTLOWAT and set to a default small value with
--tcp-write-times
o default histogram max binning to 10 seconds
o add a max timestamp to histogram outputs so user can find packets in
pcaps or equivalent
o autoconf change for struct ip_mreqn
o print errno on writen fail

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return