On Fri, Feb 2, 2024 at 1:21 AM Scheffenegger, Richard <rsch...@freebsd.org> wrote:
> > Hi, > > We have run a test for a RPC workload with 1MB IO sizes, and collected the > tcp_default_output() len(gth) during the first pass in the output loop. > > In such a scenario, where the application frequently introduces small > pauses (since the next large IO is only sent after the corresponding > request from the client has been received and processed) between sending > additional data, the current TSO limit of 64kB TSO maximum (45*1448 in > effect) requires multiple passes in the output routine to send all the > allowable (cwnd limited) data. > > I'll try to get a data collection with better granulariy above 90 000 > bytes - but even here the average strongly indicates that a majority of > transmission opportunities are in the 512 kB area - probably also having to > do with LRO and ACK thinning effects by the client. > > With other words, the tcp output has to run about 9 times with TSO, to > transmit all elegible data - increasing the FreeBSD supported maximum TSO > size to what current hardware could handle (256kB..1MB) would reduce the > CPU burden here. > > > Is increasing the sofware supported TSO size to allow for what the NICs > could nowadays do something anyone apart from us would be interested in (in > particular, those who work with the drivers)? > Reposted after joining freebsd-net@... A factor here is the if_hw_tsomaxsegcount limit. For example, a 1Mbyte NFS write request or read reply will result in a 514 element mbuf chain. Each of these (mostly 2K mbuf clusters) are non-contiguous data segments. (I suspect most NICs do not handle this many segments well, if at all.) The NFS code does know how to use M_EXTPG mbufs (for NFS over TLS, for the ktls), but I do not know what it would take to make these work for non-KTLS TSO? I do not know how the TSO loop in tcp_output handles M_EXTPG mbufs. Does it assume each M_EXTPG mbuf is one contiguous data segment? I do see that ip_output() will call mb_unmapped_to_ext() when the NIC does not have IFCAP_MEXTPG set. (If IFCAP_MEXTPG is set, do the pages need to be contiguous so that it can become a single contiguous data segment for TSO or ???) If TSO and the code beneath it (NIC and maybe mb_unmapped_to_ext() being called) were to all work ok for M_EXTPG mbufs, it would be easy to enable that for NFS (non-TLS case). I do not want to hijack this thread, but do others know how TSO interacts with M_EXTPG mbufs? rick > Best regards, > > Richard > > > > > tso size (transmissions < 1448 would not be accounted here at all) > > # count > > <1000 0 > <2000 23 > <3000 111 > <4000 40 > <5000 30 > <7000 14 > <8000 134 > <9000 442 > <10000 9396 > <20000 46227 > <30000 25646 > <40000 33060 > <60000 23162 > <70000 24368 > <80000 19772 > <90000 40101 > >=90000 75384169 > Average: 578844.44 > > CAUTION: This email originated from outside of the University of Guelph. > Do not click links or open attachments unless you recognize the sender and > know the content is safe. If in doubt, forward suspicious emails to > ith...@uoguelph.ca. > >
