The nuttx simulator running tcpecho doesn't appear to have any problem with large TCP sends or large number of large TCP sends. So the problem with tcpecho / tcpblaster on the SAMA5D36 seems to be in the SAMA5 code somewhere.
The GMAC driver fix that I have only works when net logging is turned, and it tested it more thoroughly today. It definitely works. But I don't understand how it works, or why it doesn't work at high speeds. -adam On Tue, Feb 11, 2020 at 6:18 PM Adam Feuer <a...@starcat.io> wrote: > I did some more debugging today. It seems like the GMAC is sending > multiple buffers sometimes when the TCP send is larger than MSS. So it's > not sending not just a single one by itself. When that happens, the GMAC > driver marks the first TX buffer descriptor as used (sent), txdesc->status > = GMACTXD_STA_USED; but subsequent buffers are unmarked until the last, > which is marked with GMACTXD_STA_LAST. Buffers that are not marked never > get returned to the free pool. Hence the GMAC driver leaks buffers, > eventually runs out of TX buffers, stops receiving, can't send, and > eventually the txtimeout happens. Then the driver reinitializes the GMAC > and the TX buffer pool, and frames start sending again. And the process > repeats. > > I have a fix that works at slow speeds– with some logging turned on. It > can do unlimited TCP sends of any size. With logging off, it hangs. I'm > guessing that's because at high speeds it can send longer chains of packets > and buffers all at once. I'm probably not handling chains of more than two > consecutive packets correctly. I'll work on it some more tomorrow. > > -adam > > On Mon, Feb 10, 2020 at 5:38 PM Adam Feuer <a...@starcat.io> wrote: > >> I put a log statement in sam_txdone()... what appears to be happening is >> that the TX descriptor-in-use case is executed at sam_txdone(), line 1364, >> in sam_gmac.c, so the TX buffer fills up. >> >> So it seems like the GMAC driver or hardware gets into a state where it >> is not able to send packets until the driver and hardware is reset. Still >> not sure why that's happening. >> >> -adam >> >> On Mon, Feb 10, 2020 at 3:50 PM Adam Feuer <a...@starcat.io> wrote: >> >>> It seems like the SAMA5D36 GMAC driver in sam_gmac.c is running out of >>> TX buffers. I can set a large number of buffers, but it always seems to run >>> out. I get these entries in the log: >>> >>> [ 402.310000] sam_transmit: Disabling RX interrupts >>> >>> That seems to come from line 769 which notes that it's doing this >>> because it can't transmit due to lack of TX buffers. Immediately >>> afterwards, the transmission stalls, and then a minute later there is a >>> txtimeout fired, the driver restarts, packets can be sent again until it >>> runs out of TX buffers again. >>> >>> So it seems like there might be something wrong here. Could it be my >>> configuration? Or could it be that the GMAC driver is leaking TX buffers? >>> Does anyone have any ideas on how to find the leak? >>> >>> cheers >>> adam >>> >>> On Mon, Feb 10, 2020 at 1:51 PM Adam Feuer <a...@starcat.io> wrote: >>> >>>> Well, at least that is a lot of progress for a day. Going from >>>> clueless >>>> >>>>> to pinpointing the failure. >>>>> >>>> >>>> Greg, >>>> >>>> Yes! Thank you for the help. >>>> >>>> cheers >>>> adam >>>> -- >>>> Adam Feuer <a...@starcat.io> >>>> >>> >>> >>> -- >>> Adam Feuer <a...@starcat.io> >>> >> >> >> -- >> Adam Feuer <a...@starcat.io> >> > > > -- > Adam Feuer <a...@starcat.io> > -- Adam Feuer <a...@starcat.io>
