I did some more debugging today. It seems like the GMAC is sending multiple buffers sometimes when the TCP send is larger than MSS. So it's not sending not just a single one by itself. When that happens, the GMAC driver marks the first TX buffer descriptor as used (sent), txdesc->status = GMACTXD_STA_USED; but subsequent buffers are unmarked until the last, which is marked with GMACTXD_STA_LAST. Buffers that are not marked never get returned to the free pool. Hence the GMAC driver leaks buffers, eventually runs out of TX buffers, stops receiving, can't send, and eventually the txtimeout happens. Then the driver reinitializes the GMAC and the TX buffer pool, and frames start sending again. And the process repeats.
I have a fix that works at slow speeds– with some logging turned on. It can do unlimited TCP sends of any size. With logging off, it hangs. I'm guessing that's because at high speeds it can send longer chains of packets and buffers all at once. I'm probably not handling chains of more than two consecutive packets correctly. I'll work on it some more tomorrow. -adam On Mon, Feb 10, 2020 at 5:38 PM Adam Feuer <a...@starcat.io> wrote: > I put a log statement in sam_txdone()... what appears to be happening is > that the TX descriptor-in-use case is executed at sam_txdone(), line 1364, > in sam_gmac.c, so the TX buffer fills up. > > So it seems like the GMAC driver or hardware gets into a state where it is > not able to send packets until the driver and hardware is reset. Still not > sure why that's happening. > > -adam > > On Mon, Feb 10, 2020 at 3:50 PM Adam Feuer <a...@starcat.io> wrote: > >> It seems like the SAMA5D36 GMAC driver in sam_gmac.c is running out of TX >> buffers. I can set a large number of buffers, but it always seems to run >> out. I get these entries in the log: >> >> [ 402.310000] sam_transmit: Disabling RX interrupts >> >> That seems to come from line 769 which notes that it's doing this because >> it can't transmit due to lack of TX buffers. Immediately afterwards, the >> transmission stalls, and then a minute later there is a txtimeout fired, >> the driver restarts, packets can be sent again until it runs out of TX >> buffers again. >> >> So it seems like there might be something wrong here. Could it be my >> configuration? Or could it be that the GMAC driver is leaking TX buffers? >> Does anyone have any ideas on how to find the leak? >> >> cheers >> adam >> >> On Mon, Feb 10, 2020 at 1:51 PM Adam Feuer <a...@starcat.io> wrote: >> >>> Well, at least that is a lot of progress for a day. Going from clueless >>> >>>> to pinpointing the failure. >>>> >>> >>> Greg, >>> >>> Yes! Thank you for the help. >>> >>> cheers >>> adam >>> -- >>> Adam Feuer <a...@starcat.io> >>> >> >> >> -- >> Adam Feuer <a...@starcat.io> >> > > > -- > Adam Feuer <a...@starcat.io> > -- Adam Feuer <a...@starcat.io>
