On Sat, Jan 27, 2024 at 08:08:35AM +0100, Hrvoje Popovski wrote:
> On 26.1.2024. 22:47, Alexander Bluhm wrote:
> > On Fri, Jan 26, 2024 at 11:41:49AM +0100, Hrvoje Popovski wrote:
> >> I've manage to reproduce TSO em problem on anoter setup, unfortunatly
> >> production.
> > What helped debugging a similar issue with ixl(4) and TSO was to
> > remove all TSO specific code from the driver. Then only this part
> > remains from the original em(4) TSO diff.
> >
> > error = bus_dmamap_create(sc->sc_dmat, EM_TSO_SIZE,
> > EM_MAX_SCATTER / (sc->pcix_82544 ? 2 : 1),
> > EM_TSO_SEG_SIZE, 0, BUS_DMA_NOWAIT, &pkt->pkt_map);
> >
> > The parameters that changed when adding TSO are:
> >
> > bus_size_t size: MAX_JUMBO_FRAME_SIZE 16128 -> EM_TSO_SIZE 65535
> > bus_size_t maxsegsz: MAX_JUMBO_FRAME_SIZE 16128 -> EM_TSO_SEG_SIZE
> > 4096
> >
> > I suspect that this is the cause for the regression as disabling
> > TSO did not help. Would it be possible to run the diff below? I
> > expect that the problem will still be there. But then we know it
> > must be the change of one of the bus_dmamap_create() arguments.
> >
> > bluhm
>
> Hi,
>
> with this diff em0 seems happy and em watchdog is gone.
This is very interesting. That means that the bus_dmamap_create()
argument does not cause the regression.
Did you see anywhere "output TSO packets hardware processed in"
netstat -s. In some iteration of testing you turned TSO off with
sysctl net.inet.tcp.tso=0, but it did not help. So no TSO packets
from the stack.
In another mail you mentioned
> Setup is very simple
> em0 - carp <- uplink
> em1 - pfsync
> ix1 - vlans - carp
ix supports LRO. If you forward from ix1 to em0 the LRO packets
from ix hardware are split by TSO on em hardware. And the ix does
vlan offloading + LRO, so em must do vlan offloading properly with
TSO. Or do you use a vlan interface?
Does it help to disable LRO, ifconfig ix1 -tcplro ?
I see this vlan code with mac_type checks. Can we end in a
configuration where we enable TSO but cannot do VLAN offloading?
#if NVLAN > 0
/* Find out if we are in VLAN mode */
if (m->m_flags & M_VLANTAG && (sc->hw.mac_type < em_82575 ||
sc->hw.mac_type > em_i210)) {
/* Set the VLAN id */
desc->upper.fields.special = htole16(m->m_pkthdr.ether_vtag);
/* Tell hardware to add tag */
desc->lower.data |= htole32(E1000_TXD_CMD_VLE);
}
#endif
Hrvoje, I know you do great tests in your lab. Did you try this
setup:
Send bulk TCP traffic in vlan that will trigger LRO.
Do VLAN + LRO offloading in ix.
Forward it to em with TSO.
In theory when going through vlan interface it should remove
M_VLANTAG. But something must be wrong and I wonder what.
bluhm
