> On Aug 18, 2015, at 12:49 AM, Rick Macklem <rmack...@uoguelph.ca> wrote: > > Daniel Braniss wrote: >> >>> On Aug 17, 2015, at 3:21 PM, Rick Macklem <rmack...@uoguelph.ca> wrote: >>> >>> Daniel Braniss wrote: >>>> >>>>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron <csforge...@gmail.com> >>>>> wrote: >>>>> >>>>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD >>>>> 10.1. Before 10.1 it was less. >>>>> >>>> >>>> this is NOT iperf/3 where i do get close to wire speed, >>>> it’s NFS writes, i.e., almost real work :-) >>>> >>>>> I used to tweak the card settings, but now it's just stock. You may want >>>>> to >>>>> check your settings, the Mellanox may just have better defaults for your >>>>> switch. >>>>> >>> Have you tried disabling TSO for the Intel? With TSO enabled, it will be >>> copying >>> every transmitted mbuf chain to a new chain of mbuf clusters via. >>> m_defrag() when >>> TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the >>> 82599 chip >>> these days?) >>> >> >> hi Rick >> >> how can i check the chip? >> > Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling TSO > helped, > I'll bet you don't have a 82598.) > >>> This has been fixed in the driver very recently, but those fixes won't be >>> in 10.1. >>> >>> rick >>> ps: If you could test with 10.2, it would be interesting to see how the ix >>> does with >>> the current driver fixes in it? >> >> I new TSO was involved! >> ok, firstly, it’s 10.2 stable. >> with TSO enabled, ix is bad, around 64MGB/s. >> disabling TSO it’s better, around 130 >> > Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at > around > line#2500? > /* TSO parameters */ > 2572 ifp->if_hw_tsomax = 65518; > 2573 ifp->if_hw_tsomaxsegcount = IXGBE_82599_SCATTER; > 2574 ifp->if_hw_tsomaxsegsize = 2048; > > They are in stable/10. I didn't look at releng/10.2. (And if they're in a > #ifdef > for FreeBSD11, take the #ifdef away.) > If they are there and not ifdef'd, I can't explain why disabling TSO would > help. > Once TSO is fixed so that it handles the 64K transmit segments without > copying all > the mbufs, I suspect you might get better perf. with it enabled? >
this is 10.2 : they are on lines 2509-2511 and I don’t see any #ifdefs around it. the plot thickens :-) danny > Good luck with it, rick > >> still, mlxen0 is about 250! with and without TSO >> >> >>> >>>>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov <s...@zxy.spb.ru >>>>> <mailto:s...@zxy.spb.ru>> wrote: >>>>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote: >>>>> >>>>>> hi, >>>>>> I have a host (Dell R730) with both cards, connected to an HP8200 >>>>>> switch at 10Gb. >>>>>> when writing to the same storage (netapp) this is what I get: >>>>>> ix0: ~130MGB/s >>>>>> mlxen0 ~330MGB/s >>>>>> this is via nfs/tcpv3 >>>>>> >>>>>> I can get similar (bad) performance with the mellanox if I increase >>>>>> the file size >>>>>> to 512MGB. >>>>> >>>>> Look like mellanox have internal beffer for caching and do ACK >>>>> acclerating. >>>>> >>>>>> so at face value, it seems the mlxen does a better use of resources >>>>>> than the intel. >>>>>> Any ideas how to improve ix/intel's performance? >>>>> >>>>> Are you sure about netapp performance? >>>>> _______________________________________________ >>>>> freebsd-...@freebsd.org <mailto:freebsd-...@freebsd.org> mailing list >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>>>> <https://lists.freebsd.org/mailman/listinfo/freebsd-net> >>>>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org >>>>> <mailto:freebsd-net-unsubscr...@freebsd.org>" >>>>> >>>> >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" >> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" _______________________________________________ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"