Re: ix(intel) vs mlxen(mellanox) 10Gb performance
On Aug 18, 2015, at 12:49 AM, Rick Macklem rmack...@uoguelph.ca wrote: Daniel Braniss wrote: On Aug 17, 2015, at 3:21 PM, Rick Macklem rmack...@uoguelph.ca wrote: Daniel Braniss wrote: On Aug 17, 2015, at 1:41 PM, Christopher Forgeron csforge...@gmail.com wrote: FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD 10.1. Before 10.1 it was less. this is NOT iperf/3 where i do get close to wire speed, it’s NFS writes, i.e., almost real work :-) I used to tweak the card settings, but now it's just stock. You may want to check your settings, the Mellanox may just have better defaults for your switch. Have you tried disabling TSO for the Intel? With TSO enabled, it will be copying every transmitted mbuf chain to a new chain of mbuf clusters via. m_defrag() when TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the 82599 chip these days?) hi Rick how can i check the chip? Haven't a clue. Does dmesg tell you? (To be honest, since disabling TSO helped, I'll bet you don't have a 82598.) This has been fixed in the driver very recently, but those fixes won't be in 10.1. rick ps: If you could test with 10.2, it would be interesting to see how the ix does with the current driver fixes in it? I new TSO was involved! ok, firstly, it’s 10.2 stable. with TSO enabled, ix is bad, around 64MGB/s. disabling TSO it’s better, around 130 Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at around line#2500? /* TSO parameters */ 2572 ifp-if_hw_tsomax = 65518; 2573 ifp-if_hw_tsomaxsegcount = IXGBE_82599_SCATTER; 2574 ifp-if_hw_tsomaxsegsize = 2048; They are in stable/10. I didn't look at releng/10.2. (And if they're in a #ifdef for FreeBSD11, take the #ifdef away.) If they are there and not ifdef'd, I can't explain why disabling TSO would help. Once TSO is fixed so that it handles the 64K transmit segments without copying all the mbufs, I suspect you might get better perf. with it enabled? this is 10.2 : they are on lines 2509-2511 and I don’t see any #ifdefs around it. the plot thickens :-) danny Good luck with it, rick still, mlxen0 is about 250! with and without TSO On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov s...@zxy.spb.ru mailto:s...@zxy.spb.ru wrote: On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote: hi, I have a host (Dell R730) with both cards, connected to an HP8200 switch at 10Gb. when writing to the same storage (netapp) this is what I get: ix0:~130MGB/s mlxen0 ~330MGB/s this is via nfs/tcpv3 I can get similar (bad) performance with the mellanox if I increase the file size to 512MGB. Look like mellanox have internal beffer for caching and do ACK acclerating. so at face value, it seems the mlxen does a better use of resources than the intel. Any ideas how to improve ix/intel's performance? Are you sure about netapp performance? ___ freebsd-...@freebsd.org mailto:freebsd-...@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org mailto:freebsd-net-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ix(intel) vs mlxen(mellanox) 10Gb performance
Daniel Braniss wrote:
On Aug 18, 2015, at 12:49 AM, Rick Macklem rmack...@uoguelph.ca wrote:
Daniel Braniss wrote:
On Aug 17, 2015, at 3:21 PM, Rick Macklem rmack...@uoguelph.ca wrote:
Daniel Braniss wrote:
On Aug 17, 2015, at 1:41 PM, Christopher Forgeron
csforge...@gmail.com
wrote:
FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD
10.1. Before 10.1 it was less.
this is NOT iperf/3 where i do get close to wire speed,
it’s NFS writes, i.e., almost real work :-)
I used to tweak the card settings, but now it's just stock. You may
want
to
check your settings, the Mellanox may just have better defaults for
your
switch.
Have you tried disabling TSO for the Intel? With TSO enabled, it will be
copying
every transmitted mbuf chain to a new chain of mbuf clusters via.
m_defrag() when
TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the
82599 chip
these days?)
hi Rick
how can i check the chip?
Haven't a clue. Does dmesg tell you? (To be honest, since disabling TSO
helped,
I'll bet you don't have a 82598.)
This has been fixed in the driver very recently, but those fixes won't be
in 10.1.
rick
ps: If you could test with 10.2, it would be interesting to see how the
ix
does with
the current driver fixes in it?
I new TSO was involved!
ok, firstly, it’s 10.2 stable.
with TSO enabled, ix is bad, around 64MGB/s.
disabling TSO it’s better, around 130
Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at
around
line#2500?
/* TSO parameters */
2572 ifp-if_hw_tsomax = 65518;
2573 ifp-if_hw_tsomaxsegcount =
IXGBE_82599_SCATTER;
2574 ifp-if_hw_tsomaxsegsize = 2048;
They are in stable/10. I didn't look at releng/10.2. (And if they're in a
#ifdef
for FreeBSD11, take the #ifdef away.)
If they are there and not ifdef'd, I can't explain why disabling TSO would
help.
Once TSO is fixed so that it handles the 64K transmit segments without
copying all
the mbufs, I suspect you might get better perf. with it enabled?
this is 10.2 :
they are on lines 2509-2511 and I don’t see any #ifdefs around it.
the plot thickens :-)
If this is just a test machine, maybe you could test with these lines (at about
#880)
in sys/netinet/tcp_output.c commented out? (It looks to me like this will
disable TSO
for almost all the NFS writes.)
- around line #880 in sys/netinet/tcp_output.c:
/*
* In case there are too many small fragments
* don't use TSO:
*/
if (len = max_len) {
len = max_len;
sendalot = 1;
tso = 0;
}
This was added along with the other stuff that did the if_hw_tsomaxsegcount,
etc and I
never noticed it until now (not my patch).
rick
danny
Good luck with it, rick
still, mlxen0 is about 250! with and without TSO
On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov s...@zxy.spb.ru
mailto:s...@zxy.spb.ru wrote:
On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote:
hi,
I have a host (Dell R730) with both cards, connected to an HP8200
switch at 10Gb.
when writing to the same storage (netapp) this is what I get:
ix0:~130MGB/s
mlxen0 ~330MGB/s
this is via nfs/tcpv3
I can get similar (bad) performance with the mellanox if I
increase
the file size
to 512MGB.
Look like mellanox have internal beffer for caching and do ACK
acclerating.
so at face value, it seems the mlxen does a better use of
resources
than the intel.
Any ideas how to improve ix/intel's performance?
Are you sure about netapp performance?
___
freebsd-...@freebsd.org mailto:freebsd-...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
mailto:freebsd-net-unsubscr...@freebsd.org
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to
freebsd-stable-unsubscr...@freebsd.org
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
___
freebsd-...@freebsd.org mailing list
ping from web application
Hi, I use web applicaton (net-mgmt/phpipam) which should have the ability to check hosts' availability via ping. I can even specify path to ping executable. This functionality does not work on FreeBSD by default, and suggested workaround is to set setuid bit on /sbin/ping. I don't like to modify permissions of base files. Is there an alternative solution? e.g. a port? Thank you in advance, -- Marko Cupać https://www.mimar.rs/ ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Issue with /boot/menu.rc.local after 10.2 update
I just found out the problem. I don't know who added the following phrase to /boot/menu-commands.4th: only forth definitions Line 31 and 354. It just broke /boot/menu.rc.local as i said before. After removing that everything worked again. Hope it helps someone else. I'm going to open a PR anyway. Thanks. On Tue, Aug 18, 2015 at 11:13 AM, Cassiano Peixoto peixotocassi...@gmail.com wrote: Hi guys, I've been using /boot/menu.rc.local for years to my custom menu option. But after 10.2 update, it's not working anymore. I always get the message toggle_menuitem not found on beastie menu. Here is my /boot/menu.rc.local: set optionsmenu_init[7]=init_console set optionsmenu_caption[7]=[C]onsole. Off set optionstoggled_text[7]=[C]onsole. On set optionsmenu_command[7]=toggle_console set optionsmenu_keycode[7]=118 set optionsansi_caption[7]=^[[1mC^[[37monsole. ^[[34;1mOff^[[37m set optionstoggled_ansi[7]=^[[1mC^[[37monsole. ^[[32;7mOn^[[0;37m \ \ Console Boot \ : console_enabled? ( -- flag ) s boot_single getenv -1 dup if drop ( c-addr flag -- flag ) then ; : console_enable ( -- ) s set console=comconsole,vidconsole evaluate ; : console_disable ( -- ) s set console=vidconsole,comconsole evaluate ; : toggle_console ( N -- N TRUE ) toggle_menuitem menu-redraw \ Now we're going to make the change effective dup toggle_stateN @ 0= if console_disable else console_enable then TRUE \ loop menu again ; So what has changed? What should i do to make it works again? Thanks. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Swap Questions
On Mon, 17 Aug 2015, Tim Daneliuk wrote: On 08/17/2015 12:53 PM, Antony Uspensky wrote: On Sat, 15 Aug 2015, Tim Daneliuk wrote: So -L does fix the problem - sort of. The machine picks up the file as additional swap on boot just fine. HWOEVER, when I try to reboot or shut down the host, I get a panic telling me some noise about not being able to shutdown swap for some reason. Try to swapoff (by hands) before shutdown. Shutdown sequence, I think, unmounts carrying disk before swapping off a carried file. If I am right, -L should be processed on shutdown also. Just a guess. Yes, that did it. But, isn't this kind of an operational bug? Shouldn't the shutdown logic do the swapoff before the unmount if it sees files being used for swap? Yes. Must. i.e. Should I enter this as a bug report? Yes, please. The only reason this matters - and it's a pretty big reason - is for production servers when someone logs in remotely, becomes root, and issued reboot. The machine hangs at the panic and never comes back ... something you do not see unless you are in a console of some sort ... ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Swap Questions
On 08/18/2015 12:29 PM, Antony Uspensky wrote: On Mon, 17 Aug 2015, Tim Daneliuk wrote: On 08/17/2015 12:53 PM, Antony Uspensky wrote: On Sat, 15 Aug 2015, Tim Daneliuk wrote: So -L does fix the problem - sort of. The machine picks up the file as additional swap on boot just fine. HWOEVER, when I try to reboot or shut down the host, I get a panic telling me some noise about not being able to shutdown swap for some reason. Try to swapoff (by hands) before shutdown. Shutdown sequence, I think, unmounts carrying disk before swapping off a carried file. If I am right, -L should be processed on shutdown also. Just a guess. Yes, that did it. But, isn't this kind of an operational bug? Shouldn't the shutdown logic do the swapoff before the unmount if it sees files being used for swap? Yes. Must. i.e. Should I enter this as a bug report? Yes, please. The only reason this matters - and it's a pretty big reason - is for production servers when someone logs in remotely, becomes root, and issued reboot. The machine hangs at the panic and never comes back ... something you do not see unless you are in a console of some sort ... Done. Bug report #202420 -- Tim Daneliuk tun...@tundraware.com PGP Key: http://www.tundraware.com/PGP/ ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ping from web application
On Aug 18, 2015, at 7:15 AM, Lowell Gilbert freebsd-stable-lo...@be-well.ilk.org wrote: Marko Cupać marko.cu...@mimar.rs writes: I use web applicaton (net-mgmt/phpipam) which should have the ability to check hosts' availability via ping. I can even specify path to ping executable. This functionality does not work on FreeBSD by default, and suggested workaround is to set setuid bit on /sbin/ping. I don't like to modify permissions of base files. Is there an alternative solution? e.g. a port? In what way does ping(8) not work? A look at its error output should tell you what the problem is. Additionally, the standard permissions on /sbin/ping *are* suid root. It certainly won't work if you've changed that, so just change it back. And yes, there are other ping programs present, including some with pretty graphical web page UIs. But there's no reason that PHP should have trouble calling /sbin/ping. It is a pretty standard issue: only apps running as root can send ICMP directly, as ping does. PHP runs in Apache, and to prevent security issues with privilege escalation setuid programs are forced to run as an unprivileged user. I would check to see how “fping” in Nagios solved this issue. Tom ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: [FreeBSD-Announce] FreeBSD Errata Notice FreeBSD-EN-15:12.netstat
sorry, wrong cc. On Tue, Aug 18, 2015 at 4:25 PM, Philip Gollucci pgollu...@gocurb.com wrote: We do not run 32bit. On Tue, Aug 18, 2015 at 4:15 PM, FreeBSD Errata Notices errata-noti...@freebsd.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 = FreeBSD-EN-15:12.netstatErrata Notice The FreeBSD Project Topic: Incorrect netstat(1) data handling on 32-bit systems Category: core Module: netstat Announced: 2015-08-18 Credits:Mark Johnston Affects:FreeBSD 10.2-RELEASE Corrected: 2015-07-31 00:21:41 UTC (stable/10, 10.2-STABLE) 2015-08-18 19:30:17 UTC (releng/10.2, 10.2-RC3-p1) 2015-08-18 19:30:17 UTC (releng/10.2, 10.2-RELEASE-p1) For general information regarding FreeBSD Errata Notices and Security Advisories, including descriptions of the fields above, security branches, and the following sections, please visit URL:https://security.freebsd.org/. I. Background The netstat(1) utility displays the contents of various network related data structures. II. Problem Description The netstat(1) utility incorrectly handles reported values on 32-bit systems. III. Impact Due to how netstat(1) processes IPSEC counters, the utility may produce incorrect output on 32-bit systems. IV. Workaround No workaround is available, however systems without IPSEC compiled into the kernel are not affected. V. Solution Perform one of the following: 1) Upgrade your system to a supported FreeBSD stable or release / security branch (releng) dated after the correction date. 2) To update your present system via a binary patch: Systems running a RELEASE version of FreeBSD on the i386 or amd64 platforms can be updated via the freebsd-update(8) utility: # freebsd-update fetch # freebsd-update install 3) To update your present system via a source code patch: The following patches have been verified to apply to the applicable FreeBSD release branches. a) Download the relevant patch from the location below, and verify the detached PGP signature using your PGP utility. # fetch https://security.FreeBSD.org/patches/EN-15:12/netstat.patch # fetch https://security.FreeBSD.org/patches/EN-15:12/netstat.patch.asc # gpg --verify netstat.patch.asc b) Apply the patch. Execute the following commands as root: # cd /usr/src # patch /path/to/patch c) Recompile the operating system using buildworld and installworld as described in URL:https://www.FreeBSD.org/handbook/makeworld.html. VI. Correction details The following list contains the correction revision numbers for each affected branch. Branch/path Revision - - stable/10/r286099 releng/10.2/ r286901 - - To see which files were modified by a particular revision, run the following command, replacing NN with the revision number, on a machine with Subversion installed: # svn diff -cNN --summarize svn://svn.freebsd.org/base Or visit the following URL, replacing NN with the revision number: URL:https://svnweb.freebsd.org/base?view=revisionrevision=NN VII. References URL:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=201700 The latest revision of this Errata Notice is available at https://security.FreeBSD.org/advisories/FreeBSD-EN-15:12.netstat.asc -BEGIN PGP SIGNATURE- Version: GnuPG v2.1.7 (FreeBSD) iQIcBAEBCgAGBQJV05A1AAoJEO1n7NZdz2rnAzIQANyLdOOQhe9dHyAV4N5YKM3B Z/86dY/KIIViVb1uzkBASBNnkHlG+mCMOQpzX2x8yCPF4i7bIEfPa4r2Bzw9pvCF RWRKvZXERESh/RndQFhxcmJMAyYPq7MdK0IZzG53vinlMoSz2WTKj2vSR7t2jfo+ ObTfgdkqN/PZs/W+AQY8a4DMdxCLg1KCZiSpQRO7ea+4AxsI8qNgoytvG6HRno/z uGe6Ad82ZfysKgqe9JO4gvRTR77ebQAVSSr3qylQcOGHohy9tFHcI2FEAAqLJrQY b5DDLOawLRsQm0hwkLCTOZX2QvIFgz0gGRpvPcN9ZKValMc5DKQv36z3hOByK+3i dDHFG/Diy2JNP0tsKtW8IyyLvW2DAUoTs1nVaWMvLKkMUr+loOYvoaLdGT0xQP2d M6UT40mRMznfH/Gq/0DJFArsYcyB9YRl7rD0dy1HhqApogHQrTjsT+1vtBtpaTmv LHA77tHyzI0TxOvmx3hglj/z4BLZDPU6ydXr3zeOYBpLz5p02GKxHUc+JrmWBfOV Jep0+Fr2fYST5bGVtExNQV6cTlBZPnGR4JxJEUQA6a+FdyJcDuzOTNcs0YzwjuSC dIk5pdxI3nkc+zf9GZLXUdLcxXfo6jBUy0fSWkzirGzBfo0wxseE6cbxxTH7vumx CLGGmHiqxVuF/nP4ScHi =3aK1 -END PGP SIGNATURE- ___ freebsd-annou...@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-announce To unsubscribe, send any mail to freebsd-announce-unsubscr...@freebsd.org -- *PHILIP M. GOLLUCCI* Sr. Director of IT | *Curb* *w.* 703-579-6947 *m.* 703-336-9354 @gocurb
Re: ix(intel) vs mlxen(mellanox) 10Gb performance
Hans Petter Selasky wrote:
On 08/18/15 14:53, Rick Macklem wrote:
If this is just a test machine, maybe you could test with these lines (at
about #880)
in sys/netinet/tcp_output.c commented out? (It looks to me like this will
disable TSO
for almost all the NFS writes.)
- around line #880 in sys/netinet/tcp_output.c:
/*
* In case there are too many small fragments
* don't use TSO:
*/
if (len = max_len) {
len = max_len;
sendalot = 1;
tso = 0;
}
This was added along with the other stuff that did the
if_hw_tsomaxsegcount, etc and I
never noticed it until now (not my patch).
FYI:
These lines are needed by other hardware, like the mlxen driver. If you
remove them mlxen will start doing m_defrag(). I believe if you set the
correct parameters in the struct ifnet for the TSO size/count limits
this problem will go away. If you print the len and max_len and also
the cases where TSO limits are reached, you'll see what parameter is
triggering it and needs to be increased.
Well, if the driver isn't setting if_hw_tsomaxsegcount correctly, then it
is the driver that needs to be fixed.
Having the above code block disable TSO for all of the NFS writes, including
the ones that set if_hw_tsomaxsegcount correctly doesn't make sense to me.
If the driver authors don't set these, the drivers do lots of m_defrag()
calls. I have posted more than once to freebsd-net@ asking the driver authors
to set these and some now have. (I can't do it, because I don't have the
hardware to test it with.)
I do think that most/all of them don't subtract 1 for the tcp/ip header and
I don't think they should be expected to, since the driver isn't supposed to
worry about the protocol at that level.
-- I think tcp_output() should subtract one from the if_hw_tsomaxsegcount
provided by the driver to handle this, since it chooses to count mbufs
(the while() loop at around line #825 in sys/netinet/tcp_output.c.)
before it prepends the tcp/ip header mbuf.
rick
--HPS
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ix(intel) vs mlxen(mellanox) 10Gb performance
Daniel Braniss wrote: On Aug 18, 2015, at 12:49 AM, Rick Macklem rmack...@uoguelph.ca wrote: Daniel Braniss wrote: On Aug 17, 2015, at 3:21 PM, Rick Macklem rmack...@uoguelph.ca wrote: Daniel Braniss wrote: On Aug 17, 2015, at 1:41 PM, Christopher Forgeron csforge...@gmail.com wrote: FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeBSD 10.1. Before 10.1 it was less. this is NOT iperf/3 where i do get close to wire speed, it’s NFS writes, i.e., almost real work :-) I used to tweak the card settings, but now it's just stock. You may want to check your settings, the Mellanox may just have better defaults for your switch. Have you tried disabling TSO for the Intel? With TSO enabled, it will be copying every transmitted mbuf chain to a new chain of mbuf clusters via. m_defrag() when TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the 82599 chip these days?) Oops, I think I screwed up. It looks like t_maxopd is limited to somewhat less than the mtu. If that is the case, the code block wouldn't do what I thought it would do. However, if_hw_tsomaxsegcount does need to be one less than the limit for the driver, since the tcp/ip header isn't yet prepended when it is counted. I think the code in tcp_output() should subtract 1, but you can change it in the driver to test this. Thanks for doing this, rick hi Rick how can i check the chip? Haven't a clue. Does dmesg tell you? (To be honest, since disabling TSO helped, I'll bet you don't have a 82598.) This has been fixed in the driver very recently, but those fixes won't be in 10.1. rick ps: If you could test with 10.2, it would be interesting to see how the ix does with the current driver fixes in it? I new TSO was involved! ok, firstly, it’s 10.2 stable. with TSO enabled, ix is bad, around 64MGB/s. disabling TSO it’s better, around 130 Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at around line#2500? /* TSO parameters */ 2572 ifp-if_hw_tsomax = 65518; 2573 ifp-if_hw_tsomaxsegcount = IXGBE_82599_SCATTER; 2574 ifp-if_hw_tsomaxsegsize = 2048; They are in stable/10. I didn't look at releng/10.2. (And if they're in a #ifdef for FreeBSD11, take the #ifdef away.) If they are there and not ifdef'd, I can't explain why disabling TSO would help. Once TSO is fixed so that it handles the 64K transmit segments without copying all the mbufs, I suspect you might get better perf. with it enabled? this is 10.2 : they are on lines 2509-2511 and I don’t see any #ifdefs around it. the plot thickens :-) danny Good luck with it, rick still, mlxen0 is about 250! with and without TSO On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov s...@zxy.spb.ru mailto:s...@zxy.spb.ru wrote: On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote: hi, I have a host (Dell R730) with both cards, connected to an HP8200 switch at 10Gb. when writing to the same storage (netapp) this is what I get: ix0:~130MGB/s mlxen0 ~330MGB/s this is via nfs/tcpv3 I can get similar (bad) performance with the mellanox if I increase the file size to 512MGB. Look like mellanox have internal beffer for caching and do ACK acclerating. so at face value, it seems the mlxen does a better use of resources than the intel. Any ideas how to improve ix/intel's performance? Are you sure about netapp performance? ___ freebsd-...@freebsd.org mailto:freebsd-...@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org mailto:freebsd-net-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-...@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: [FreeBSD-Announce] FreeBSD Errata Notice FreeBSD-EN-15:12.netstat
We do not run 32bit. On Tue, Aug 18, 2015 at 4:15 PM, FreeBSD Errata Notices errata-noti...@freebsd.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 = FreeBSD-EN-15:12.netstatErrata Notice The FreeBSD Project Topic: Incorrect netstat(1) data handling on 32-bit systems Category: core Module: netstat Announced: 2015-08-18 Credits:Mark Johnston Affects:FreeBSD 10.2-RELEASE Corrected: 2015-07-31 00:21:41 UTC (stable/10, 10.2-STABLE) 2015-08-18 19:30:17 UTC (releng/10.2, 10.2-RC3-p1) 2015-08-18 19:30:17 UTC (releng/10.2, 10.2-RELEASE-p1) For general information regarding FreeBSD Errata Notices and Security Advisories, including descriptions of the fields above, security branches, and the following sections, please visit URL:https://security.freebsd.org/. I. Background The netstat(1) utility displays the contents of various network related data structures. II. Problem Description The netstat(1) utility incorrectly handles reported values on 32-bit systems. III. Impact Due to how netstat(1) processes IPSEC counters, the utility may produce incorrect output on 32-bit systems. IV. Workaround No workaround is available, however systems without IPSEC compiled into the kernel are not affected. V. Solution Perform one of the following: 1) Upgrade your system to a supported FreeBSD stable or release / security branch (releng) dated after the correction date. 2) To update your present system via a binary patch: Systems running a RELEASE version of FreeBSD on the i386 or amd64 platforms can be updated via the freebsd-update(8) utility: # freebsd-update fetch # freebsd-update install 3) To update your present system via a source code patch: The following patches have been verified to apply to the applicable FreeBSD release branches. a) Download the relevant patch from the location below, and verify the detached PGP signature using your PGP utility. # fetch https://security.FreeBSD.org/patches/EN-15:12/netstat.patch # fetch https://security.FreeBSD.org/patches/EN-15:12/netstat.patch.asc # gpg --verify netstat.patch.asc b) Apply the patch. Execute the following commands as root: # cd /usr/src # patch /path/to/patch c) Recompile the operating system using buildworld and installworld as described in URL:https://www.FreeBSD.org/handbook/makeworld.html. VI. Correction details The following list contains the correction revision numbers for each affected branch. Branch/path Revision - - stable/10/r286099 releng/10.2/ r286901 - - To see which files were modified by a particular revision, run the following command, replacing NN with the revision number, on a machine with Subversion installed: # svn diff -cNN --summarize svn://svn.freebsd.org/base Or visit the following URL, replacing NN with the revision number: URL:https://svnweb.freebsd.org/base?view=revisionrevision=NN VII. References URL:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=201700 The latest revision of this Errata Notice is available at https://security.FreeBSD.org/advisories/FreeBSD-EN-15:12.netstat.asc -BEGIN PGP SIGNATURE- Version: GnuPG v2.1.7 (FreeBSD) iQIcBAEBCgAGBQJV05A1AAoJEO1n7NZdz2rnAzIQANyLdOOQhe9dHyAV4N5YKM3B Z/86dY/KIIViVb1uzkBASBNnkHlG+mCMOQpzX2x8yCPF4i7bIEfPa4r2Bzw9pvCF RWRKvZXERESh/RndQFhxcmJMAyYPq7MdK0IZzG53vinlMoSz2WTKj2vSR7t2jfo+ ObTfgdkqN/PZs/W+AQY8a4DMdxCLg1KCZiSpQRO7ea+4AxsI8qNgoytvG6HRno/z uGe6Ad82ZfysKgqe9JO4gvRTR77ebQAVSSr3qylQcOGHohy9tFHcI2FEAAqLJrQY b5DDLOawLRsQm0hwkLCTOZX2QvIFgz0gGRpvPcN9ZKValMc5DKQv36z3hOByK+3i dDHFG/Diy2JNP0tsKtW8IyyLvW2DAUoTs1nVaWMvLKkMUr+loOYvoaLdGT0xQP2d M6UT40mRMznfH/Gq/0DJFArsYcyB9YRl7rD0dy1HhqApogHQrTjsT+1vtBtpaTmv LHA77tHyzI0TxOvmx3hglj/z4BLZDPU6ydXr3zeOYBpLz5p02GKxHUc+JrmWBfOV Jep0+Fr2fYST5bGVtExNQV6cTlBZPnGR4JxJEUQA6a+FdyJcDuzOTNcs0YzwjuSC dIk5pdxI3nkc+zf9GZLXUdLcxXfo6jBUy0fSWkzirGzBfo0wxseE6cbxxTH7vumx CLGGmHiqxVuF/nP4ScHi =3aK1 -END PGP SIGNATURE- ___ freebsd-annou...@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-announce To unsubscribe, send any mail to freebsd-announce-unsubscr...@freebsd.org -- *PHILIP M. GOLLUCCI* Sr. Director of IT | *Curb* *w.* 703-579-6947 *m.* 703-336-9354 @gocurb http://twitter.com/gocurb Enter code* p6magic* for $10 off your first ride
Panic [page fault] in _ieee80211_crypto_delkey(): stable/10/amd64 @r286878
I was minding my own business in a staff meeting this afternoon, and my laptop rebooted; seems it got a panic. I've copied the core.txt.0 file to http://www.catwhisker.org/~david/FreeBSD/stable_10/, along with a verbose dmesg.boot from this morning and output of pciconf -l -v. This was running: FreeBSD localhost 10.2-STABLE FreeBSD 10.2-STABLE #122 r286878M/286880:1002500: Tue Aug 18 04:06:33 PDT 2015 r...@g1-252.catwhisker.org:/common/S1/obj/usr/src/sys/CANARY amd64 Excerpts from core.txt.0: panic: page fault ... Unread portion of the kernel message buffer: panic: page fault cpuid = 2 KDB: stack backtrace: #0 0x80946e00 at kdb_backtrace+0x60 #1 0x8090a9e6 at vpanic+0x126 #2 0x8090a8b3 at panic+0x43 #3 0x80c8467b at trap_fatal+0x36b #4 0x80c8497d at trap_pfault+0x2ed #5 0x80c8401a at trap+0x47a #6 0x80c6a1b2 at calltrap+0x8 #7 0x809eff5e at ieee80211_crypto_delkey+0x1e #8 0x80a04d45 at ieee80211_ioctl_delkey+0x65 #9 0x80a03bd2 at ieee80211_ioctl_set80211+0x572 #10 0x80a2c323 at in_control+0x203 #11 0x809cd57b at ifioctl+0x15eb #12 0x8095ecf5 at kern_ioctl+0x255 #13 0x8095e9f0 at sys_ioctl+0x140 #14 0x80c84f97 at amd64_syscall+0x357 #15 0x80c6a49b at Xfast_syscall+0xfb Uptime: 9h45m0s ... Loaded symbols for /usr/local/modules/rtc.ko #0 doadump (textdump=value optimized out) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=value optimized out) at pcpu.h:219 #1 0x8090a642 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:451 #2 0x8090aa25 in vpanic (fmt=value optimized out, ap=value optimized out) at /usr/src/sys/kern/kern_shutdown.c:758 #3 0x8090a8b3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:687 #4 0x80c8467b in trap_fatal (frame=value optimized out, eva=value optimized out) at /usr/src/sys/amd64/amd64/trap.c:851 #5 0x80c8497d in trap_pfault (frame=0xfe060d88b510, usermode=value optimized out) at /usr/src/sys/amd64/amd64/trap.c:674 #6 0x80c8401a in trap (frame=0xfe060d88b510) at /usr/src/sys/amd64/amd64/trap.c:440 #7 0x80c6a1b2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 #8 0x809f003a in _ieee80211_crypto_delkey () at /usr/src/sys/net80211/ieee80211_crypto.c:105 #9 0x809eff5e in ieee80211_crypto_delkey (vap=0xfe03d907, key=0xfe03d9070800) at /usr/src/sys/net80211/ieee80211_crypto.c:461 #10 0x80a04d45 in ieee80211_ioctl_delkey (vap=0xfe03d907, ireq=value optimized out) at /usr/src/sys/net80211/ieee80211_ioctl.c:1252 #11 0x80a03bd2 in ieee80211_ioctl_set80211 () at /usr/src/sys/net80211/ieee80211_ioctl.c:2814 #12 0x80a2c323 in in_control (so=value optimized out, cmd=9214790412651315593, data=0xfe060d88bb80 , ifp=0x3, td=value optimized out) at /usr/src/sys/netinet/in.c:308 #13 0x809cd57b in ifioctl (so=0xfe03d9070800, cmd=2149607914, data=0xfe060d88b8e0 wlan0, td=0xf80170abb940) at /usr/src/sys/net/if.c:2770 #14 0x8095ecf5 in kern_ioctl (td=0xf80170abb940, fd=value optimized out, com=18446741891212314624) at file.h:320 #15 0x8095e9f0 in sys_ioctl (td=0xf80170abb940, uap=0xfe060d88ba40) at /usr/src/sys/kern/sys_generic.c:718 #16 0x80c84f97 in amd64_syscall (td=0xf80170abb940, traced=0) at subr_syscall.c:134 #17 0x80c6a49b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 #18 0x0008012a2f9a in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) Physical 802.11 hardware is iwn(4). I can copy the vmcore.0 file itself after I get home -- it's ~625MB, and I'd rather not try to get that through over a WAN before I need to catch the shuttle to get home. :-} Peace, david -- David H. Wolfskill da...@catwhisker.org Those who would murder in the name of God or prophet are blasphemous cowards. See http://www.catwhisker.org/~david/publickey.gpg for my public key. pgpWb4UItbSf2.pgp Description: PGP signature
Re: ix(intel) vs mlxen(mellanox) 10Gb performance
On 08/18/15 14:53, Rick Macklem wrote: 2572 ifp-if_hw_tsomax = 65518; 2573 ifp-if_hw_tsomaxsegcount = IXGBE_82599_SCATTER; 2574 ifp-if_hw_tsomaxsegsize = 2048; Hi, If IXGBE_82599_SCATTER is the maximum scatter/gather entries the hardware can do, remember to subtract one fragment for the TCP/IP-header mbuf! I think there is an off-by-one here: ifp-if_hw_tsomax = 65518; ifp-if_hw_tsomaxsegcount = IXGBE_82599_SCATTER - 1; ifp-if_hw_tsomaxsegsize = 2048; Refer to: * * NOTE: The TSO limits only apply to the data payload part of * a TCP/IP packet. That means there is no need to subtract * space for ethernet-, vlan-, IP- or TCP- headers from the * TSO limits unless the hardware driver in question requires * so. In sys/net/if_var.h Thank you! --HPS ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ping from web application
Marko Cupać marko.cu...@mimar.rs writes: I use web applicaton (net-mgmt/phpipam) which should have the ability to check hosts' availability via ping. I can even specify path to ping executable. This functionality does not work on FreeBSD by default, and suggested workaround is to set setuid bit on /sbin/ping. I don't like to modify permissions of base files. Is there an alternative solution? e.g. a port? In what way does ping(8) not work? A look at its error output should tell you what the problem is. Additionally, the standard permissions on /sbin/ping *are* suid root. It certainly won't work if you've changed that, so just change it back. And yes, there are other ping programs present, including some with pretty graphical web page UIs. But there's no reason that PHP should have trouble calling /sbin/ping. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ix(intel) vs mlxen(mellanox) 10Gb performance
On 08/18/15 14:53, Rick Macklem wrote:
If this is just a test machine, maybe you could test with these lines (at about
#880)
in sys/netinet/tcp_output.c commented out? (It looks to me like this will
disable TSO
for almost all the NFS writes.)
- around line #880 in sys/netinet/tcp_output.c:
/*
* In case there are too many small fragments
* don't use TSO:
*/
if (len = max_len) {
len = max_len;
sendalot = 1;
tso = 0;
}
This was added along with the other stuff that did the if_hw_tsomaxsegcount,
etc and I
never noticed it until now (not my patch).
FYI:
These lines are needed by other hardware, like the mlxen driver. If you
remove them mlxen will start doing m_defrag(). I believe if you set the
correct parameters in the struct ifnet for the TSO size/count limits
this problem will go away. If you print the len and max_len and also
the cases where TSO limits are reached, you'll see what parameter is
triggering it and needs to be increased.
--HPS
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Issue with /boot/menu.rc.local after 10.2 update
Hi guys, I've been using /boot/menu.rc.local for years to my custom menu option. But after 10.2 update, it's not working anymore. I always get the message toggle_menuitem not found on beastie menu. Here is my /boot/menu.rc.local: set optionsmenu_init[7]=init_console set optionsmenu_caption[7]=[C]onsole. Off set optionstoggled_text[7]=[C]onsole. On set optionsmenu_command[7]=toggle_console set optionsmenu_keycode[7]=118 set optionsansi_caption[7]=^[[1mC^[[37monsole. ^[[34;1mOff^[[37m set optionstoggled_ansi[7]=^[[1mC^[[37monsole. ^[[32;7mOn^[[0;37m \ \ Console Boot \ : console_enabled? ( -- flag ) s boot_single getenv -1 dup if drop ( c-addr flag -- flag ) then ; : console_enable ( -- ) s set console=comconsole,vidconsole evaluate ; : console_disable ( -- ) s set console=vidconsole,comconsole evaluate ; : toggle_console ( N -- N TRUE ) toggle_menuitem menu-redraw \ Now we're going to make the change effective dup toggle_stateN @ 0= if console_disable else console_enable then TRUE \ loop menu again ; So what has changed? What should i do to make it works again? Thanks. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ix(intel) vs mlxen(mellanox) 10Gb performance
On Tue, Aug 18, 2015 at 05:09:41PM +0300, Daniel Braniss wrote: sorry, it's been a tough day, we had a major meltdown, caused by a faulty gbic :-( anyways, could you tell me what to do? comment out, fix the off by one? the machine is not yet production. Can you collect this information? https://lists.freebsd.org/pipermail/freebsd-stable/2015-August/083113.html And 'show interface' (or equivalent: error/collsion/events counters) from both ports from HP8200. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ix(intel) vs mlxen(mellanox) 10Gb performance
sorry, it’s been a tough day, we had a major meltdown, caused by a faulty gbic :-( anyways, could you tell me what to do? comment out, fix the off by one? the machine is not yet production. thanks, danny On 18 Aug 2015, at 16:32, Hans Petter Selasky h...@selasky.org wrote: On 08/18/15 14:53, Rick Macklem wrote: 2572 ifp-if_hw_tsomax = 65518; 2573ifp-if_hw_tsomaxsegcount = IXGBE_82599_SCATTER; 2574ifp-if_hw_tsomaxsegsize = 2048; Hi, If IXGBE_82599_SCATTER is the maximum scatter/gather entries the hardware can do, remember to subtract one fragment for the TCP/IP-header mbuf! I think there is an off-by-one here: ifp-if_hw_tsomax = 65518; ifp-if_hw_tsomaxsegcount = IXGBE_82599_SCATTER - 1; ifp-if_hw_tsomaxsegsize = 2048; Refer to: * * NOTE: The TSO limits only apply to the data payload part of * a TCP/IP packet. That means there is no need to subtract * space for ethernet-, vlan-, IP- or TCP- headers from the * TSO limits unless the hardware driver in question requires * so. In sys/net/if_var.h Thank you! --HPS ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ix(intel) vs mlxen(mellanox) 10Gb performance
Hans Petter Selasky wrote:
On 08/18/15 14:53, Rick Macklem wrote:
2572 ifp-if_hw_tsomax = 65518;
2573 ifp-if_hw_tsomaxsegcount =
IXGBE_82599_SCATTER;
2574 ifp-if_hw_tsomaxsegsize = 2048;
Hi,
If IXGBE_82599_SCATTER is the maximum scatter/gather entries the
hardware can do, remember to subtract one fragment for the TCP/IP-header
mbuf!
Ouch! Yes, I now see that the code that counts the # of mbufs is before the
code that adds the tcp/ip header mbuf.
In my opinion, this should be fixed by setting if_hw_tsomaxsegcount to whatever
the driver provides - 1. It is not the driver's responsibility to know if a
tcp/ip
header mbuf will be added and is a lot less confusing that expecting the driver
author to know to subtract one. (I had mistakenly thought that tcp_output() had
added the tc/ip header mbuf before the loop that counts mbufs in the list. Btw,
this tcp/ip header mbuf also has leading space for the MAC layer header.)
I think there is an off-by-one here:
ifp-if_hw_tsomax = 65518;
ifp-if_hw_tsomaxsegcount = IXGBE_82599_SCATTER - 1;
ifp-if_hw_tsomaxsegsize = 2048;
Refer to:
*
* NOTE: The TSO limits only apply to the data payload part of
* a TCP/IP packet. That means there is no need to subtract
* space for ethernet-, vlan-, IP- or TCP- headers from the
* TSO limits unless the hardware driver in question requires
* so.
This comment suggests that the driver author doesn't need to do this.
However, unless this is fixed in tcp_output(), the above patch should be
applied to the driver.
In sys/net/if_var.h
Thank you!
--HPS
The problem I see is that, after doing the calculation of how many mbufs can
be in the TSO segment, the code in tcp_output() will have calculated a value
for len that will always be less that tp-t_maxopd - optlen when the
if_hw_tsosegcount limit has been hit (see where it does a break; out of
the while loop).
-- This does not imply too many small fragments for NFS, just that the
driver's transmit segment limit has been reached, where most of them
are mbuf clusters, but not the first ones.
As such the code:
/*
* In case there are too many small fragments
* don't use TSO:
*/
if (len = max_len) {
len = max_len;
sendalot = 1;
tso = 0;
}
Will always happen for this case and tso gets set to 0. Not what we want to
happen, imho.
The above code block was what I suggested should be commented out or deleted
for the test.
It appears you should also add the - 1 in the driver sys/dev/ixgbe/if_ix.c.
rick
___
freebsd-...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
