Hi Sven,
I see two parameters that we have set to non-default values that are not
in your list of options still to configure.
verbsRdmasPerConnection (256) and
socketMaxListenConnections (1024)
I remember we had to set socketMaxListenConnections because our cluster
consist of +550 nodes.
Are these settings still needed, or is this also tackled in the code?
Thank you!!
Cheers,
Kenneth
On 02/09/17 00:42, Sven Oehme wrote:
Hi Ed,
yes the defaults for that have changed for customers who had not
overridden the default settings. the reason we did this was that many
systems in the field including all ESS systems that come pre-tuned
where manually changed to 8k from the 16k default due to better
performance that was confirmed in multiple customer engagements and
tests with various settings , therefore we change the default to what
it should be in the field so people are not bothered to set it anymore
(simplification) or get benefits by changing the default to provides
better performance.
all this happened when we did the communication code overhaul that did
lead to significant (think factors) of improved RPC performance for
RDMA and VERBS workloads.
there is another round of significant enhancements coming soon , that
will make even more parameters either obsolete or change some of the
defaults for better out of the box performance.
i see that we should probably enhance the communication of this
changes, not that i think this will have any negative effect compared
to what your performance was with the old setting i am actually pretty
confident that you get better performance with the new code, but by
setting parameters back to default on most 'manual tuned' probably
makes your system even faster.
if you have a Scale Client on 4.2.3+ you really shouldn't have
anything set beside maxfilestocache, pagepool, workerthreads and
potential prefetch , if you are a protocol node, this and settings
specific to an export (e.g. SMB, NFS set some special settings) ,
pretty much everything else these days should be set to default so the
code can pick the correct parameters., if its not and you get better
performance by manual tweaking something i like to hear about it.
on the communication side in the next release will eliminate another
set of parameters that are now 'auto set' and we plan to work on NSD
next.
i presented various slides about the communication and simplicity
changes in various forums, latest public non NDA slides i presented
are here -->
http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf
hope this helps .
Sven
On Fri, Sep 1, 2017 at 1:56 PM Edward Wahl <ew...@osc.edu
<mailto:ew...@osc.edu>> wrote:
Howdy. Just noticed this change to min RDMA packet size and I
don't seem to
see it in any patch notes. Maybe I just skipped the one where
this changed?
mmlsconfig verbsRdmaMinBytes
verbsRdmaMinBytes 16384
(in case someone thinks we changed it)
[root@proj-nsd01 ~]# mmlsconfig |grep verbs
verbsRdma enable
verbsRdma disable
verbsRdmasPerConnection 14
verbsRdmasPerNode 1024
verbsPorts mlx5_3/1
verbsPorts mlx4_0
verbsPorts mlx5_0
verbsPorts mlx5_0 mlx5_1
verbsPorts mlx4_1/1
verbsPorts mlx4_1/2
Oddly I also see this in config, though I've seen these kinds of
things before.
mmdiag --config |grep verbsRdmaMinBytes
verbsRdmaMinBytes 8192
We're on a recent efix.
Current GPFS build: "4.2.2.3 efix21 (1028007)".
--
Ed Wahl
Ohio Supercomputer Center
614-292-9302 <tel:%28614%29%20292-9302>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
