modern operating systems/ kernel-level(since > 10 years) really occupy only buffers, which are "in use" .. so no waste of space, only a ssh session(socket) needs less space
but - in case of network issue/ outages.. network is preferred so all open sockets may fill up to the maximum.. so I agree, one need to be a bit carful here...
but as I said.. you can not cheat physics.. with RTT 180 MS you need to have according socket sizes.. (believe it or not ;-) ... sure, it does not mean, that your problem will disappear , because there are many potential possibilities for improvements... to complex to discuss/solve by email here.. I would recommend to open a PMR ...
in addition:
nfsprefetchstrategy ... ranges from 1...10 ..
when a multithreaded NFSd gets IOs from outside und directs them to GPFS .. the ordership of these IOs may not be consistent.. making it a bit harder to detect sequential IO access pattern... setting this value advises GPFS to consider all incoming IOs between <value> block boundaries as sequential
cheers
laff
Mit freundlichen Grüßen / Kind regards
Olaf Weiser
EMEA Storage Competence Center Mainz, German / IBM Systems, Storage Platform,
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
IBM Allee 1
71139 Ehningen
Phone: +49-170-579-44-66
E-Mail: olaf.wei...@de.ibm.com
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Martina Koederitz (Vorsitzende), Susanne Peter, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940
From: Loic Tortay <tor...@cc.in2p3.fr>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
Date: 11/10/2016 07:39 AM
Subject: Re: [gpfsug-discuss] Tuning AFM for high throughput/high IO over _really_ long distances
Sent by: gpfsug-discuss-boun...@spectrumscale.org
On 09/11/2016 21:53, Olaf Weiser wrote:
> let's say you have a RRT of 180 ms
> what you then need is your theoretical link speed - let's say 10 Gbit/s ...
> easily let's take 1 GB/s
>
> this means, you socket must be capable to take your bandwidth (data stream)
> during the "first" 180ms because it will take at least this time to get back the
> first ACKs .. .
> so 1 GB / s x 0,180 s = 1024 MB/s x 0,180 s ==>> 185 MB this means, you have
> to allow the operating system to accept socketsizes in that range...
>
> set something like this - but increase these values to 185 MB
> sysctl -w net.ipv4.tcp_rmem="12194304 12194304 12194304"
> sysctl -w net.ipv4.tcp_wmem="12194304 12194304 12194304"
> sysctl -w net.ipv4.tcp_mem="12194304 12194304 12194304"
> sysctl -w net.core.rmem_max=12194304
> sysctl -w net.core.wmem_max=12194304
> sysctl -w net.core.rmem_default=12194304
> sysctl -w net.core.wmem_default=12194304
> sysctl -w net.core.optmem_max=12194304
>
Hello,
In my opinion, some of these changes are, at best, misguided.
For instance, the unit for "tcp_mem" is not bytes but pages. It's also
not a parameter for buffers but a parameter influencing global kernel
memory management for TCP (source: Linux kernel documentation/source).
Or setting the maximum TCP ancillary data buffer size ("optmem_max") to
a very large value when, as far a I know/saw when testing AFM w/ NFS,
there is no ancillary data used.
Setting the min, default and max to the same value for the buffers is
also, in my opinion, highly debatable (do you really want, for instance,
each and every SSH connection to have 185 MB TCP buffers? -- 185 MB
being the value suggested above).
I have seen the same suggestions in the AFM documentation, and in my
opinion, along with the unhelpful "nfsPrefetchStrategy" recommandation
("it's critical: set it to at least 5 to 10", OK but how do I chose
between 5 to 10 or should I use 42?, what's the unit?, what are the
criteria?), these do not contribute to give a good understanding of the
configuration (let alone "optimization") required for AFM over NFS.
I must add that, in my opinion, I have "enough" experience with setting
these "sysctl" parameters of NFS "tuning" (so I'm not overwhelmed by the
complexity or whatever), to think something is really not right in that
part of the AFM documentation.
Loïc.
--
| Loïc Tortay <tor...@cc.in2p3.fr> - IN2P3 Computing Centre |
[attachment "smime.p7s" deleted by Olaf Weiser/Germany/IBM] _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Olaf Weiser
EMEA Storage Competence Center Mainz, German / IBM Systems, Storage Platform,
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
IBM Allee 1
71139 Ehningen
Phone: +49-170-579-44-66
E-Mail: olaf.wei...@de.ibm.com
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Martina Koederitz (Vorsitzende), Susanne Peter, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940
From: Loic Tortay <tor...@cc.in2p3.fr>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
Date: 11/10/2016 07:39 AM
Subject: Re: [gpfsug-discuss] Tuning AFM for high throughput/high IO over _really_ long distances
Sent by: gpfsug-discuss-boun...@spectrumscale.org
On 09/11/2016 21:53, Olaf Weiser wrote:
> let's say you have a RRT of 180 ms
> what you then need is your theoretical link speed - let's say 10 Gbit/s ...
> easily let's take 1 GB/s
>
> this means, you socket must be capable to take your bandwidth (data stream)
> during the "first" 180ms because it will take at least this time to get back the
> first ACKs .. .
> so 1 GB / s x 0,180 s = 1024 MB/s x 0,180 s ==>> 185 MB this means, you have
> to allow the operating system to accept socketsizes in that range...
>
> set something like this - but increase these values to 185 MB
> sysctl -w net.ipv4.tcp_rmem="12194304 12194304 12194304"
> sysctl -w net.ipv4.tcp_wmem="12194304 12194304 12194304"
> sysctl -w net.ipv4.tcp_mem="12194304 12194304 12194304"
> sysctl -w net.core.rmem_max=12194304
> sysctl -w net.core.wmem_max=12194304
> sysctl -w net.core.rmem_default=12194304
> sysctl -w net.core.wmem_default=12194304
> sysctl -w net.core.optmem_max=12194304
>
Hello,
In my opinion, some of these changes are, at best, misguided.
For instance, the unit for "tcp_mem" is not bytes but pages. It's also
not a parameter for buffers but a parameter influencing global kernel
memory management for TCP (source: Linux kernel documentation/source).
Or setting the maximum TCP ancillary data buffer size ("optmem_max") to
a very large value when, as far a I know/saw when testing AFM w/ NFS,
there is no ancillary data used.
Setting the min, default and max to the same value for the buffers is
also, in my opinion, highly debatable (do you really want, for instance,
each and every SSH connection to have 185 MB TCP buffers? -- 185 MB
being the value suggested above).
I have seen the same suggestions in the AFM documentation, and in my
opinion, along with the unhelpful "nfsPrefetchStrategy" recommandation
("it's critical: set it to at least 5 to 10", OK but how do I chose
between 5 to 10 or should I use 42?, what's the unit?, what are the
criteria?), these do not contribute to give a good understanding of the
configuration (let alone "optimization") required for AFM over NFS.
I must add that, in my opinion, I have "enough" experience with setting
these "sysctl" parameters of NFS "tuning" (so I'm not overwhelmed by the
complexity or whatever), to think something is really not right in that
part of the AFM documentation.
Loïc.
--
| Loïc Tortay <tor...@cc.in2p3.fr> - IN2P3 Computing Centre |
[attachment "smime.p7s" deleted by Olaf Weiser/Germany/IBM] _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
