On 09/11/2016 21:53, Olaf Weiser wrote:
let's say you have a RRT of 180 ms what you then need is your theoretical link speed - let's say 10 Gbit/s ... easily let's take 1 GB/sthis means, you socket must be capable to take your bandwidth (data stream) during the "first" 180ms because it will take at least this time to get back the first ACKs .. . so 1 GB / s x 0,180 s = 1024 MB/s x 0,180 s ==>> 185 MB this means, you have to allow the operating system to accept socketsizes in that range... set something like this - but increase these values to 185 MB sysctl -w net.ipv4.tcp_rmem="12194304 12194304 12194304" sysctl -w net.ipv4.tcp_wmem="12194304 12194304 12194304" sysctl -w net.ipv4.tcp_mem="12194304 12194304 12194304" sysctl -w net.core.rmem_max=12194304 sysctl -w net.core.wmem_max=12194304 sysctl -w net.core.rmem_default=12194304 sysctl -w net.core.wmem_default=12194304 sysctl -w net.core.optmem_max=12194304
Hello, In my opinion, some of these changes are, at best, misguided.For instance, the unit for "tcp_mem" is not bytes but pages. It's also not a parameter for buffers but a parameter influencing global kernel memory management for TCP (source: Linux kernel documentation/source). Or setting the maximum TCP ancillary data buffer size ("optmem_max") to a very large value when, as far a I know/saw when testing AFM w/ NFS, there is no ancillary data used. Setting the min, default and max to the same value for the buffers is also, in my opinion, highly debatable (do you really want, for instance, each and every SSH connection to have 185 MB TCP buffers? -- 185 MB being the value suggested above). I have seen the same suggestions in the AFM documentation, and in my opinion, along with the unhelpful "nfsPrefetchStrategy" recommandation ("it's critical: set it to at least 5 to 10", OK but how do I chose between 5 to 10 or should I use 42?, what's the unit?, what are the criteria?), these do not contribute to give a good understanding of the configuration (let alone "optimization") required for AFM over NFS.
I must add that, in my opinion, I have "enough" experience with setting these "sysctl" parameters of NFS "tuning" (so I'm not overwhelmed by the complexity or whatever), to think something is really not right in that part of the AFM documentation.
Loïc. -- | Loïc Tortay <tor...@cc.in2p3.fr> - IN2P3 Computing Centre |
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
