Hi Joachim.

Joachim Fahrner - 11.08.17, 17:43:
> Linux uses all available more for caching of filesystems. When copying
> large files to slow network filesystems (nfs, smb, sshfs, davfs) it
> takes a long time until such allocated memory becomes free. When these
> network filesystems saturate memory linux becomes very unresponsive. It
> can take minutes to start applications.
> 
> Is there a way to limit memory usage of network filesystems?

Not specifically for network filesystems, but for all filesystems.

See

- /proc/sys/vm/dirty_background_bytes (and ratio… but I´d set it in bytes)
- sysctl
- /etc/sysctl.d
- Kernel documentation: sysctl/vm.txt
- If you want to get rid of caches for specific files or run a workload in a 
way 
that tells the kernel not to cache, see package nocache

Linus once recommended to set this to about what your storage can write out in 
about 2-3 seconds AFAIR. I concur. I´d set this to about no more than 4-5 
seconds. Its enough for Linux to optimize write ordering for rotating disks. 
Linus said the usual tresholds of 10% was made in times when computers had way 
less memory. Unfortunately the default value was never replaced by some kind 
of autotuning.

I use this for example:

martin@merkaba:~> cat /etc/sysctl.d/vm.conf 
# Only cache what my SSDs can write a few seconds
vm.dirty_background_bytes = 1073741824
vm.dirty_bytes = 2147483648

Especially when local storage reads are faster than NFS writes then its 
crucical to reduce this to a sane value. This can help especially well in 
combination with rsync which sync()´s on a file after writing it. I once found 
a 2x speedup between large and low dirty memory threshold. Also it tends to 
create bulky writeout before (piling up dirty memory, then having to write it 
in a sudden… pile it up again and so on).

This is the short version (lol!). I teach this in my Linux performance 
analysis & tuning courses. Beware of old pages about this. They may contain 
misleading recommendations.

Thanks,
-- 
Martin
_______________________________________________
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Reply via email to