Re: [DNG] Caching leads to unresponsiveness
> Linux uses all available more for caching of filesystems. When copying large > files to slow network filesystems (nfs, smb, sshfs, davfs) it takes a long > time until such allocated memory becomes free. When these network > filesystems saturate memory linux becomes very unresponsive. It can take > minutes to start applications. > > Is there a way to limit memory usage of network filesystems? I can think of two, but both might require a bit of coding: - open(2) the target file(s) with O_DIRECT or maybe O_SYNC. I don't think cp(1) has that as an argument though. And few remote filesystems support a sync flag... - rate limit the file transfer - in other words issue the write(2) calls at a pace which matches the bandwidth available. Again, I don't believe standard cp(1) has that feature. If it were me I would use the double tar trick, but write a rate-limiting pipe program to fit between them: tar -c -C /home/source -f- . | pipelimit -r 2M | tar -xv -C /mnt/target -f- where a simple pipelimit looks like: result = read(STDIN_FILENO, buffer, BUFFER_SIZE); usleep(small_delay); write(STDOUT_FILENO, buffer, result); Conventional unix/linux assumes that filesystems are "fast", and one doesn't have to worry about the link properties/bandwidth to the disk - you are bumping your head against that assumption regards marc ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] Caching leads to unresponsiveness
Hi Joachim. Joachim Fahrner - 11.08.17, 17:43: > Linux uses all available more for caching of filesystems. When copying > large files to slow network filesystems (nfs, smb, sshfs, davfs) it > takes a long time until such allocated memory becomes free. When these > network filesystems saturate memory linux becomes very unresponsive. It > can take minutes to start applications. > > Is there a way to limit memory usage of network filesystems? Not specifically for network filesystems, but for all filesystems. See - /proc/sys/vm/dirty_background_bytes (and ratio… but I´d set it in bytes) - sysctl - /etc/sysctl.d - Kernel documentation: sysctl/vm.txt - If you want to get rid of caches for specific files or run a workload in a way that tells the kernel not to cache, see package nocache Linus once recommended to set this to about what your storage can write out in about 2-3 seconds AFAIR. I concur. I´d set this to about no more than 4-5 seconds. Its enough for Linux to optimize write ordering for rotating disks. Linus said the usual tresholds of 10% was made in times when computers had way less memory. Unfortunately the default value was never replaced by some kind of autotuning. I use this for example: martin@merkaba:~> cat /etc/sysctl.d/vm.conf # Only cache what my SSDs can write a few seconds vm.dirty_background_bytes = 1073741824 vm.dirty_bytes = 2147483648 Especially when local storage reads are faster than NFS writes then its crucical to reduce this to a sane value. This can help especially well in combination with rsync which sync()´s on a file after writing it. I once found a 2x speedup between large and low dirty memory threshold. Also it tends to create bulky writeout before (piling up dirty memory, then having to write it in a sudden… pile it up again and so on). This is the short version (lol!). I teach this in my Linux performance analysis & tuning courses. Beware of old pages about this. They may contain misleading recommendations. Thanks, -- Martin ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] Caching leads to unresponsiveness
On 08/11/2017 10:43 AM, Joachim Fahrner wrote: Linux uses all available more for caching of filesystems. When copying large files to slow network filesystems (nfs, smb, sshfs, davfs) it takes a long time until such allocated memory becomes free. When these network filesystems saturate memory linux becomes very unresponsive. It can take minutes to start applications. Is there a way to limit memory usage of network filesystems? I'm not sure if there is a way to limit cache usage by network filesystems specifically. The best resource I've seen on Linux using memory to cache filesystems is here: http://www.linuxatemyram.com/index.html If the cache truly is conflicting with your applications, the notes at the bottom of this page about /proc/sys/vm/swappiness may help: http://www.linuxatemyram.com/play.html Patrick ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] Caching leads to unresponsiveness
Am 2017-08-11 17:43, schrieb Joachim Fahrner: Linux uses all available more Should be "memory", not "more" ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
