Re: mlock() issues

2010-10-22 Thread Michael van Elst
On Thu, Oct 21, 2010 at 10:40:15PM +0100, Sad Clouds wrote: I do realise this reinvents kernel file cache, but it gives you a lot more flexibility over what files get cached in memory and you can plug custom algorithms over how files get evicted from cache. NIH is the driving force for many

Re: mlock() issues

2010-10-22 Thread Matthew Mondor
On Fri, 22 Oct 2010 10:18:52 +0100 Sad Clouds cryintotheblue...@googlemail.com wrote: A pipelined request, say for 10 small files can be served with a single writev() system call (provided those files are cached in RAM), if you rely on kernel file cache, you need to issue 10 write() system

Re: mlock() issues

2010-10-22 Thread Matthew Mondor
On Fri, 22 Oct 2010 12:06:37 +0100 Sad Clouds cryintotheblue...@googlemail.com wrote: Well if you're allocating memory yourself, then you've just created your own application cache. Say many files were mapped in the process's address space, the OS would still be responsible of keeping

Re: mlock() issues

2010-10-21 Thread David Laight
On Wed, Oct 20, 2010 at 11:17:17PM +0100, Sad Clouds wrote: On Thu, 21 Oct 2010 00:02:53 +0200 Michael van Elst mlel...@serpens.de wrote: The UVM limit is global to all processes, obviously there should be such a limit to keep the system provided with unlocked pages. You could

Re: mlock() issues

2010-10-21 Thread Sad Clouds
On Thu, 21 Oct 2010 19:13:13 +0100 David Laight da...@l8s.co.uk wrote: A non-root user can then increase its own limit to 1/3 physmem, and root can change its own 'hard' and 'soft' limits to any value it cares. I think for some applications, having control over locking the entire physical

Re: mlock() issues

2010-10-20 Thread der Mouse
proc.curproc.rlimit.memorylocked.soft = 697976149 With all of the above set, for some reason it's not possible to lock more than 666MB. Well, 697976149 bytes is 665.6419+ MB, so it sounds to me as though it's doing exactly what it should be. Unless you're a disk manufacturer, in which case

Re: mlock() issues

2010-10-20 Thread Christos Zoulas
In article 20101020182953.752bfd63.cryintotheblue...@googlemail.com, Sad Clouds cryintotheblue...@googlemail.com wrote: Hi, I've been trying to figure out why it's not possible to lock more than 666MB of memory, and I'm beginning to think it might be a kernel issue. This is what I'm doing: Run

Re: mlock() issues

2010-10-20 Thread Sad Clouds
On Wed, 20 Oct 2010 20:06:41 + (UTC) mlel...@serpens.de (Michael van Elst) wrote: The soft rlimit and the UVM limit happen to be the same size, which is one third of the real memory. uvm_pdaemon.c: uvmexp.wiredmax = uvmexp.npages / 3; kern_proc.c: lim = MIN(VM_MAXUSER_ADDRESS,

Re: mlock() issues

2010-10-20 Thread Michael van Elst
On Wed, Oct 20, 2010 at 10:55:46PM +0100, Sad Clouds wrote: On Wed, 20 Oct 2010 20:06:41 + (UTC) mlel...@serpens.de (Michael van Elst) wrote: The soft rlimit and the UVM limit happen to be the same size, which is one third of the real memory. uvm_pdaemon.c: uvmexp.wiredmax =

Re: mlock() issues

2010-10-20 Thread Sad Clouds
On Thu, 21 Oct 2010 00:02:53 +0200 Michael van Elst mlel...@serpens.de wrote: The UVM limit is global to all processes, obviously there should be such a limit to keep the system provided with unlocked pages. You could probably make this configurable, but so far nobody had the need to lock a