Re: HAVE_MMAP still necessary ?
* MP singular...@gmail.com schrieb: We could have some get that file into memory call, that will try to use mmap if possible and store pointer to freeing the block (that would call munmap, free or some other method depending on how the block was acquired) Exactly :) The actual fs (and only it) should be responsible for getting the block into memory and later freeing it again - the clients even don't have to know that something like mmap exists at all. Such an interface could be very useful for everyone who just needs some file area in memory and doesnt want to care about sequential reading. Let's first try it out libmvfs, once it works fine, we can add it to mcvfs ... But we need to cope with situations, where the file won't fit in RAM and won't fit in virtual memory either. For example 8gb file on i386 architecture with 2 gb of ram. The vfs call will simply return an appropriate error if there's not enough memory available (whether virtual or physical is out the client's scope). cu -- - Enrico Weigelt== metux IT service - http://www.metux.de/ - Please visit the OpenSource QM Taskforce: http://wiki.metux.de/public/OpenSource_QM_Taskforce Patches / Fixes for a lot dozens of packages in dozens of versions: http://patches.metux.de/ - ___ Mc-devel mailing list http://mail.gnome.org/mailman/listinfo/mc-devel
Re: HAVE_MMAP still necessary ?
I believe the idea is to be able to view parts of huge file without loading them to RAM first. (for really big files, they may not even fit in RAM.) Some really big files won't fit even in virtual memory. For example on 32bit architectures (like commonly used i386) you have 2gb usable virtul memory for each process (rest 2 gb is reserverd for kernelspace). Since some part of the 2gb space is already occupied by libraries, etc, you usually can get perhaps 1 gb of virtual memory in which you can mmap things (since the virtual memory range in which you will mmap the file have to be contiguous). And files larger than 4 Gb are quite common (large archives, apache logs from busy servers, etc ...). So mmap may be unavailable at compile time (due to platform issues ...) or at runtime (if you request block too large) We could have some get that file into memory call, that will try to use mmap if possible and store pointer to freeing the block (that would call munmap, free or some other method depending on how the block was acquired) But we need to cope with situations, where the file won't fit in RAM and won't fit in virtual memory either. For example 8gb file on i386 architecture with 2 gb of ram. Martin Petricek ___ Mc-devel mailing list http://mail.gnome.org/mailman/listinfo/mc-devel
Re: HAVE_MMAP still necessary ?
On Sat 2008-12-27 21:43:53, Enrico Weigelt wrote: * Enrico Weigelt weig...@metux.de schrieb: Hi folks, I really wonder whether the mmap() stuff is still needed at all. It doesnt seem to be really used anywhere. Ups, didn't look hard enough (just scanned the vfs subdir) ;-O Okay, there're mainly two mmap()-using places: a) cmd.c: compare_files() - it uses the mmap() call directly (w/o going over mcvfs), and it seems to work on local files only. wouldn't it make sense to let it run via mcvfs ? b) view.c: it tries to mmap() in the file, obviously to let the kernel do all the loading. BUT: do we *really* want mmap() here, or just some get me that file into memory()-call (same in cmd.c) ? I believe the idea is to be able to view parts of huge file without loading them to RAM first. (for really big files, they may not even fit in RAM.) -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html ___ Mc-devel mailing list http://mail.gnome.org/mailman/listinfo/mc-devel
Re: HAVE_MMAP still necessary ?
a) cmd.c: compare_files() - it uses the mmap() call directly (w/o going over mcvfs), and it seems to work on local files only. wouldn't it make sense to let it run via mcvfs ? b) view.c: it tries to mmap() in the file, obviously to let the kernel do all the loading. BUT: do we *really* want mmap() here, or just some get me that file into memory()-call (same in cmd.c) ? mmap is more efficient, because the kernel can throw those pages out at any time (as it knows what the backing store for the file is), so under memory pressure it can alleviate the system load easily. Loading the file ourselves means that we load it into dirty pages, effectively taking memory, and forcing the kernel to write the data to swap under load, or to keep the data in memory even when not needed. I do not like the idea of dropping mmap. In case we just want to have an faster way for loading files into memory (in case it's supported), I suggest some new vfs operation for that, let's call it loadFile() - it returns some FILE_DATA structure, containing size, buffer ptr and a callback vector for free'ing. Everyone who wants the whole file (or large blocks) just uses this call instead of ugly #ifdef's, and it's up to vfs to decide what to do behind the scenes. You are trying to invent a bloated replacement for something that the kernel can do really well. ___ Mc-devel mailing list http://mail.gnome.org/mailman/listinfo/mc-devel
Re: HAVE_MMAP still necessary ?
* Miguel de Icaza mig...@novell.com schrieb: Hi, mmap is more efficient, because the kernel can throw those pages out at any time (as it knows what the backing store for the file is), so under memory pressure it can alleviate the system load easily. Absolutely right, but we have to keep in mind, that it's only supported on some filesystems (not even all local ones). So we need some clean way to handle it. The current implementation is quite unclean, sometimes called mcvfs's mmap, sometimes the libc's one. That should really be cleaned up. In case we just want to have an faster way for loading files into memory (in case it's supported), I suggest some new vfs operation for that, let's call it loadFile() - it returns some FILE_DATA structure, containing size, buffer ptr and a callback vector for free'ing. Everyone who wants the whole file (or large blocks) just uses this call instead of ugly #ifdef's, and it's up to vfs to decide what to do behind the scenes. You are trying to invent a bloated replacement for something that the kernel can do really well. My point being, we probably don't really want to have mmap itself, but an efficient way for getting some file (or large parts of it) into memory - that's a totally different requirement, and using mmap() is just one way to do it (if supported by the underlying fs). For the (mcfs-)clients this is an additional logic, which should be hidden behind the scenes - the individual fs should know best what to do, once we've introduced an appropriate API call. BTW: some fs'es (on certain platforms) might find a more clever way, even if mmap() isn't directly supported (eg. emulating it in userspace ;-P) - we really should leave this to the fs. cu -- - Enrico Weigelt== metux IT service - http://www.metux.de/ - Please visit the OpenSource QM Taskforce: http://wiki.metux.de/public/OpenSource_QM_Taskforce Patches / Fixes for a lot dozens of packages in dozens of versions: http://patches.metux.de/ - ___ Mc-devel mailing list http://mail.gnome.org/mailman/listinfo/mc-devel
Re: HAVE_MMAP still necessary ?
* Enrico Weigelt weig...@metux.de schrieb: Hi folks, I really wonder whether the mmap() stuff is still needed at all. It doesnt seem to be really used anywhere. Ups, didn't look hard enough (just scanned the vfs subdir) ;-O Okay, there're mainly two mmap()-using places: a) cmd.c: compare_files() - it uses the mmap() call directly (w/o going over mcvfs), and it seems to work on local files only. wouldn't it make sense to let it run via mcvfs ? b) view.c: it tries to mmap() in the file, obviously to let the kernel do all the loading. BUT: do we *really* want mmap() here, or just some get me that file into memory()-call (same in cmd.c) ? In case we just want to have an faster way for loading files into memory (in case it's supported), I suggest some new vfs operation for that, let's call it loadFile() - it returns some FILE_DATA structure, containing size, buffer ptr and a callback vector for free'ing. Everyone who wants the whole file (or large blocks) just uses this call instead of ugly #ifdef's, and it's up to vfs to decide what to do behind the scenes. cu -- - Enrico Weigelt== metux IT service - http://www.metux.de/ - Please visit the OpenSource QM Taskforce: http://wiki.metux.de/public/OpenSource_QM_Taskforce Patches / Fixes for a lot dozens of packages in dozens of versions: http://patches.metux.de/ - ___ Mc-devel mailing list http://mail.gnome.org/mailman/listinfo/mc-devel
