On Mon, Apr 1, 2013 at 4:38 AM, Phong Vo <[email protected]> wrote: >> Are there any plans to implement support for releasing memory obtained >> via |mmap(..., MAP_ANON)| to the system (via |munmap()|) anytime soon >> ? >> The issue I got bug reports about is that if ksh93 allocates very >> large arrays as function-local variable then leaving the function >> doesn't release the memory... even if it's in the two-digit GB range. >> The issue is that this puts the system's VM subsystem under major >> stress... and IMO more stress than allocators which explicitly tell >> the underlying kernel that the memory is temporarily not needed. > >> The algorithm I'm looking for should work like this: >> 1. Each time memory is compacted the allocator should issue a >> |madvise(data, s, MADV_DONTNEED)| over the affected data (note that >> |data| must be page aligned and that the backingstore for these pages >> is "released"... which means the next time these pages are accessed >> they may be zero-filled). See >> ttp://lxr.mozilla.org/mozilla-central/source/js/src/gc/Memory.cpp#374 >> for an example how this is done. >> 2. If there is excessive (say... 1/128 of the currently mapped memory >> is in use) memory mapped by the libast allocator but not in use by the >> consuming applications some of this unused memory should explicitly be >> |munmap()|'ed. > > This will require some work. Currently, the allocation algorithm > treats all mmap-ed memory that is contiguous as a single segment > and a segment can be released all or none.
Erm... even if two different memory areas obtained via |mmap(...,MAP_ANON)| are near to each other they are normally seperated at least by so-called "red (zone) pages" (before and after the mapping) which trigger SIGSEGV/SIGBUS on any access... so they can never follow each other directly... so one empty segment may likely span a whole |mmap()|ed area... > It will take some time > for me to change that. Unfortunately, I don't have much time at > the moment. OK... ... what about implementing at least the first part of the proposal, e.g. if memory areas get "compacted" and a continuous area is longer than 8*pagesize then align the pointer to pagesize (and adjust the size) and then call |madvise(data, s, MADV_DONTNEED)| for that area... would that be possible (note that |madvise(data, s, MADV_DONTNEED)| _MAY_ have the same effect as a |memset(data, 0, s)| because the kernel will simply release the underlying backing store for the pages... the next time they are accessed they _MAY_ be subject to zero-fill) ? AFAIK it should significantly improve the issue in "big data" scenarious... ---- Bye, Roland -- __ . . __ (o.\ \/ /.o) [email protected] \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer /O /==\ O\ TEL +49 641 3992797 (;O/ \/ \O;) _______________________________________________ ast-developers mailing list [email protected] http://lists.research.att.com/mailman/listinfo/ast-developers
