On 28 June 2012 18:20, Glenn Fowler <g...@research.att.com> wrote: > > On Thu, 28 Jun 2012 16:30:37 +0200 Lionel Cons wrote: >> On 28 June 2012 09:28, Glenn Fowler <g...@research.att.com> wrote: >> > >> > On Wed, 27 Jun 2012 12:55:56 +0200 Cedric Blancher wrote: >> >> On 21 June 2012 08:18, Roland Mainz <roland.ma...@nrubsig.org> wrote: >> >> > Hi! >> >> > >> >> > ---- >> >> > >> >> > Attached (as "bigfilecommandsubstitution001.sh.txt") is a short demo >> >> > which shows a problem in ast-ksh.2012-06-12 with memory usage for >> >> > str="$( < bigfile )" (and similar usage). The problem seems to be that >> >> > a larger input file size (for example 32MB) triggers excessive memory >> >> > usage, usually well beyond 512MB on Solaris 11/AMD64. >> >> > >> >> > Running the demo triggers the following error (this comes from the >> >> > limits defined via "ulimit" in the test (e.g. we assume that a file >> >> > size of 32MB will not cause more than 64MB of extra memory usage in >> >> > ksh93 (ksh93 itself is granted 16MB for itself&&running the script, >> >> > e.g. the ulimit limits are 80MB total in this case. However even that >> >> > is very very generous)): >> >> > -- snip -- >> >> > $ env - LC_ALL=C ~/bin/ksh /tmp/bigmem.sh >> >> > -rw-r--r-- 1 test001 users 33226753 Jun 21 08:10 mytestfile >> >> > bigfilecommandsubstitution001.sh: line 36: storage allocator out of >> >> > space on 8486912 byte request ( region 545751040 segments 133 busy >> >> > 151:8769120:8421280 free 139:536966640:8355744 ) [Not enough space] >> >> > -- snip -- >> >> > >> >> > >> >> > ---- >> >> > >> >> > Bye, >> >> > Roland >> >> > >> >> > -- >> >> > __ . . __ >> >> > (o.\ \/ /.o) roland.ma...@nrubsig.org >> >> > \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer >> >> > /O /==\ O\ TEL +49 641 3992797 >> >> > (;O/ \/ \O;) >> >> > >> >> > _______________________________________________ >> >> > ast-developers mailing list >> >> > ast-developers@research.att.com >> >> > https://mailman.research.att.com/mailman/listinfo/ast-developers >> >> > >> > >> >> Glenn, David: This is still broken in ksh 2012-06-26 and is IMO >> >> showstopper because it prevents the use of larger texts in command >> >> substitutions: >> > >> >> ksh bigfile.sh >> >> -rw-r--r-- 1 ced cedhome 33226753 Jun 27 12:53 mytestfile >> >> bigfile.sh: line 35: storage allocator out of space on 2981888 byte >> >> request ( region 66813952 segments 49 busy 169:3271264:2916256 free >> >> 55:63535184:2850720 ) [Cannot allocate memory] >> > >> > this is a vmalloc fragmentation issue when mmap() is used to allocate >> > region blocks >> > coalescing adjacent mmap() blocks is not a portable op across mmap() >> > implementations >> > a side effect of this is fragmentation, sometimes very bad > >> BTW: Linux has mremap(), but that is mostly used for IO and not for >> allocations. > >> > >> > using mmap() instead of sbrk() by default started with the first >> > thread-safe vmalloc >> > implementation earlier this year; the ast beta 2012-06-28, to be posted >> > later today, >> > works around this by detecting if the current application requires >> > thread-safety >> > (via the ast aso(3) api); if it does require thread-safe then mmap() is >> > used, >> > otherwise it uses sbrk() (this is the case for all current ast commands), >> > falling >> > back to mmap() only if sbrk() fails >> > >> > the next iteration of thread-safe vmalloc is currently in test phase >> > it will use sbrk() by default in a thread-safe way (only restriction - no >> > api >> > but vmalloc(3) may call sbrk()), and the workaround above will not be >> > needed > >> Good lord. > >> Glenn, please do not use sbrk() in a shared library. This is not going >> to work well and will have serious consequences for stability if >> *ANYTHING* else uses sbrk(). We already had such issues in other >> shared libraries and they were almost impossible to find and were >> always not readily reproducible. > >> Lionel > >> FYI, we always patched ast-open and removed the sbrk() code (no, we >> did not use #if/#endif, we always removed the whole code to avoid >> having the slightest risk of running into bugs caused by multiple >> wannabe-tyrant owners of the sbrk() pool.) > > until 2011-11-11 when the thread-safe vmalloc was pulled in, sbrk() was the > default > pretty much all apps know not to use sbrk() in combination with malloc > until we see mmap()-style apis that support coalescing of already mapped > regions > (basically a mmap realloc) we are at a point where the fragmentation flagged > by this subject can only be handled by sbrk()
Why does your mmap backend have such fragmentation problems? Have you looked at libumem (*) and how they handle fragmentation? (*) ref http://en.wikipedia.org/wiki/Libumem and http://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libumem/ Notable feature is that it doesn't use sbrk by default and even overrides the user's choice if a competing allocator is detected. > anybody know any non-brk coalescing mmap api? I think the lack of coalescing is not the problem. The lack of fragmentation avoidance is IMO the main problem. Have a look at libumem and their solutions. > applications that know there will be an sbrk() conflict can use the vmalloc > discipline/method api to rule out sbrk() -- modulo those who disable sbrk() > at build time, we have not seen problems from this setup > > finally, the vmalloc sbrk() discipline already handles interlopers by always > checking the current brk with its saved brk -- if different then vmalloc > will only extend the brk beyond the current -- if all sbrk() consumers do > this, > *as they should if they were coded properly*, they can cooperate Well, it sounds that no libc allocator using sbrk does do those checks, otherwise it doesn't explain why most every Gnome app crashes when libast's sbrk allocator is used in parallel to the system one. This happens on FreeBSD, HP/UX, Linux and Solaris. That's why we *removed* libast's sbrk allocator before building and deployment. Lionel _______________________________________________ ast-developers mailing list ast-developers@research.att.com https://mailman.research.att.com/mailman/listinfo/ast-developers
