On Wed, 27 Jun 2012 14:28:29 +0200 Roland Mainz wrote: > On Wed, Jun 27, 2012 at 1:25 PM, Cedric Blancher > <cedric.blanc...@googlemail.com> wrote: > > On 27 June 2012 12:55, Cedric Blancher <cedric.blanc...@googlemail.com> > > wrote: > >> On 21 June 2012 08:18, Roland Mainz <roland.ma...@nrubsig.org> wrote: > >>> Attached (as "bigfilecommandsubstitution001.sh.txt") is a short demo > >>> which shows a problem in ast-ksh.2012-06-12 with memory usage for > >>> str="$( < bigfile )" (and similar usage). The problem seems to be that > >>> a larger input file size (for example 32MB) triggers excessive memory > >>> usage, usually well beyond 512MB on Solaris 11/AMD64. > >>> > >>> Running the demo triggers the following error (this comes from the > >>> limits defined via "ulimit" in the test (e.g. we assume that a file > >>> size of 32MB will not cause more than 64MB of extra memory usage in > >>> ksh93 (ksh93 itself is granted 16MB for itself&&running the script, > >>> e.g. the ulimit limits are 80MB total in this case. However even that > >>> is very very generous)): > >>> -- snip -- > >>> $ env - LC_ALL=C ~/bin/ksh /tmp/bigmem.sh > >>> -rw-r--r-- 1 test001 users 33226753 Jun 21 08:10 mytestfile > >>> bigfilecommandsubstitution001.sh: line 36: storage allocator out of > >>> space on 8486912 byte request ( region 545751040 segments 133 busy > >>> 151:8769120:8421280 free 139:536966640:8355744 ) [Not enough space] > >>> -- snip -- > [snip] > >> > >> Glenn, David: This is still broken in ksh 2012-06-26 and is IMO > >> showstopper because it prevents the use of larger texts in command > >> substitutions: > >> > >> ksh bigfile.sh > >> -rw-r--r-- 1 ced cedhome 33226753 Jun 27 12:53 mytestfile > >> bigfile.sh: line 35: storage allocator out of space on 2981888 byte > >> request ( region 66813952 segments 49 busy 169:3271264:2916256 free > >> 55:63535184:2850720 ) [Cannot allocate memory] > > > > I tried it with these values but it still fails: > > integer map_file_size=16 # 32MB in $( <file ) > > integer memory_limit=512 # 64MB memory limit to handle the cmd substitution > > > > So ksh 2012-06026 requires more than 512MB to process a 16MB string in > > a command substitution. This is really really bad.
> Grumpf... I agree... ;-(( > There is a 2nd issue... I am getting a couple of valgrind hits when a > libast-based application (e.g. ksh93 running the test script of this > bug) runs out of memory: > -- snip -- > $ LC_ALL=C valgrind --read-var-info=yes --num-callers=50 ~/bin/ksh bigfile.sh > ==22925== Memcheck, a memory error detector > ==22925== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al. > ==22925== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info > ==22925== Command: /home/test001/bin/ksh bigfile.sh > [snip] > ==22926== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 6 from 6) > -rw-r--r-- 1 test001 users 8306689 Jun 27 13:50 mytestfile > ==22925== Conditional jump or move depends on uninitialised value(s) > ==22925== at 0x552174: sfvprintf (sfvprintf.c:1017) > ==22925== by 0x4FE0D8: errorv (error.c:532) > ==22925== by 0x4FD280: error (error.c:350) > ==22925== by 0x52046E: nomalloc (memfatal.c:62) > ==22925== by 0x560DAA: _vmextend (vmprivate.c:85) > ==22925== by 0x55D555: bestalloc (vmbest.c:675) > ==22925== by 0x55E43E: bestresize (vmbest.c:935) > ==22925== by 0x563329: _ast_realloc (malloc.c:549) > ==22925== by 0x50D45A: stkgrow (stk.c:524) > ==22925== by 0x50C916: stkexcept (stk.c:195) > ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58) > ==22925== by 0x5559C0: sfwr (sfwr.c:231) > ==22925== by 0x555D83: sfwrite (sfwrite.c:108) > ==22925== by 0x457244: mac_copy (macro.c:2471) > ==22925== by 0x45683F: comsubst (macro.c:2244) > ==22925== by 0x4521B1: varsub (macro.c:1168) > ==22925== by 0x45079F: copyto (macro.c:633) > ==22925== by 0x44EBD1: sh_mactrim (macro.c:183) > ==22925== by 0x458AB5: nv_setlist (name.c:339) > ==22925== by 0x47FF94: sh_exec (xec.c:1208) > ==22925== by 0x418E2B: exfile (main.c:581) > ==22925== by 0x417FFE: sh_main (main.c:353) > ==22925== by 0x417128: main (pmain.c:45) > ==22925== > ==22925== Conditional jump or move depends on uninitialised value(s) > ==22925== at 0x552185: sfvprintf (sfvprintf.c:1019) > ==22925== by 0x4FE0D8: errorv (error.c:532) > ==22925== by 0x4FD280: error (error.c:350) > ==22925== by 0x52046E: nomalloc (memfatal.c:62) > ==22925== by 0x560DAA: _vmextend (vmprivate.c:85) > ==22925== by 0x55D555: bestalloc (vmbest.c:675) > ==22925== by 0x55E43E: bestresize (vmbest.c:935) > ==22925== by 0x563329: _ast_realloc (malloc.c:549) > ==22925== by 0x50D45A: stkgrow (stk.c:524) > ==22925== by 0x50C916: stkexcept (stk.c:195) > ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58) > ==22925== by 0x5559C0: sfwr (sfwr.c:231) > ==22925== by 0x555D83: sfwrite (sfwrite.c:108) > ==22925== by 0x457244: mac_copy (macro.c:2471) > ==22925== by 0x45683F: comsubst (macro.c:2244) > ==22925== by 0x4521B1: varsub (macro.c:1168) > ==22925== by 0x45079F: copyto (macro.c:633) > ==22925== by 0x44EBD1: sh_mactrim (macro.c:183) > ==22925== by 0x458AB5: nv_setlist (name.c:339) > ==22925== by 0x47FF94: sh_exec (xec.c:1208) > ==22925== by 0x418E2B: exfile (main.c:581) > ==22925== by 0x417FFE: sh_main (main.c:353) > ==22925== by 0x417128: main (pmain.c:45) > ==22925== > ==22925== Conditional jump or move depends on uninitialised value(s) > ==22925== at 0x55257F: sfvprintf (sfvprintf.c:1030) > ==22925== by 0x4FE0D8: errorv (error.c:532) > ==22925== by 0x4FD280: error (error.c:350) > ==22925== by 0x52046E: nomalloc (memfatal.c:62) > ==22925== by 0x560DAA: _vmextend (vmprivate.c:85) > ==22925== by 0x55D555: bestalloc (vmbest.c:675) > ==22925== by 0x55E43E: bestresize (vmbest.c:935) > ==22925== by 0x563329: _ast_realloc (malloc.c:549) > ==22925== by 0x50D45A: stkgrow (stk.c:524) > ==22925== by 0x50C916: stkexcept (stk.c:195) > ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58) > ==22925== by 0x5559C0: sfwr (sfwr.c:231) > ==22925== by 0x555D83: sfwrite (sfwrite.c:108) > ==22925== by 0x457244: mac_copy (macro.c:2471) > ==22925== by 0x45683F: comsubst (macro.c:2244) > ==22925== by 0x4521B1: varsub (macro.c:1168) > ==22925== by 0x45079F: copyto (macro.c:633) > ==22925== by 0x44EBD1: sh_mactrim (macro.c:183) > ==22925== by 0x458AB5: nv_setlist (name.c:339) > ==22925== by 0x47FF94: sh_exec (xec.c:1208) > ==22925== by 0x418E2B: exfile (main.c:581) > ==22925== by 0x417FFE: sh_main (main.c:353) > ==22925== by 0x417128: main (pmain.c:45) > ==22925== > ==22925== Conditional jump or move depends on uninitialised value(s) > ==22925== at 0x55258A: sfvprintf (sfvprintf.c:1030) > ==22925== by 0x4FE0D8: errorv (error.c:532) > ==22925== by 0x4FD280: error (error.c:350) > ==22925== by 0x52046E: nomalloc (memfatal.c:62) > ==22925== by 0x560DAA: _vmextend (vmprivate.c:85) > ==22925== by 0x55D555: bestalloc (vmbest.c:675) > ==22925== by 0x55E43E: bestresize (vmbest.c:935) > ==22925== by 0x563329: _ast_realloc (malloc.c:549) > ==22925== by 0x50D45A: stkgrow (stk.c:524) > ==22925== by 0x50C916: stkexcept (stk.c:195) > ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58) > ==22925== by 0x5559C0: sfwr (sfwr.c:231) > ==22925== by 0x555D83: sfwrite (sfwrite.c:108) > ==22925== by 0x457244: mac_copy (macro.c:2471) > ==22925== by 0x45683F: comsubst (macro.c:2244) > ==22925== by 0x4521B1: varsub (macro.c:1168) > ==22925== by 0x45079F: copyto (macro.c:633) > ==22925== by 0x44EBD1: sh_mactrim (macro.c:183) > ==22925== by 0x458AB5: nv_setlist (name.c:339) > ==22925== by 0x47FF94: sh_exec (xec.c:1208) > ==22925== by 0x418E2B: exfile (main.c:581) > ==22925== by 0x417FFE: sh_main (main.c:353) > ==22925== by 0x417128: main (pmain.c:45) > ==22925== > ==22925== Conditional jump or move depends on uninitialised value(s) > ==22925== at 0x552591: sfvprintf (sfvprintf.c:1030) > ==22925== by 0x4FE0D8: errorv (error.c:532) > ==22925== by 0x4FD280: error (error.c:350) > ==22925== by 0x52046E: nomalloc (memfatal.c:62) > ==22925== by 0x560DAA: _vmextend (vmprivate.c:85) > ==22925== by 0x55D555: bestalloc (vmbest.c:675) > ==22925== by 0x55E43E: bestresize (vmbest.c:935) > ==22925== by 0x563329: _ast_realloc (malloc.c:549) > ==22925== by 0x50D45A: stkgrow (stk.c:524) > ==22925== by 0x50C916: stkexcept (stk.c:195) > ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58) > ==22925== by 0x5559C0: sfwr (sfwr.c:231) > ==22925== by 0x555D83: sfwrite (sfwrite.c:108) > ==22925== by 0x457244: mac_copy (macro.c:2471) > ==22925== by 0x45683F: comsubst (macro.c:2244) > ==22925== by 0x4521B1: varsub (macro.c:1168) > ==22925== by 0x45079F: copyto (macro.c:633) > ==22925== by 0x44EBD1: sh_mactrim (macro.c:183) > ==22925== by 0x458AB5: nv_setlist (name.c:339) > ==22925== by 0x47FF94: sh_exec (xec.c:1208) > ==22925== by 0x418E2B: exfile (main.c:581) > ==22925== by 0x417FFE: sh_main (main.c:353) > ==22925== by 0x417128: main (pmain.c:45) > ==22925== > ==22925== Use of uninitialised value of size 8 > ==22925== at 0x5525D2: sfvprintf (sfvprintf.c:1030) > ==22925== by 0x4FE0D8: errorv (error.c:532) > ==22925== by 0x4FD280: error (error.c:350) > ==22925== by 0x52046E: nomalloc (memfatal.c:62) > ==22925== by 0x560DAA: _vmextend (vmprivate.c:85) > ==22925== by 0x55D555: bestalloc (vmbest.c:675) > ==22925== by 0x55E43E: bestresize (vmbest.c:935) > ==22925== by 0x563329: _ast_realloc (malloc.c:549) > ==22925== by 0x50D45A: stkgrow (stk.c:524) > ==22925== by 0x50C916: stkexcept (stk.c:195) > ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58) > ==22925== by 0x5559C0: sfwr (sfwr.c:231) > ==22925== by 0x555D83: sfwrite (sfwrite.c:108) > ==22925== by 0x457244: mac_copy (macro.c:2471) > ==22925== by 0x45683F: comsubst (macro.c:2244) > ==22925== by 0x4521B1: varsub (macro.c:1168) > ==22925== by 0x45079F: copyto (macro.c:633) > ==22925== by 0x44EBD1: sh_mactrim (macro.c:183) > ==22925== by 0x458AB5: nv_setlist (name.c:339) > ==22925== by 0x47FF94: sh_exec (xec.c:1208) > ==22925== by 0x418E2B: exfile (main.c:581) > ==22925== by 0x417FFE: sh_main (main.c:353) > ==22925== by 0x417128: main (pmain.c:45) > ==22925== > ==22925== Use of uninitialised value of size 8 > ==22925== at 0x5525E7: sfvprintf (sfvprintf.c:1030) > ==22925== by 0x4FE0D8: errorv (error.c:532) > ==22925== by 0x4FD280: error (error.c:350) > ==22925== by 0x52046E: nomalloc (memfatal.c:62) > ==22925== by 0x560DAA: _vmextend (vmprivate.c:85) > ==22925== by 0x55D555: bestalloc (vmbest.c:675) > ==22925== by 0x55E43E: bestresize (vmbest.c:935) > ==22925== by 0x563329: _ast_realloc (malloc.c:549) > ==22925== by 0x50D45A: stkgrow (stk.c:524) > ==22925== by 0x50C916: stkexcept (stk.c:195) > ==22925== by 0x566EA9: _sfexcept (sfexcept.c:58) > ==22925== by 0x5559C0: sfwr (sfwr.c:231) > ==22925== by 0x555D83: sfwrite (sfwrite.c:108) > ==22925== by 0x457244: mac_copy (macro.c:2471) > ==22925== by 0x45683F: comsubst (macro.c:2244) > ==22925== by 0x4521B1: varsub (macro.c:1168) > ==22925== by 0x45079F: copyto (macro.c:633) > ==22925== by 0x44EBD1: sh_mactrim (macro.c:183) > ==22925== by 0x458AB5: nv_setlist (name.c:339) > ==22925== by 0x47FF94: sh_exec (xec.c:1208) > ==22925== by 0x418E2B: exfile (main.c:581) > ==22925== by 0x417FFE: sh_main (main.c:353) > ==22925== by 0x417128: main (pmain.c:45) > ==22925== > bigfile.sh: line 35: storage allocator out of space on 6782976 byte > request ( region 338362368 segments 92 busy 166:7068096:6717344 free > 98:331282688:6651808 ) [Cannot allocate memory] > ==22925== > -- snip -- when I try this, with various ulimit sizes, I don't get the sfprintf() hit I either get "Valgrind's memory management: out of memory:" error meaning valgrind itself asked for memory and it failed or I get "/home/gsf/arch/linux.i386-64-test/bin/ksh: out of memory" _______________________________________________ ast-developers mailing list ast-developers@research.att.com https://mailman.research.att.com/mailman/listinfo/ast-developers