On Wed, Jun 27, 2012 at 6:40 PM, Roland Mainz <roland.ma...@nrubsig.org> wrote:
> On Thu, Jun 21, 2012 at 8:18 AM, Roland Mainz <roland.ma...@nrubsig.org>
> wrote:
>> Attached (as "bigfilecommandsubstitution001.sh.txt") is a short demo
>> which shows a problem in ast-ksh.2012-06-12 with memory usage for
>> str="$( < bigfile )" (and similar usage). The problem seems to be that
>> a larger input file size (for example 32MB) triggers excessive memory
>> usage, usually well beyond 512MB on Solaris 11/AMD64.
>>
>> Running the demo triggers the following error (this comes from the
>> limits defined via "ulimit" in the test (e.g. we assume that a file
>> size of 32MB will not cause more than 64MB of extra memory usage in
>> ksh93 (ksh93 itself is granted 16MB for itself&&running the script,
>> e.g. the ulimit limits are 80MB total in this case. However even that
>> is very very generous)):
>> -- snip --
>> $ env - LC_ALL=C ~/bin/ksh /tmp/bigmem.sh
>> -rw-r--r-- 1 test001 users 33226753 Jun 21 08:10 mytestfile
>> bigfilecommandsubstitution001.sh: line 36: storage allocator out of
>> space on 8486912 byte request ( region 545751040 segments 133 busy
>> 151:8769120:8421280 free 139:536966640:8355744 ) [Not enough space]
>> -- snip --
>
> I hit another issue which may be related...
> ... attached (as "freedesktop.org.xml.bz2",
> MD5(freedesktop.org.xml.bz2)= cbf57feee8a4244cbbc2b39817c3620a) is a
> test file.
>
> If I unpack the file into a command substutution I get a total length
> of 524289 bytes:
> -- snip --
> $ ~/bin/ksh -c 'x=$( bzcat <freedesktop.org.xml.bz2 ) ; printf "%s\n"
> "$x" | wc -c'
> 524289
> -- snip --
> ... but if I test the file outside a command substitution I get this:
> -- snip --
> $ bzcat <freedesktop.org.xml.bz2 | wc -c
> 1713510
> -- snip --
> ... any idea why this happens (yes... I checked whether the file as
> zero bytes via $ bzcat < freedesktop.org.xml.bz2 | od -t x1 | fgrep '
> 00' # ... there are none...) ?
... and finally "read -N<n>" for very large <n> reads nothing and
doesn't print an error message (test file "freedesktop.org.xml.bz2"
was attached to previous email):
-- snip --
$ ~/bin/ksh -c 'read -N1713510 xmltext < <(bzcat
freedesktop.org.xml.bz2) ; printf $"#filesize=%d\n" "${#xmltext}"'
#filesize=0
-- snip --
----
Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.ma...@nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
_______________________________________________
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers
