Glenn, David, Suse 12.2, with ast-ksh.20121016 command substitutions
still use read() to read data instead of mmap(), as you can see with
this tests:
strace ~/bin/ksh -c 'x=$(seq 400000) ; true' 2>&1 | fgrep 'read(' | wc -l
50
strace ~/bin/ksh -c 'x=$(seq 800000) ; true' 2>&1 | fgrep 'read(' | wc -l
92
strace ~/bin/ksh -c 'x=$(seq 1600000) ; true' 2>&1 | fgrep 'read(' | wc -l
187
The builtin cat command does not use mmap() either, as you can see in
this tests:
strace ~/bin/ksh -c 'builtin cat ; builtin rm ; seq 1600000 >s1 ; cat
s1 >s2 ; rm s1 s2' 2>&1 | fgrep 'read(' | wc -l
187
strace ~/bin/ksh -c 'builtin cat ; builtin rm ; seq 3200000 >s1 ; cat
s1 >s2 ; rm s1 s2' 2>&1 | fgrep 'read(' | wc -l
382
strace ~/bin/ksh -c 'builtin cat ; builtin rm ; seq 6400000 >s1 ; cat
s1 >s2 ; rm s1 s2' 2>&1 | fgrep 'read(' | wc -l
773
On Solaris, the Sun /usr/bin/cat uses mmap(), and for this reason, out
runs AST cat, even the libcmd built in version, by a factor of 1.9-2.7
for files larger than 10MB.
Olga
On Mon, Sep 24, 2012 at 10:02 PM, Glenn Fowler <g...@research.att.com> wrote:
>
> On Sat, 22 Sep 2012 19:30:17 +0200 Irek Szczesniak wrote:
>> On Sat, Sep 22, 2012 at 11:41 AM, Cedric Blancher
>> <cedric.blanc...@googlemail.com> wrote:
>> > On 22 September 2012 10:14, Cedric Blancher
>> > <cedric.blanc...@googlemail.com> wrote:
>> >> On 17 September 2012 00:15, Roland Mainz <roland.ma...@nrubsig.org> wrote:
>> >>> * Issues:
>> >>> - We have to make sure that memory allocation doesn't get excessive
>> >>> - Currently it seems that the stream returned by |sftmp()| somehow
>> >>> causes the shell not to use |mmap()| to read the temporary file's
>> >>> (used if we spill over or call an external process) content even for
>> >>> "insanely" (in this context :-) ) large file sizes (e.g. 2^20 bytes)
>> >>> ... erm... what exactlly causes this ?
>> >>
>> >> BTW: mmap() is not used for ANY files on Fedora Linux with a 64bit
>> >> ksh93. I've just observed that shcomp munches a 100MB script via
>> >> read() and not mmap(), which is IMO unfortunate.
>> >
>> > Thanks to Olga I narrowed the issue down to MAP_TYPE. The sfio code
>> > assumes mmap() is available when MAP_TYPE is set but newer versions of
>> > Linux no longer set MAP_TYPE. No MAP_TYPE, no mmap(). What a shit.
>> > Who's actually monitoring performance on Linux?
>> >
>> Calm down, Ced.
>
>> I tried to have a stake in this since neither Glenn or Roland seem to
>> reply and the problem is IMO severe. of course, first I'd like to
>> thank Al Viro and Linus Torvalds for giving me some insight into the
>> Linux VM system and the Linux mmap() implementation.
>
>> First cause of the problem is that src/lib/libast/features/mmap sets
>> does not set #define _mmap_worthy to 2. The kernel.org people shook
>> their heads in disbelieve and said the iffe test is just 'bollocks' -
>> per definition of mmap() it should always be preferred if you intend
>> to read more than one or two pages worth of data. The test doesn't
>> quality as proper test anyway since it operates on a tiny amount of
>> data (Al said around 10000 times more data must be processed to make
>> this test valid) and does not test different filesystems.
>
>> The second cause is that all these calls pass a valid buf to
>> sfsetbuf() - which disabled mmap() usage.
>> ksh -c 'builtin cat ; cat ../tst/bigscript.sh >trash ; true'
>> ksh -c 'x=$( < ../tst/bigscript.sh ) ; true'
>> ksh -c 'builtin tail ; tail -101000000 ../tst/bigscript.sh >trash ; true'
>
>> I think that fact that these three do not use mmap() is a severe
>> regression and MUST be fixed before the next alpha.
>
> I was out for a few days
> I'll address the _mmap_worthy tests and let ksh builtin buffering to dgk
>
> the sfio source
> #ifdef MAP_TYPE
> is a bug and has been changed to
> #if _mmap_worthy
>
> the _mmap_worthy test served its purpose when it was first written in 1997
> there were some bad mmap() implementations that had a noticable affect
> on sfio performance
>
> as with most iffe tests, if it doesn't result in build errors or runtime
> failures, then it is left alone; as native implementations improve,
> assumptions
> made by some tests may become obsolete, as in this case
>
> in 2012 the _mmap_worthy tests may be bollocks
> but in 1997 some mmap implementations were equally bollocks
>
> the normal course of events is for someone to ask why code, or an iffe, test
> performs this way or that, usually sans ad-codinum attacks, and that results
> in
> a revisit to the code
>
> I'm looking at a few different typical sfio usage patterns and will get back
> on that
>
--
, _ _ ,
{ \/`o;====- Olga Kryzhanovska -====;o`\/ }
.----'-/`-/ olga.kryzhanov...@gmail.com \-`\-'----.
`'-..-| / http://twitter.com/fleyta \ |-..-'`
/\/\ Solaris/BSD//C/C++ programmer /\/\
`--` `--`
_______________________________________________
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers
