Glenn, David, Suse 12.2, with ast-ksh.20121016 command substitutions still use read() to read data instead of mmap(), as you can see with this tests: strace ~/bin/ksh -c 'x=$(seq 400000) ; true' 2>&1 | fgrep 'read(' | wc -l 50 strace ~/bin/ksh -c 'x=$(seq 800000) ; true' 2>&1 | fgrep 'read(' | wc -l 92 strace ~/bin/ksh -c 'x=$(seq 1600000) ; true' 2>&1 | fgrep 'read(' | wc -l 187
The builtin cat command does not use mmap() either, as you can see in this tests: strace ~/bin/ksh -c 'builtin cat ; builtin rm ; seq 1600000 >s1 ; cat s1 >s2 ; rm s1 s2' 2>&1 | fgrep 'read(' | wc -l 187 strace ~/bin/ksh -c 'builtin cat ; builtin rm ; seq 3200000 >s1 ; cat s1 >s2 ; rm s1 s2' 2>&1 | fgrep 'read(' | wc -l 382 strace ~/bin/ksh -c 'builtin cat ; builtin rm ; seq 6400000 >s1 ; cat s1 >s2 ; rm s1 s2' 2>&1 | fgrep 'read(' | wc -l 773 On Solaris, the Sun /usr/bin/cat uses mmap(), and for this reason, out runs AST cat, even the libcmd built in version, by a factor of 1.9-2.7 for files larger than 10MB. Olga On Mon, Sep 24, 2012 at 10:02 PM, Glenn Fowler <g...@research.att.com> wrote: > > On Sat, 22 Sep 2012 19:30:17 +0200 Irek Szczesniak wrote: >> On Sat, Sep 22, 2012 at 11:41 AM, Cedric Blancher >> <cedric.blanc...@googlemail.com> wrote: >> > On 22 September 2012 10:14, Cedric Blancher >> > <cedric.blanc...@googlemail.com> wrote: >> >> On 17 September 2012 00:15, Roland Mainz <roland.ma...@nrubsig.org> wrote: >> >>> * Issues: >> >>> - We have to make sure that memory allocation doesn't get excessive >> >>> - Currently it seems that the stream returned by |sftmp()| somehow >> >>> causes the shell not to use |mmap()| to read the temporary file's >> >>> (used if we spill over or call an external process) content even for >> >>> "insanely" (in this context :-) ) large file sizes (e.g. 2^20 bytes) >> >>> ... erm... what exactlly causes this ? >> >> >> >> BTW: mmap() is not used for ANY files on Fedora Linux with a 64bit >> >> ksh93. I've just observed that shcomp munches a 100MB script via >> >> read() and not mmap(), which is IMO unfortunate. >> > >> > Thanks to Olga I narrowed the issue down to MAP_TYPE. The sfio code >> > assumes mmap() is available when MAP_TYPE is set but newer versions of >> > Linux no longer set MAP_TYPE. No MAP_TYPE, no mmap(). What a shit. >> > Who's actually monitoring performance on Linux? >> > >> Calm down, Ced. > >> I tried to have a stake in this since neither Glenn or Roland seem to >> reply and the problem is IMO severe. of course, first I'd like to >> thank Al Viro and Linus Torvalds for giving me some insight into the >> Linux VM system and the Linux mmap() implementation. > >> First cause of the problem is that src/lib/libast/features/mmap sets >> does not set #define _mmap_worthy to 2. The kernel.org people shook >> their heads in disbelieve and said the iffe test is just 'bollocks' - >> per definition of mmap() it should always be preferred if you intend >> to read more than one or two pages worth of data. The test doesn't >> quality as proper test anyway since it operates on a tiny amount of >> data (Al said around 10000 times more data must be processed to make >> this test valid) and does not test different filesystems. > >> The second cause is that all these calls pass a valid buf to >> sfsetbuf() - which disabled mmap() usage. >> ksh -c 'builtin cat ; cat ../tst/bigscript.sh >trash ; true' >> ksh -c 'x=$( < ../tst/bigscript.sh ) ; true' >> ksh -c 'builtin tail ; tail -101000000 ../tst/bigscript.sh >trash ; true' > >> I think that fact that these three do not use mmap() is a severe >> regression and MUST be fixed before the next alpha. > > I was out for a few days > I'll address the _mmap_worthy tests and let ksh builtin buffering to dgk > > the sfio source > #ifdef MAP_TYPE > is a bug and has been changed to > #if _mmap_worthy > > the _mmap_worthy test served its purpose when it was first written in 1997 > there were some bad mmap() implementations that had a noticable affect > on sfio performance > > as with most iffe tests, if it doesn't result in build errors or runtime > failures, then it is left alone; as native implementations improve, > assumptions > made by some tests may become obsolete, as in this case > > in 2012 the _mmap_worthy tests may be bollocks > but in 1997 some mmap implementations were equally bollocks > > the normal course of events is for someone to ask why code, or an iffe, test > performs this way or that, usually sans ad-codinum attacks, and that results > in > a revisit to the code > > I'm looking at a few different typical sfio usage patterns and will get back > on that > -- , _ _ , { \/`o;====- Olga Kryzhanovska -====;o`\/ } .----'-/`-/ olga.kryzhanov...@gmail.com \-`\-'----. `'-..-| / http://twitter.com/fleyta \ |-..-'` /\/\ Solaris/BSD//C/C++ programmer /\/\ `--` `--` _______________________________________________ ast-developers mailing list ast-developers@research.att.com https://mailman.research.att.com/mailman/listinfo/ast-developers