On Sat, Sep 22, 2012 at 7:30 PM, Irek Szczesniak <[email protected]> wrote:
> On Sat, Sep 22, 2012 at 11:41 AM, Cedric Blancher
> <[email protected]> wrote:
>> On 22 September 2012 10:14, Cedric Blancher
>> <[email protected]> wrote:
>>> On 17 September 2012 00:15, Roland Mainz <[email protected]> wrote:
>>>> * Issues:
>>>> - We have to make sure that memory allocation doesn't get excessive
>>>> - Currently it seems that the stream returned by |sftmp()| somehow
>>>> causes the shell not to use |mmap()| to read the temporary file's
>>>> (used if we spill over or call an external process) content even for
>>>> "insanely" (in this context :-) ) large file sizes (e.g. 2^20 bytes)
>>>> ... erm... what exactlly causes this ?
>>>
>>> BTW: mmap() is not used for ANY files on Fedora Linux with a 64bit
>>> ksh93. I've just observed that shcomp munches a 100MB script via
>>> read() and not mmap(), which is IMO unfortunate.
>>
>> Thanks to Olga I narrowed the issue down to MAP_TYPE. The sfio code
>> assumes mmap() is available when MAP_TYPE is set but newer versions of
>> Linux no longer set MAP_TYPE. No MAP_TYPE, no mmap(). What a shit.
>> Who's actually monitoring performance on Linux?
>>
> Calm down, Ced.
>
> I tried to have a stake in this since neither Glenn or Roland seem to
Roland is sick. Please wait.
> reply and the problem is IMO severe. of course, first I'd like to
> thank Al Viro and Linus Torvalds for giving me some insight into the
> Linux VM system and the Linux mmap() implementation.
>
> First cause of the problem is that src/lib/libast/features/mmap sets
> does not set #define _mmap_worthy to 2. The kernel.org people shook
> their heads in disbelieve and said the iffe test is just 'bollocks' -
> per definition of mmap() it should always be preferred if you intend
> to read more than one or two pages worth of data. The test doesn't
> quality as proper test anyway since it operates on a tiny amount of
> data (Al said around 10000 times more data must be processed to make
> this test valid) and does not test different filesystems.
In the now Oracle-canceled opensolaris.org ksh93-integration project
Roland, I and the Sun performance team, always hard coded _mmap_worthy
to 2, because the iffe script was flipping between 0, 1, 2 all the
time, depending on file system used (ZFS, UFS, tmpfs, NFS) and machine
load. I made performance comparisons hard.
Based on the feed back of the Sun performance team, _mmap_worthy
should always be 2. If there is ever a case that read() for a lot of
data, is faster than mmap() IO, then this is a bug in the Solaris
kernel, and must be fixed.
>
> The second cause is that all these calls pass a valid buf to
> sfsetbuf() - which disabled mmap() usage.
> ksh -c 'builtin cat ; cat ../tst/bigscript.sh >trash ; true'
> ksh -c 'x=$( < ../tst/bigscript.sh ) ; true'
> ksh -c 'builtin tail ; tail -101000000 ../tst/bigscript.sh >trash ; true'
>
> I think that fact that these three do not use mmap() is a severe
> regression and MUST be fixed before the next alpha.
Yes, please. This can not stand. IMO these cases should use mmap() for
IO, and performance.
Olga
--
, _ _ ,
{ \/`o;====- Olga Kryzhanovska -====;o`\/ }
.----'-/`-/ [email protected] \-`\-'----.
`'-..-| / http://twitter.com/fleyta \ |-..-'`
/\/\ Solaris/BSD//C/C++ programmer /\/\
`--` `--`
_______________________________________________
ast-developers mailing list
[email protected]
https://mailman.research.att.com/mailman/listinfo/ast-developers
