Glenn, shared mmap() mapping do not have any impact on fork() performance, at least on VM architectures who can share pages (this is common practice since at least SystemV, and no modern Unix or Linux exists which does not do copy-on-write, but more on that below) The pages are not even touched, or looked at at fork() time, so even millions of mmap() pages have no impact. Only if the pages are touched the VM system will realize a fork() has happened, and *may* create a copy-on-write copy if you write to it. If you only read the pages nothing will happen.
Olga On Fri, Sep 13, 2013 at 8:55 PM, Glenn Fowler <g...@research.att.com> wrote: > > On Fri, 13 Sep 2013 19:08:31 +0200 Lionel Cons wrote: >> On 13 September 2013 18:50, Glenn Fowler <g...@research.att.com> wrote: >> > >> > On Fri, 13 Sep 2013 17:14:03 +0200 Lionel Cons wrote: >> >> On 18 June 2013 10:20, Glenn Fowler <g...@research.att.com> wrote: >> >> > >> >> > you showed ast grep strace output but not gnu grep >> >> > gnu /usr/bin/grep does read of varying chunk sizes on my redhat linux >> >> > for NFS files the chunk sizes were around 32Ki >> >> > >> >> > I added some test options to the SFIO_OPTIONS env var >> >> > >> >> > SFIO_OPTIONS=nomaxmap # disable mmap(), force read() >> >> > SFIO_OPTIONS=maxmap=-1 # map entire file >> >> > SFIO_OPTIONS=maxmap=1Gi # map 1Gi chunks etc. >> >> > >> >> > as long ast the buffer isn't trivially small I don't think lines >> >> > spanning buffer boundaries will be a drag on timing >> >> > >> >> > you might be able to set up a few experiments to see if there is >> >> > a knee in the performance curves where the space-time tradeoffs meet >> > >> >> The test results are in the email below: >> >> - bigger mmap() windows for sfio file IO. The break even we calculated >> >> is somewhere between 108-140MB window size on an Intel Sandy Bridge >> >> four way server and 144-157 for an AMD Jaguar prototype machine >> >> (X1150). This is likely to 'fix' most of the grep performance issue. >> >> Also forwarding a clarifying comment: " explain them that this is NOT >> >> akin to buffer bloat. It only the defines the maximum size of the >> >> address space window a process can have for this file. The kernel >> >> itself is in charge to select an appropriate number of pages it can >> >> donate to pass data through this window" >> > >> > thanks for the report >> > >> > based on this for the next alpha the default will be 128Mi on machines >> > where sizeof(void*)>=8 >> > >> > it can be tweaked with more feedback from other users / systems >> > > >> Glenn, there is a slight misconception: I said 'break even' (starting >> point where it starts to make sense), but 'the more, the better' is >> appropriate as well. Give that 64bit platforms have a very large >> address space something like >=1GB might be much better, at least in >> the presence of hugepages/largepages and page sharing, i.e. where >> multiple processes mapping the same file pages share a mapping. > > I did misinterpret a bit > there is still some concern about one size (1Gi) fits all > vs the range of hw ast may build on > > dgk also has concerns about the effect of 1 or more >=1Gi mappings on fork() > for this next release we'll keep it at 128Mi > and advertize SFIO_OPTIONS=maxmap=2Gi for those who wish to experiment > from those experiments we'd like to see the effects, if any, on fork() > performance > > _______________________________________________ > ast-users mailing list > ast-users@lists.research.att.com > http://lists.research.att.com/mailman/listinfo/ast-users -- , _ _ , { \/`o;====- Olga Kryzhanovska -====;o`\/ } .----'-/`-/ olga.kryzhanov...@gmail.com \-`\-'----. `'-..-| / http://twitter.com/fleyta \ |-..-'` /\/\ Solaris/BSD//C/C++ programmer /\/\ `--` `--` _______________________________________________ ast-users mailing list ast-users@lists.research.att.com http://lists.research.att.com/mailman/listinfo/ast-users
