On 16 April 2013 15:07, Glenn Fowler <[email protected]> wrote: > > On Tue, 16 Apr 2013 11:27:44 +0200 Wendy Lin wrote: >> On 15 April 2013 20:37, Phong Vo <[email protected]> wrote: >> > >> > Exhausting VA space is not likely but keeping processes behaving nicely >> > toward >> > one another should be a good thing. You need to think about cases when many >> > processes do large I/O at the same time and the physical memory available >> > on >> > the machine is far less than what the VA space can accommodate. >> > >> > It's little known but Sfio does adaptive buffer filling to reduce read I/O, >> > esp. when many seeks are done (hence most read data are wasted). The same >> > strategy could be adapted to mapped I/O. We'll look into that. > >> Commenting here since it affects the builtin grep > >> Phong, please keep the implementation KISS ("Keep it simple, stupid"). >> The VM system in Linux and AIX better work with large mappings which >> do not change than a moving target which changes frequently. >> In a 64bit application libast should map the whole file and let the >> kernel worry about the rest. > > we are concerned about ast/sfio being a good neighbor and not hogging all > resources
This should not be a problem. A 10TB file mirrored into a process's address space doesn't consume 10TB of main memory, it only maps those pages which have been accessed by the application. If the number of pages for a specific mapping or the number of file pages mapped exceeds certain limits the kernel will start to reclaim less used pages automatically using a garbage collection algorithm. This is done automatically and usually in parallel to the threads operating on the same file mapping since today's machines always have a spare CPU which can be used for such work. If the number of mapped pages is still growing faster than they can be reclaimed by the garbage collector then the accessing threads are throttled priority-wise to avoid that other processes or threads are being handled in an unfair manner. So the concern that too many mappings can hog all resources is unfounded. The modern Unix and Linux kernels protect themselves from such problems. > its fine for every app on a single user desktop to map in 16Gib files > how about each or 2^t threads 65536 threads is the limit. AFAIK all modern systems can not have more than 65535 file descriptors open. > doing that with 2^f different files of size 2^z > there has to be a point of diminishing returns > and that is what sfio concerns itself with > so that ast apps do not have to worry Use the proposed limit of 44bits then? > > _______________________________________________ > ast-developers mailing list > [email protected] > http://lists.research.att.com/mailman/listinfo/ast-developers Ced -- Cedric Blancher <[email protected]> Institute Pasteur _______________________________________________ ast-developers mailing list [email protected] http://lists.research.att.com/mailman/listinfo/ast-developers
