Exhausting VA space is not likely but keeping processes behaving nicely toward one another should be a good thing. You need to think about cases when many processes do large I/O at the same time and the physical memory available on the machine is far less than what the VA space can accommodate.
It's little known but Sfio does adaptive buffer filling to reduce read I/O, esp. when many seeks are done (hence most read data are wasted). The same strategy could be adapted to mapped I/O. We'll look into that. Phong > From [email protected] Mon Apr 15 10:53:27 2013 > Subject: Re: [ast-developers] mmap() for command substitutions still not > living up to its fullest potential? > To: Phong Vo <[email protected]> > Cc: [email protected], [email protected] > On Mon, Apr 15, 2013 at 4:40 PM, Phong Vo <[email protected]> wrote: > > > > The default size of the mapped memory for a stream is driven by a #define > > in a header file so it isn't hard to change it. I believe an application can > > also use sfsetbuf(f, NULL, size) to set a desired size for the mapped > > buffer. > > > > Generally speaking, Sfio and our other core libraries like CDT and Vmalloc > > have been around for a very long time and their default parameters tend to > > stay as they are until someone notices a performance issue. Well, not so > > much > > for Vmalloc anymore because it has been completely rewritten recently to > > deal > > with concurrency, both multiple threads and multiple processes for shared > > memory. > > > > Anyway, it is good that you bring up this issue with Sfio now. > > What do people think is a reasonable size to set the default mapped > > size to on a 64-bit machine? Keep in mind that there apps with many dozens > > files opened at the same time along with other large requirements for > > memory. > I think Roland's patch from > http://lists.research.att.com/pipermail/ast-developers/2013q2/002431.html > is sufficient for now because long before we exhaust VA space we run > out of patience to wait for the jobs to complete :) If that's not > sufficient then I'll suggest the '44bit clamp' Roland proposed to > partition 64bit VA space into 65536 files with 44bit chunks mapped > simultaneously. That just leaves 16 times the same amount of memory > for anon memory pages (which is many times the amount of world-wide > installed main memory in 2012). > Irek _______________________________________________ ast-developers mailing list [email protected] http://lists.research.att.com/mailman/listinfo/ast-developers
