On Mon, Apr 28, 2008 at 8:39 AM, Gwern Branwen <[EMAIL PROTECTED]> wrote: > > I'm not sure what you mean by co_slurpy being strinct. It looks to me like > > it's got adequate unsafeInterleaveIO to make it lazy. > > -- > > David Roundy > > Well, it does have plenty of unsafeInterleaveIO, that is true, but the issue > here is readFilePS: readFilePS is completely strict, it reads the entire file > into memory (per docs and implementation). So, actually running readFilePS > may get delayed to the last second, but once readFilePS gets inspected, it'll > immediately do its best to suck in all 9 gigs or whatever. > > This is why replacing readFilePS in co_slurp_helper with mmapFilePS is such > a time saver - it is lazy and pretends to read in all 9 gigs immediately, but > since with -s, we ultimately only read the first 4096 characters, only a > little bit will ever actually get page-faulted into memory. > > (The problem with mmapFilePS is that as lispy mentions, on my 64-bit system, > mmapFilePS can no longer handle >3 gig files while readFilePS scaled up to at > least 9gigs, albeit slowly.)
The other problem is that mmapFilePS will cause darcs to fail entirely on large repositories (with more than 1k files) due to sucking up all the system's file handles. I think this is a more common use case in darcs than 9g files. Of course, we could refuse to mmap small files (we already do this for very small files), and that could alleviate the problem considerably. Another problem is that using mmap on files in the working directory can lead to segfaults, since the user is allowed to edit files in the working directory while darcs runs--or at least I don't want to segfault if the user does this. David _______________________________________________ darcs-users mailing list [email protected] http://lists.osuosl.org/mailman/listinfo/darcs-users
