Probably a bit off topic at this point, but but your response confuses me - MADV_WILLNEED and POSIX_FADV_WILLNEED will bring the pages into ram, just like faulting in mmap()'ed pages by hand, or read()ing it into memory. In my experience, read() and fadvise() are faster than mmap()+faulting everything in, or madvise(). Of course, read()ing it in means it has to be swapped out and can't just be dropped. If you want to suck the entire file in at some point, probably the best way is to fadvise() it in, then mmap() it and use it from there.
-- Steve On Thu, Oct 22, 2009 at 2:52 PM, Scott Hess <[email protected]> wrote: > Faulting it in by hand is only helpful if we're right! If we're > wrong, it could evict other stuff from memory to support a feature > which a user may not use until the memory is faulted back out anyhow. > > [From the rest of the thread, though, it sounds like maybe we should > just fix hunspell to be more efficient for our usage.] > > -scott > > > On Thu, Oct 22, 2009 at 2:42 PM, Steve Vandebogart <[email protected]> > wrote: > > It's been awhile since I looked at this, but the email I was able to dig > up > > suggests that madvise is no faster than faulting in the mmap()ed region > by > > hand. However, using posix_fadvise should give the same speeds as > read()ing > > it into memory. IIRC though, posix_fadvise will only read so much in a > > single request and will let readahead handle the rest after that. > > -- > > Steve > > > > On Thu, Oct 22, 2009 at 2:27 PM, Scott Hess <[email protected]> wrote: > >> > >> On Linux what about mmap() and then madvise() with MADV_WILLNEED? [or > >> posix_fadvise() with POSIX_FADV_WILLNEED on the file descriptor). > >> > >> -scott > >> > >> > >> On Thu, Oct 22, 2009 at 2:06 PM, Steve Vandebogart < > [email protected]> > >> wrote: > >> > If you plan to read the entire file, mmap()ing it, then faulting it in > >> > will > >> > be slower than read()ing it, at least in some Linux versions. I never > >> > pinned down exactly why, but I think the kernel read-ahead mechanism > >> > works > >> > slightly differently. > >> > -- > >> > Steve > >> > > >> > On Thu, Oct 22, 2009 at 2:02 PM, Chris Evans <[email protected]> > >> > wrote: > >> >> > >> >> There's also option 3) > >> >> Pre-fault the mmap()ed region in the file thread upon dictionary > >> >> initialization. > >> >> On Linux at least, that may give you better behaviour than malloc() + > >> >> read() in the event of memory pressure. > >> >> Cheers > >> >> Chris > >> >> > >> >> On Thu, Oct 22, 2009 at 1:39 PM, Evan Stade <[email protected]> > >> >> wrote: > >> >>> > >> >>> Hi all, > >> >>> > >> >>> At its last meeting the jank task force discussed improving > >> >>> responsiveness of the spellchecker but we didn't come to a solid > >> >>> conclusion so I thought I'd bring it up here to see if anyone else > has > >> >>> opinions. The main concern is that we don't block the IO thread on > >> >>> file access. To this end, I recently moved initialization of the > >> >>> spellchecker from the IO thread to the file thread. However, instead > >> >>> of reading in the spellchecker dictionary in one solid chunk, we > >> >>> memory-map it. Then later we check individual words on the IO > thread, > >> >>> which will be slow since the dictionary starts off effectively > >> >>> completely paged out. The proposal is that we read in the dictionary > >> >>> at spellchecker intialization instead of memory mapping it. > >> >>> > >> >>> Memory mapping pros: > >> >>> - possibly uses less overall memory, depending on the structure of > the > >> >>> dictionary and the usage pattern of the user. > >> >>> - <strike>loading the dictionary doesn't block for a long > >> >>> time</strike> this one no longer occurs either way due to my recent > >> >>> refactoring > >> >>> > >> >>> Reading it all at once pros: > >> >>> - costly disk accesses are kept to the file thread (excepting future > >> >>> memory paging) > >> >>> - overall disk access time is probably lower (since we can read in > the > >> >>> dict in one chunk) > >> >>> > >> >>> For reference, the English dictionary is about 500K, and most > >> >>> dictionaries are under 2 megs, some (such as Hungarian) are much > >> >>> higher, but no dictionary is over 10 megs. > >> >>> > >> >>> Opinions? > >> >>> > >> >>> -- Evan Stade > >> >>> > >> >>> > >> >> > >> >> > >> >> > >> > > >> > > >> > > >> > > >> > > > > > > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
