Cool, your investigations are interesting. From the source code of hotspot, you have seen that there is MappedByteBuffer.load0 (which is protected, so invisible to user code), which does the same like MappedByteBuffer.load(), but without the stupid touching of every single page.
We could try to use reflection to invoke the native load0() method on MappedByteBuffer (we already use reflection to support unmapping). In that case we can issue madvise(MADV_WILLNEED) without loading the whole stuff into memory (as MappedByteBuffer.load() does). We could make this optional in MMapDirectory/ByteBufferIndexInput (like the unmapping support). ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Greg Bowyer [mailto:gbow...@fastmail.co.uk] > Sent: Tuesday, December 11, 2012 6:34 AM > To: lucene-...@apache.org > Subject: madvise and gregs hallucinations > > Since its too long (and has too much HTML and pictures and such forth) for > the mailing list I have a more detailed write up here > > http://people.apache.org/~gbowyer/madvise-perf/index.html > > However the short version. > > At $DAYJOB as part of moving to lucene 4.0 I have been looking at what we > can change in our modes of thinking, on of these relates (for us) to having a > separate legacy system that serves the stored data for the search engine > (which we imaginatively call docserve). > > Whilst playing with this I noticed that I lost a lot of performance rather > quickly, after a lot of digging it looks like getting a call in mmaping to > madvise > (specifically a call of the form madvise(addr, > MADV_WILLNEED) might improve search performance. > > Anyone have any thoughts and or ideas here, I am going to try to get a whole > heap more investigation done before I start messing with lucene's behavior > on mmap (because whilst it improves /my/ performance a lot it might be I > only noticed it since I am poking an open-wound YMMV), but there is > something of interest here. > > Then again this might be my own personal insanity .. :S > > -- Greg > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional > commands, e-mail: dev-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
