I think readahead would be less interesting to Lucene; while we definitely want a certain amount of readahead (to "amortize" the seeking), too much readahead starts means evicting things from the IO cache. OSs already do a fair job (I think) of some amount of readahead, though if we do gain posix_fadvise in Java and we use it to advise to not IO cache those reads, I wonder how that impacts the OS's readahead...
Some serious empirical testing is needed. Let the machines tell us how they work ;) Mike On Tue, Jun 16, 2009 at 11:20 PM, Jason Rutherglen<jason.rutherg...@gmail.com> wrote: > Sorry, not portable, but POSIX_FADV_WILLNEED is which can be used with > posix_fadvise. > > On Tue, Jun 16, 2009 at 8:12 PM, Jason Rutherglen > <jason.rutherg...@gmail.com> wrote: >> >> Perhaps we'd also like to request readahead be included in JDK7? >> >> http://linux.die.net/man/2/readahead >> >> On Tue, Jun 16, 2009 at 9:03 AM, Michael McCandless >> <luc...@mikemccandless.com> wrote: >>> >>> Hmm... posix_fadvise lets you do this with a file descriptor; this >>> would be better for Lucene (per descriptor not per mapped region of >>> RAM) since we could "advise" independent of which FSDir impl is in >>> use... >>> >>> Mike >>> >>> On Tue, Jun 16, 2009 at 10:32 AM, Uwe Schindler<u...@thetaphi.de> wrote: >>> > But to use it, we should change MMapDirectory to also use the mapping >>> > when >>> > writing to files. I thought about it, it is very simple to implement >>> > (just >>> > copy the IndexInput and change all gets() to sets()) >>> > >>> > ----- >>> > Uwe Schindler >>> > H.-H.-Meier-Allee 63, D-28213 Bremen >>> > http://www.thetaphi.de >>> > eMail: u...@thetaphi.de >>> > >>> >> -----Original Message----- >>> >> From: Michael McCandless [mailto:luc...@mikemccandless.com] >>> >> Sent: Tuesday, June 16, 2009 4:22 PM >>> >> To: java-dev@lucene.apache.org >>> >> Cc: Alan Bateman; nio-disc...@openjdk.java.net >>> >> Subject: Re: madvise(ptr, len, MADV_SEQUENTIAL) >>> >> >>> >> Lucene could really make use of this method. When a segment merge >>> >> takes place, we can read & write many GB of data, which without >>> >> madvise on many OSs would effectively flush the IO cache (thus hurting >>> >> our search performance). >>> >> >>> >> Mike >>> >> >>> >> On Mon, Jun 15, 2009 at 6:01 PM, Jason >>> >> Rutherglen<jason.rutherg...@gmail.com> wrote: >>> >> > Thanks Alan. >>> >> > >>> >> > I cross posted this to the Lucene dev list where we are discussing >>> >> > using >>> >> > madvise for minimizing unnecessary IO cache usage when merging >>> >> > segments >>> >> > (where we really want the newly merged segments in the IO cache >>> >> > rather >>> >> than >>> >> > the old segment files). >>> >> > >>> >> > How would the advise method work? Would there need to be a hint in >>> >> > the >>> >> > FileChannel.map method? >>> >> > >>> >> > -J >>> >> > >>> >> > On Mon, Jun 15, 2009 at 12:36 AM, Alan Bateman >>> >> > <alan.bate...@sun.com> >>> >> wrote: >>> >> >> >>> >> >> Jason Rutherglen wrote: >>> >> >>> >>> >> >>> Is there going to be a way to do this in the new Java IO APIs? >>> >> >> >>> >> >> Good question, as it has come up a few times and is needed for some >>> >> >> important use-cases. A while back I looked into adding a >>> >> >> MappedByteBuffer#advise method to allow the application provide >>> >> >> hints >>> >> on the >>> >> >> expected usage but didn't complete it. We should probably look at >>> >> >> this >>> >> again >>> >> >> for jdk7. >>> >> >> >>> >> >> -Alan. >>> >> >> >>> >> > >>> >> > >>> >> >>> >> --------------------------------------------------------------------- >>> >> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >>> >> For additional commands, e-mail: java-dev-h...@lucene.apache.org >>> > >>> > >>> > >>> > --------------------------------------------------------------------- >>> > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >>> > For additional commands, e-mail: java-dev-h...@lucene.apache.org >>> > >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-dev-h...@lucene.apache.org >>> >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
