I have drafted the proposal on the official GSoC website . This is the link to my proposal http://goo.gl/uYXrV . Please do let me know if anything needs to be changed ,added or removed.
I will keep on working on it till the deadline on the 8th. On Wed, Apr 6, 2011 at 11:41 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > That test code looks good -- you really should have seen awful > performance had you used O_DIRECT since you read byte by byte. > > A more realistic test is to read a whole buffer (eg 4 KB is what > Lucene now uses during merging, but we'd probably up this to like 1 MB > when using O_DIRECT). > > Linus does hate O_DIRECT (see http://kerneltrap.org/node/7563), and > for good reason: its existence means projects like ours can use it to > "work around" limitations in the Linux IO apis that control the buffer > cache when, otherwise, we might conceivably make patches to fix Linux > correctly. It's an escape hatch, and we all use the escape hatch > instead of trying to fix Linux for real... > > For example the NOREUSE flag is a no-op now in Linux, which is a > shame, because that's precisely the flag we'd want to use for merging > (along with SEQUENTIAL). Had that flag been implemented well, it'd > give better results than our workaround using O_DIRECT. > > Anyway, giving how things are, until we can get more control (waaaay > up in Javaland) over the buffer cache, O_DIRECT (via native directory > impl through JNI) is our only real option, today. > > More details here: > http://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html > > Note that other OSs likely do a better job and actually implement > NOREUSE, and similar APIs, so the generic Unix/WindowsNativeDirectory > would simply use NOREUSE on these platforms for I/O during segment > merging. > > Mike > > http://blog.mikemccandless.com > > On Wed, Apr 6, 2011 at 11:56 AM, Varun Thacker > <varunthacker1...@gmail.com> wrote: > > Hi. I wrote a sample code to test out speed difference between SEQUENTIAL > > and O_DIRECT( I used the madvise flag-MADV_DONTNEED) reads . > > > > This is the link to the code: http://pastebin.com/8QywKGyS > > > > There was a speed difference which when i switched between the two flags. > I > > have not used the O_DIRECT flag because Linus had criticized it. > > > > Is this what the flags are intended to be used for ? This is just a > sample > > code with a test file . > > > > On Wed, Apr 6, 2011 at 12:11 PM, Simon Willnauer > > <simon.willna...@googlemail.com> wrote: > >> Hey Varun, > >> On Tue, Apr 5, 2011 at 11:07 PM, Michael McCandless > >> <luc...@mikemccandless.com> wrote: > >>> Hi Varun, > >>> > >>> Those two issues would make a great GSoC! Comments below... > >> +1 > >>> > >>> On Tue, Apr 5, 2011 at 1:56 PM, Varun Thacker > >>> <varunthacker1...@gmail.com> wrote: > >>> > >>>> I would like to combine two tasks as part of my project > >>>> namely-Directory createOutput and openInput should take an IOContext > >>>> (Lucene-2793) and compliment it by Generalize DirectIOLinuxDir to > >>>> UnixDir (Lucene-2795). > >>>> > >>>> The first part of the project is aimed at significantly reducing time > >>>> taken to search during indexing by adding an IOContext which would > >>>> store buffer size and have options to bypass the OS’s buffer cache > >>>> (This is what causes the slowdown in search ) and other hints. Once > >>>> completed I would move on to Lucene-2795 and generalize the Directory > >>>> implementation to make a UnixDirectory . > >>> > >>> So, the first part (LUCENE-2793) should cause no change at all to > >>> performance, functionality, etc., because it's "merely" installing the > >>> plumbing (IOContext threaded throughout the low-level store APIs in > >>> Lucene) so that higher levels can send important details down to the > >>> Directory. We'd fix IndexWriter/IndexReader to fill out this > >>> IOContext with the details (merging, flushing, new reader, etc.). > >>> > >>> There's some fun/freedom here in figuring out just what details should > >>> be included in IOContext... (eg: is it low level "set buffer size to 4 > >>> KB" > >>> or is it high level "I am opening a new near-real-time reader"). > >>> > >>> This first step is a rote cutover, just changing APIs but in no way > >>> taking advantage of the new APIs. > >>> > >>> The 2nd step (LUCENE-2795) would then take advantage of this plumbing, > >>> by creating a UnixDir impl that, using JNI (C code), passes advanced > >>> flags when opening files, based on the incoming IOContext. > >>> > >>> The goal is a single UnixDir that has ifdefs so that it's usable > >>> across multiple Unices, and eg would use direct IO if the context is > >>> merging. If we are ambitious we could rope Windows into the mix, too, > >>> and then this would be NativeDir... > >>> > >>> We can measure success by validating that a big merge while searching > >>> does not hurt search performance? (Ie we should be able to reproduce > >>> the results from > >>> http://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html > ). > >> > >> Thanks for the summary mike! > >>> > >>>> I have spoken to Micheal McCandless and Simon Willnauer about > >>>> undertaking these tasks. Micheal McCandless has agreed to mentor me . > >>>> I would love to be able to contribute and learn from Apache Lucene > >>>> community this summer. Also I would love suggestions on how to make my > >>>> application proposal stronger. > >>> > >>> I think either Simon or I can be the "official" mentor, and then the > >>> other one of us (and other Lucene committers) will support/chime > >>> in... > >> > >> I will take the official responsibility here once we are there! > >> simon > >>> > >>> This is an important change for Lucene! > >>> > >>> Mike > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >>> For additional commands, e-mail: dev-h...@lucene.apache.org > >>> > >>> > >> > > > > > > > > -- > > > > > > Regards, > > Varun Thacker > > http://varunthacker.wordpress.com > > > > > > > > > -- Regards, Varun Thacker http://varunthacker.wordpress.com
