I would like to see lucene operate with hadoop As you rightly pointed out, writing using FSDirectory to DFS would be a performance issue.
I am interested in the idea. But i do not know how much time i can contribute to this because of the little time which i can spare. If anyone else is interested, can they join ? We can work on this together Rgds Prabhu On 3/26/06, Igor Bolotin <[EMAIL PROTECTED]> wrote: > > In my current project we needed a way to create very large Lucene indexes > on > Hadoop distributed file system. When we tried to do it directly on DFS > using > Nutch FsDirectory class - we immediately found that indexing fails because > DfsIndexOutput.seek() method throws UnsupportedOperationException. The > reason for this behavior is clear - DFS does not support random updates > and > so seek() method can't be supported (at least not easily). > > Well, if we can't support random updates - the question is: do we really > need them? Search in the Lucene code revealed 2 places which call > IndexOutput.seek() method: one is in TermInfosWriter and another one in > CompoundFileWriter. As we weren't planning to use CompoundFileWriter - the > only place that concerned us was in TermInfosWriter. > > TermInfosWriter uses IndexOutput.seek() in its close() method to write > total > number of terms in the file back into the beginning of the file. It was > very > simple to change file format a little bit and write number of terms into > last 8 bytes of the file instead of writing them into beginning of file. > The > only other place that should be fixed in order for this to work is in > SegmentTermEnum constructor - to read this piece of information at > position > = file length - 8. > > With this format hack - we were able to use FsDirectory to write index > directly to DFS without any problems. Well - we still don't index directly > to DFS for performance reasons, but at least we can build small local > indexes and merge them into the main index on DFS without copying big main > index back and forth. > > If somebody is interested - I can post our changes in TermInfosWriter and > SegmentTermEnum code, although they are pretty trivial. > > Best regards! > Igor > >
