Hi,
I implemented this, and I do have a question --
What should the MergeFactor and MaxMergeDocuments be ?
Cheers,
Winton
>Well, I don't know if there's an archive of the list, so this what Doug
>wrote:
>
>"
>A more efficient and slightly more complex approach would be to build
>large
>indexes in RAM, and copy them to disk with IndexWriter.addIndexes:
> IndexWriter fsWriter = new IndexWriter(new File(...), analyzer, true);
> while (... more docs to index...)
> RAMDirectory ramDir = new RAMDirectory();
> IndexWriter ramWriter = new IndexWriter(ramDir, analyzer, true);
> ... add 100,000 docs to ramWriter ...
> ramWriter.optimize();
> ramWriter.close();
> fsWriter.addIndexes(new Directory[] { ramDir });
> }
> fsWriter.optimize();
> fsWriter.close();
>"
>
>Scott
>
>> -----Original Message-----
>> From: Paul Friedman [mailto:[EMAIL PROTECTED]]
>> Sent: Friday, November 02, 2001 9:13 AM
>> To: 'Lucene Users List'
>> Subject: RE: Indexing problem
>>
>>
>> Where can I get Doug's example of indexing in memory and then
>> writing it out
>> to disk? I just recently subscribed to this list and I can't
>> find it in the
>> archive.
>>
>> Thanks.
>> Paul
>>
>> -----Original Message-----
>> From: Scott Ganyo [mailto:[EMAIL PROTECTED]]
>> Sent: Friday, November 02, 2001 7:14 AM
>> To: 'Lucene Users List'
>> Subject: RE: Indexing problem
>>
>>
>> Yes. You have too many open files. There are a few things
>> you can try. 1)
>> Increase the number of file handles your system has
>> available. Yes, there
>> is a setting for this in Windows. 2) Make sure that you have the
>> IndexWriter.maxMergeDocs set to Integer.MAX_VALUE (the
>> default). 3) Try
>> smaller values for IndexWriter.mergeFactor (default is 10).
>> 4) When all
>> else fails, do all your indexing in memory and then write it
>> out to disk
>> when you're done. Doug posted an example of this just a
>> couple days ago.
>>
>> Scott
>>
>> > -----Original Message-----
>> > From: Daryl Thachuk [mailto:[EMAIL PROTECTED]]
>> > Sent: Thursday, November 01, 2001 7:46 PM
>> > To: [EMAIL PROTECTED]
>> > Subject: Indexing problem
>> >
>> >
>> > Hi
>> >
>> > Since upgrading to 1.2 we've started getting the following
>> > error when
>> > creating an index in a directory with a large amount of
>> > files. Previous
>> > versions of Lucene were quite happy to index this directory.
>> >
>> > Any thoughts as to the cause?
>> >
>> > -d
>> >
>> > java.io.FileNotFoundException:
>> > /private/Network/Servers/montage/Volumes/Disk2/Users/daryl/Lib
>> > rary/Index/index5.
>> > mtlibx/_n8.f3 (Too many open files)
>> > at java.io.RandomAccessFile.open(Native Method)
>> > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
>> > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:143)
>> > at
>> > org.apache.lucene.store.FSInputStream$Descriptor.<init>(Unknown
>> > Source)
>> > at org.apache.lucene.store.FSInputStream.<init>(Unknown Source)
>> > at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
>> > at
>> > org.apache.lucene.index.SegmentReader.openNorms(Unknown Source)
>> > at org.apache.lucene.index.SegmentReader.<init>(Unknown Source)
>> > at
>> > org.apache.lucene.index.IndexWriter.mergeSegments(Unknown Source)
>> > at
>> > org.apache.lucene.index.IndexWriter.mergeSegments(Unknown Source)
>> > at
>> > org.apache.lucene.index.IndexWriter.maybeMergeSegments(Unknown
>> > Source)
>> > at
>> > org.apache.lucene.index.IndexWriter.addDocument(Unknown Source)
>> > at IndexCreator.indexDocs(IndexCreator.java:75)
>> > at IndexCreator.indexDocs(IndexCreator.java:67)
>> > at IndexCreator.indexDocs(IndexCreator.java:67)
>> > at IndexCreator.indexDocs(IndexCreator.java:67)
>> > at IndexCreator.indexDocs(IndexCreator.java:67)
>> > at IndexCreator.indexDocs(IndexCreator.java:67)
>> > at IndexCreator.createIndex(IndexCreator.java:44)
>> >
>> > ------
>> > Daryl Thachuk [EMAIL PROTECTED]
>> > Montage Technologies Inc.
>> > http://www.montagetech.com
>> >
>> >
>> > --
>> > To unsubscribe, e-mail:
>> <mailto:[EMAIL PROTECTED]>
>> For additional commands, e-mail:
>> <mailto:[EMAIL PROTECTED]>
>>
>> --
>> To unsubscribe, e-mail:
>> <mailto:[EMAIL PROTECTED]>
>> For additional commands, e-mail:
>> <mailto:[EMAIL PROTECTED]>
>>
--
To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
