Well, I don't know if there's an archive of the list, so this what Doug
wrote:
"
A more efficient and slightly more complex approach would be to build large
indexes in RAM, and copy them to disk with IndexWriter.addIndexes:
IndexWriter fsWriter = new IndexWriter(new File(...), analyzer, true);
while (... more docs to index...)
RAMDirectory ramDir = new RAMDirectory();
IndexWriter ramWriter = new IndexWriter(ramDir, analyzer, true);
... add 100,000 docs to ramWriter ...
ramWriter.optimize();
ramWriter.close();
fsWriter.addIndexes(new Directory[] { ramDir });
}
fsWriter.optimize();
fsWriter.close();
"
Scott
> -----Original Message-----
> From: Paul Friedman [mailto:[EMAIL PROTECTED]]
> Sent: Friday, November 02, 2001 9:13 AM
> To: 'Lucene Users List'
> Subject: RE: Indexing problem
>
>
> Where can I get Doug's example of indexing in memory and then
> writing it out
> to disk? I just recently subscribed to this list and I can't
> find it in the
> archive.
>
> Thanks.
> Paul
>
> -----Original Message-----
> From: Scott Ganyo [mailto:[EMAIL PROTECTED]]
> Sent: Friday, November 02, 2001 7:14 AM
> To: 'Lucene Users List'
> Subject: RE: Indexing problem
>
>
> Yes. You have too many open files. There are a few things
> you can try. 1)
> Increase the number of file handles your system has
> available. Yes, there
> is a setting for this in Windows. 2) Make sure that you have the
> IndexWriter.maxMergeDocs set to Integer.MAX_VALUE (the
> default). 3) Try
> smaller values for IndexWriter.mergeFactor (default is 10).
> 4) When all
> else fails, do all your indexing in memory and then write it
> out to disk
> when you're done. Doug posted an example of this just a
> couple days ago.
>
> Scott
>
> > -----Original Message-----
> > From: Daryl Thachuk [mailto:[EMAIL PROTECTED]]
> > Sent: Thursday, November 01, 2001 7:46 PM
> > To: [EMAIL PROTECTED]
> > Subject: Indexing problem
> >
> >
> > Hi
> >
> > Since upgrading to 1.2 we've started getting the following
> > error when
> > creating an index in a directory with a large amount of
> > files. Previous
> > versions of Lucene were quite happy to index this directory.
> >
> > Any thoughts as to the cause?
> >
> > -d
> >
> > java.io.FileNotFoundException:
> > /private/Network/Servers/montage/Volumes/Disk2/Users/daryl/Lib
> > rary/Index/index5.
> > mtlibx/_n8.f3 (Too many open files)
> > at java.io.RandomAccessFile.open(Native Method)
> > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:143)
> > at
> > org.apache.lucene.store.FSInputStream$Descriptor.<init>(Unknown
> > Source)
> > at org.apache.lucene.store.FSInputStream.<init>(Unknown Source)
> > at org.apache.lucene.store.FSDirectory.openFile(Unknown Source)
> > at
> > org.apache.lucene.index.SegmentReader.openNorms(Unknown Source)
> > at org.apache.lucene.index.SegmentReader.<init>(Unknown Source)
> > at
> > org.apache.lucene.index.IndexWriter.mergeSegments(Unknown Source)
> > at
> > org.apache.lucene.index.IndexWriter.mergeSegments(Unknown Source)
> > at
> > org.apache.lucene.index.IndexWriter.maybeMergeSegments(Unknown
> > Source)
> > at
> > org.apache.lucene.index.IndexWriter.addDocument(Unknown Source)
> > at IndexCreator.indexDocs(IndexCreator.java:75)
> > at IndexCreator.indexDocs(IndexCreator.java:67)
> > at IndexCreator.indexDocs(IndexCreator.java:67)
> > at IndexCreator.indexDocs(IndexCreator.java:67)
> > at IndexCreator.indexDocs(IndexCreator.java:67)
> > at IndexCreator.indexDocs(IndexCreator.java:67)
> > at IndexCreator.createIndex(IndexCreator.java:44)
> >
> > ------
> > Daryl Thachuk [EMAIL PROTECTED]
> > Montage Technologies Inc.
> > http://www.montagetech.com
> >
> >
> > --
> > To unsubscribe, e-mail:
> <mailto:[EMAIL PROTECTED]>
> For additional commands, e-mail:
> <mailto:[EMAIL PROTECTED]>
>
> --
> To unsubscribe, e-mail:
> <mailto:[EMAIL PROTECTED]>
> For additional commands, e-mail:
> <mailto:[EMAIL PROTECTED]>
>
