I've indexed 250,000 items (database rows, not files) with Lucene on a system like this:
[otis@kyle blink]$ ulimit unlimited [otis@kyle blink]$ tcsh > limit cputime unlimited filesize unlimited datasize unlimited stacksize 8192 kbytes coredumpsize 1000000 kbytes memoryuse unlimited descriptors 1024 memorylocked unlimited maxproc 8192 openfiles 1024 I didn't encounter the 'too many open files' error. You could change your code to optimize index after adding every X documents to the index. If you add Documents to a RAMDirectory-based IndexWriter, and them use IndexWriter's addIndexes method, you will have an optimized index after calling addIndexes method, plus you'll speed up indexing. Otis --- "Nader S. Henein" <[EMAIL PROTECTED]> wrote: > The issue is the same with Lucene when you index, if you're > indexing 200 000 files the amount of files created by the index cause > the > index system to run out of file handles, is there an equation to find > out > how many files will be created by the indexer based on the number of > files > we want indexed ? , becasue that would give me an idea of how many > file > handles > I would need > > -----Original Message----- > From: Otis Gospodnetic [mailto:[EMAIL PROTECTED]] > Sent: Tuesday, April 09, 2002 6:42 PM > To: Lucene Users List; [EMAIL PROTECTED] > Subject: RE: too many open files in system > > > This sounds like a question for Cocoon people, as what you are asking > about seems to be related to Cocoon's usage of Lucene, not the core > Lucene API. > > Otis > > --- Ian Forsyth <[EMAIL PROTECTED]> wrote: > > I'm calling in response to the LuceneCocconIndexer, is this class > an > > XML > > file indexer? (excuse my ignorance i am just stepping into this > whole > > thing..) > > > > I do a lot of development with PHP, on different platforms > > (WIN,*NIXES) and > > I want to get into indexing data... > > > > I am wondering if there are classes existing that index XML files, > > and for > > search results, spit out an xml stream.. > > > > I am figuring I would send results to lucene via the cmd line, but > i > > am not > > sure how i would capture the xml results.. Do I have to have Lucene > > make a > > physical XML file called for instance searchresults.xml then i set > my > > XSLT > > parser to that file.. ? > > > > I am very curious about this whole thing and any help would be a > > super help. > > > > Regards, > > Ian > > > > -----Original Message----- > > From: Nader S. Henein [mailto:[EMAIL PROTECTED]] > > Sent: Tuesday, April 09, 2002 8:56 AM > > To: Lucene Users List; [EMAIL PROTECTED] > > Subject: RE: too many open files in system > > > > > > that depends on how many files you're indexing .. I still have to > > figure out > > too what logic does the LuceneCocoonIndexer adhere when it is > > creating the > > index files > > > > > > -----Original Message----- > > From: root [mailto:[EMAIL PROTECTED]] > > Sent: Tuesday, April 09, 2002 4:50 PM > > To: Lucene Users List > > Subject: Re: too many open files in system > > > > > > On Tuesday, 9. April 2002 14:08, you wrote: > > > root wrote: > > > > Doesn't Lucene releases the filehandles?? > > > > > > > > because I get "too many open files in system" after running > > lucene a > > > > while! > > > > > > Are you closing the readers and writers after you've finished > using > > them? > > > > > > cheers, > > > > > > Chris > > > > > > Yes I close the readers and writers! > > > > > > @Nader S. Henein > > > > If I increase the filehandles, to what count should I increase > them? > > > > > > -- > > To unsubscribe, e-mail: > > <mailto:[EMAIL PROTECTED]> > > For additional commands, e-mail: > > <mailto:[EMAIL PROTECTED]> > > > > > > > > -- > > To unsubscribe, e-mail: > > <mailto:[EMAIL PROTECTED]> > > For additional commands, e-mail: > > <mailto:[EMAIL PROTECTED]> > > > > > > > > -- > > To unsubscribe, e-mail: > > <mailto:[EMAIL PROTECTED]> > > For additional commands, e-mail: > > <mailto:[EMAIL PROTECTED]> > > > > > __________________________________________________ > Do You Yahoo!? > Yahoo! Tax Center - online filing with TurboTax > http://taxes.yahoo.com/ > > -- > To unsubscribe, e-mail: > <mailto:[EMAIL PROTECTED]> > For additional commands, e-mail: > <mailto:[EMAIL PROTECTED]> > > > > -- > To unsubscribe, e-mail: > <mailto:[EMAIL PROTECTED]> > For additional commands, e-mail: > <mailto:[EMAIL PROTECTED]> > __________________________________________________ Do You Yahoo!? Yahoo! Tax Center - online filing with TurboTax http://taxes.yahoo.com/ -- To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]> For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
