Thanks Mike for your directions. Yes, I am in fact using a single computer for my application, and your saying that in this case, multiple threads with a single IndexWriter wll give a better performance. Hmmm. I just wonder that since each IndexWriter has a single write.lock, this means that sitting on the CPU, we observe that at a particular instant, only a single thread is using the CPU, while others are kept waiting (obviously, there is no busy-waiting, just waiting. So, in this case, won't using a single-thread-using-one-IndexWriter be a better choice than multiple-threads-using-one-IndexWriter, since that would save the context-switching times ... (Just my two cents .. )
Your comments are eagerly awaited. Thanks Ajay Garg Michael McCandless-2 wrote: > > > If you have a single IndexWriter, then the buffer is flushed @ 16 MB > regardless of how many threads are adding to that buffer. > > If you are using multiple IndexWriters, writing to separate > directories and then merging at the end, then each one uses 16 MB. > But this isn't recommended for a single computer -- using multiple > threads with one IndexWriter should get you better performance. > > Note that this does not mean you can limit your JVM's heap to 16 MB > (for the single writer case): other things require memory, eg > merging, pulling documents from some source, etc. Furthermore, it's > generally best not to cut the heap size so close to your actual > memory usage that GC is forced to run excessively since that'll hurt > performance. > > Mike > > ajay_garg wrote: > >> >> Hi all. >> >> Lucene latest version - 2.3.0 says that the default behaviour of >> flushing >> from memory to file-system based index is based upon RAM usage - >> with 16 MB >> being the default value. Fine. Works for me, as long as I am using >> a single >> thread to write into the index. >> >> However, I have been trying to use multiple threads (say 'n'), to >> write into >> their corresponding individual indexes, all based into the file- >> system. Now, >> occasionally and randomly, I am getting "OutOfMemoryException - >> Java Memory >> Heap", while it works other times. I know, that there may be issues >> with my >> environment, but I haven't been able to figure out, why it works >> sometimes, >> and it does not. >> >> So, I will be obliged if the following doubts are cleared : >> >> a) Let's say there are 'n' threads, writing into 'n' indexes. Each >> thread >> has a unique individual indexer; this is implemented by passing the >> corresponding reference to the thread, which is then used in it's >> run() >> method. Now, as we know, 16 MB is the default size, before the docs >> are >> flushed to the file-system based index. Thus, my question is : >> >> does this hold for each thread indexer. In other words, do >> we need >> to have at least '16n'MB >> of memory at our disposal, before we are sure that there >> will be no >> "outOfMemoryException" ? If >> that is the case, does that also mean that if we are >> working with a >> single main thread only, and >> providing anything less than 16MB of memory to the JVM, >> then the >> exception would always occur ? >> >> -- >> View this message in context: http://www.nabble.com/Query-in- >> Lucene-2.3.0-tp15175141p15175141.html >> Sent from the Lucene - Java Users mailing list archive at Nabble.com. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > -- View this message in context: http://www.nabble.com/Query-in-Lucene-2.3.0-tp15175141p15197794.html Sent from the Lucene - Java Users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
