For better performance, you should not send 100 threads to IndexWriter, but rather a number of threads in proportion to how many CPUs the machine has. E.g. if your CPU has 8 cores then use at most 12 (=8 * 1.5) indexing threads. It's fine to have 100 client threads sending documents, but drop these documents into a queue and have the 12 indexing threads pull from there.
If you will have more than 8 threads in IndexWriter at once, then you should call IndexWriterConfig.setMaxThreadStates to increase the default (8). Also, your benchmark does not allow for JVM warming, so you are measuring e.g. hotspot compilation time. It's better to make a long running test and then measure the indexing throughput at steady state, once the JVM is warmed. In Lucene's nightly benchmark (https://people.apache.org/~mikemccand/lucenebench/indexing.html ), we index ~ 1 KB sized docs around 44.9 Kdocs/sec, or ~ 145 GB/hour, but that's a very different test than what you are running (e.g., it uses .addDocument not the more costly .updateDocument)... Mike McCandless http://blog.mikemccandless.com On Thu, Feb 13, 2014 at 5:08 AM, sree <mail2sreed...@gmail.com> wrote: > Thanks for your reply. We are using 100 threads and each indexes 100 > documents. Now we created a standalone project which uses lucene to index > 100 documents for 100 theads concurrently and we can see that each thread > uses an average of more than 1 sec. > > lucene-group.zip > <http://lucene.472066.n3.nabble.com/file/n4117133/lucene-group.zip> > <http://lucene.472066.n3.nabble.com/file/n4117133/lucene_profiler.jpg> > > > Please find attached source files, excel sheet and profiler image for more > information. > > thanks > Sreedeep > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Suggestions-Required-110-Concurrency-users-indexing-on-Lucene-dont-finish-in-200-ms-tp4116625p4117133.html > Sent from the Lucene - Java Users mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
