RE: Lucene going non-responsive under heavy load

2009-12-23 Thread Uwe Schindler
iraj Haider [mailto:si...@jobdiva.com] > Sent: Wednesday, December 23, 2009 12:00 AM > To: java-user@lucene.apache.org > Subject: Re: Lucene going non-responsive under heavy load > > Hi Uwe, > Following are the info: > Java VM: "1.6.0_05" > Java Options: -server -Xms512m

Re: Lucene going non-responsive under heavy load

2009-12-23 Thread Michael McCandless
Are you using IndexReader.reopen to open a new reader, from an existing one? That's much more efficient than opening a new reader. I think a good next step is to run with IndexWriter.setInfoStream on, and run your JRE with verbose GC, to see more details. Mike On Wed, Dec 23, 2009 at 9:12 AM, S

Re: Lucene going non-responsive under heavy load

2009-12-23 Thread Siraj Haider
We have dual cpu intel xeon machines running "Red Hat Enterprise Linux ES release 3 (Taroon Update 6)". We have 4GB memory on these machines with 2GB allocated to tomcat. After modifying the index we open a new one, warm it up, make it live and then close the old one. -siraj Michael McCandle

Re: Lucene going non-responsive under heavy load

2009-12-22 Thread Michael McCandless
A merge shouldn't make the machine completely non-responsive, just, slower to run searches / index documents. What kind of machine / IO system is this? You can set maxMergeDocs to limit how large the merge is allowed to be. But, be careful, since if you set this too small you'll wind up with way

Re: Lucene going non-responsive under heavy load

2009-12-22 Thread Siraj Haider
ider [mailto:si...@jobdiva.com] Sent: Tuesday, December 22, 2009 11:19 PM To: java-user@lucene.apache.org Subject: Lucene going non-responsive under heavy load Hello guys, We have a dilemma on a few of our lucene machines. We have a tomcat running our servlets for searching and indexing on each of the

Re: Lucene going non-responsive under heavy load

2009-12-22 Thread Siraj Haider
Hi Mike, You are right, sometimes there is an implicit merge running when the machine goes non-responsive. How can we avoid running those merges during the day and how can we minimize the effect it will have on searches? -siraj Michael McCandless wrote: Is it possible a large merge is runni

RE: Lucene going non-responsive under heavy load

2009-12-22 Thread Uwe Schindler
er-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Siraj Haider [mailto:si...@jobdiva.com] > Sent: Tuesday, December 22, 2009 11:19 PM > To: java-user@lucene.apache.org > Subject: Lucene going non-responsive under heavy load &

Re: Lucene going non-responsive under heavy load

2009-12-22 Thread Michael McCandless
Is it possible a large merge is running? You can turn on IndexWriter.setInfoStream to see more details about what IW is doing, including merging. Mike On Tue, Dec 22, 2009 at 5:19 PM, Siraj Haider wrote: > Hello guys, > We have a dilemma on a few of our lucene machines.  We have a tomcat runnin

Lucene going non-responsive under heavy load

2009-12-22 Thread Siraj Haider
Hello guys, We have a dilemma on a few of our lucene machines. We have a tomcat running our servlets for searching and indexing on each of these machines. Its a live index where documents are being added to index while online searches are also being served at the same time. Indexing happens