I restarted the solr and stopped all searches. After that, the commit() was normal (2 secs) and it's been working for 3h without problems (indexing and a few searches too)... I haven't done any optimize yet, mainly because I had no deletes on the index and the performance is ok, so no need to optimize I think..
I had tried this procedure a few times in the morning and the commit always hanged so.. I have no explanation for it start working suddenly.. I'm making a commit every 2m (because I need the results updated on searches), so propably when I have more searches at the same time the commit will hang again right? Sorry for the newbie questions and thanks for your help and explanation Erik. BR, Frederico -----Original Message----- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: terça-feira, 12 de Janeiro de 2010 15:15 To: solr-user@lucene.apache.org Subject: Re: Problem comitting on 40GB index Rebooting the machine certainly closes the searchers, but depending upon how you shut it down there may be stale files.... After reboot (but before you start SOLR), how much space is on your disk? If it's 40G, you have no stale files.... Yes, IR is IndexReader, which is a searcher. I'll have to leave it to others if you don't have stale files hanging around, although if you're optimizing while searchers are running, you'll use up to 3X the index size... Otherwise I'll have to leave it to others for additional insights.... Best Erick On Tue, Jan 12, 2010 at 9:22 AM, Frederico Azeiteiro < frederico.azeite...@cision.com> wrote: > Hi Erik, > > I'm a newbie to solr... By IR, you mean searcher? Is there a place where I > can check the open searchers? And rebooting the machine shouldn't closed > that searchers? > > Thanks, > > -----Original Message----- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: terça-feira, 12 de Janeiro de 2010 13:54 > To: solr-user@lucene.apache.org > Subject: Re: Problem comitting on 40GB index > > There are several possibilities: > > 1> you have some process holding open your indexes, probably > other searchers. You *probably* are OK just committing > new changes if there is exactly *one* searcher keeping > your index open. If you have some process whereby > you periodically open a new search but you fail to close > the old one, then you'll use up an extra 40G for every > version of your index held open by your processes. That's > confusing... I'm saying that if you open any number of IRs, > you'll have 40G consumed. Then if you add > some more documents and open *another* IR, you'll have > another 40G consumed. They'll stay around until you close > your readers. > > 2> If you optimize, there can be up to 3X the index size being > consumed if you also have a previous reader opened. > > So I suspect that sometime recently you've opened another > IR..... > > HTH > Erick > > > > On Tue, Jan 12, 2010 at 8:03 AM, Frederico Azeiteiro < > frederico.azeite...@cision.com> wrote: > > > Hi all, > > > > I started working with solr about 1 month ago, and everything was > > running well both indexing as searching documents. > > > > I have a 40GB index with about 10 000 000 documents available. I index > > 3k docs for each 10m and commit after each insert. > > > > Since yesterday, I can't commit no articles to index. I manage to search > > ok, and index documents without commiting. But when I start the commit > > is takes a long time and eats all of the available disk space > > left(60GB). The commit eventually stops with full disk and I have to > > restart SOLR and get the 60GB returned to system. > > > > Before this, the commit was taking a few seconds to complete. > > > > Can someone help to debug the problem? Where should I start? Should I > > try to copy the index to other machine with more free space and try to > > commit? Should I try an optimize? > > > > Log for the last commit I tried: > > > > INFO: start > > commit(optimize=false,waitFlush=false,waitSearcher=true,expungeDeletes=f > > alse) > > (Then, after a long time...) > > Exception in thread "Lucene Merge Thread #0" > > org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: > > No space left on device > > at > > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Co > > ncurrentMergeScheduler.java:351) > > at > > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr > > entMergeScheduler.java:315) > > Caused by: java.io.IOException: No space left on device > > > > I'm using Ubuntu 9.04 and Solr 1.4.0. > > > > Thanks in advance, > > > > Frederico > > >
