Hi, we are running Solr 3.2.0 on Jetty for a web application. Since we just went online and are still in beta tests, we don't have very much load on our servers (indeed, they're currently much oversized for the current usage), and our index size on file system ist just 1.1 MB.
We have one dedicated Solr instance for updates, and two replicated read-only servers for requests. The update server gets filled by three different Java web servers, each has a distinct Quartz job for its updates. Every such Quartz job takes all collected updates, sends them via Solrj's addBeans() method, and from time to time, they send an additional commit() after that. Each update job has a CommonHTTPSolrServer instance, which is a Spring controlled singleton. We already had LockObtainFailedExceptions before, raising every few days. Sometimes, we had such an exception before: org.apache.solr.common.SolrException: java.io.IOException: directory '/data/solr/data/index' exists and is a directory, but cannot be listed: list() returned null This looks like if there were no more file handles from the operating system. This is strange, since the only index directory never had more than 100 files, if ever. However, we raised ulimit -n from 1024 to 4096, and reduced mergeFactor from 10 to 5, which firsted helped us with our problem. Until yesterday. Again, we had this: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: SimpleFSLock@solr/main/data/index/write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:84) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1114) at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:83) at org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:101) at org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:174) at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:222) at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61) ..... When we deleted the write.lock file without restarting Solr, several hours later we had 441 same log entries: Jul 18, 2011 7:20:29 AM org.apache.solr.update.SolrIndexWriter finalize SEVERE: SolrIndexWriter was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!! Wow, if there really were 441 open IndexWriters trying to access the index directory, it's no wonder that there will be Lock timeouts sooner or later! However, I have no clue why there are so many IndexWriters opened and never closed. The only accessing Solr instances are pure Java applications using Solrj. Each application only has one SolrServer instance - and even of not, this shouldn't harm, AFAIK. The update job is started every five seconds. The installation is a pure 3.2.0 Solr, without additional jars. And all jars are of the correct revision. The solrconfig.xml is based on the example configuration, with nothing special. We currently don't have any own extensions running. There is absolutely only one jetty instance running on the machine. And I checked the solr.xml, it's only one core defined, and we don't do any additional core administration. I'm using Solr since the beginning of 2010, but never had such a problem. Any help is welcome. Greetings, Kuli