Karl, this does not seem to be a leaking file handle problem. What you are seeing when you do a ls -1 | wc is you index after the failed merge. A single merge exception does not mean that all concurrently running merges failed. There might have been more than 1024 open files during that time but just a single merge thread hit the limit. It really depends on you IW settings, ram buffer etc. Especially if you have a searcher / reader open you OS will keep the files open while you are not seeing them anymore with ls -1 Those files are kept open until the searcher is closed. Are you indexing with intermediate commits or do you do any kind of batches with fixed size and subsequently warming the searcher?
Can you try to raise the limit of open files? simon On Fri, Sep 10, 2010 at 7:27 PM, <karl.wri...@nokia.com> wrote: > Hi folks, > > I am running into what appears to be a file handle leak in trunk during > indexing. It’s not clear yet what the causative event is, although the > indexing runs for more than an hour before it occurs. The system is Ubuntu, > and has 1024 file handles (per process). This is on a trunk checkout from > about 3 hours ago. > > The exception I start seeing is: > > [java] Exception in thread "Lucene Merge Thread #0" > org.apache.lucene.index.MergePolicy$MergeException: > java.io.FileNotFoundException: > /root/solr-dym/solr-dym/solr_home_v2/nose/data/index/_5l.fdx (Too many open > files) > [java] at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:471) > [java] at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:435) > [java] Caused by: java.io.FileNotFoundException: > /root/solr-dym/solr-dym/solr_home_v2/nose/data/index/_5l.fdx (Too many open > files) > [java] at java.io.RandomAccessFile.open(Native Method) > [java] at > java.io.RandomAccessFile.<init>(RandomAccessFile.java:212) > [java] at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:69) > [java] at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:90) > [java] at > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:91) > [java] at > org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78) > [java] at > org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:104) > [java] at > org.apache.lucene.index.SegmentReader$CoreReaders.openDocStores(SegmentReader.java:243) > [java] at > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:538) > [java] at > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:635) > [java] at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3976) > [java] at > org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3628) > [java] at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:339) > [java] at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:407) > > > Here’s the ulimit –a output: > > r...@duck6:~/solr-dym/solr-dym# ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 20 > file size (blocks, -f) unlimited > pending signals (-i) 16382 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) unlimited > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > The actual number of files in the index at this time is relatively low: > > r...@duck6:~/solr-dym/solr-dym/solr_home_v2/nose/data/index# ls -1 | wc > 179 179 1532 > r...@duck6:~/solr-dym/solr-dym/solr_home_v2/nose/data/index# > > Anyone willing to work with me to narrow down the problem? > > Thanks, > Karl > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
