2010/1/27 Symeon (Akis) Papadopoulos <[email protected]>: > >>>> See more information at >>>> http://components.neo4j.org/neo4j-index/apidocs/index.html >>>> >>>> Are you using the "index" component btw (artifactId: neo4j-index)? >>>> >>>> >>> Yes. >>> >>> Is it possible that the problem is caused due to the fact that we >>> perform several writes in both the graph and the index and we don't >>> commit and that we read in between the transaction commits? >>> >> No that shouldn't cause any problems. This looks like a bug then. >> Could you give us more information about your code, some snippets >> perhaps or a test case which can trigger it again? >> > In fact I've tried increasing the ulimit -n parameter of the linux > machine where I was running the experiment (it was 1024 and I set it to > 10000). It appears to work so far (still running the test). What I > understand is that there are many open files from lucene due to the fact > that no optimize() is called on time. Since optimize() is not exposed > from the LuceneIndexService, should I expect that it is handled internally? Great, lucene handles merging in the background automatically if the files are too sharded. So this error shouldn't occur unless there are some corner case where some IndexReader/IndexWriter isn't closed properly... so that's why I'm suspecting a bug here :)
Great that you found a work-around, but I suspect it could happen even with a higher ulimit though. > > The project I'm running is rather large, but at some point I will try to > prepare a script in order to replicate the error. That would be great indeed! > > > >>> >>>> 2010/1/26 Symeon (Akis) Papadopoulos <[email protected]>: >>>> >>>> >>>>> Hi all >>>>> >>>>> While populating a Neo graph, I got the attached exception >>>>> (org.apache.lucene.index.MergePolicy$MergeException. See file for >>>>> details: I >>>>> replaced a local path with [some-local-path]). >>>>> My setup is like this: I want to benchmark Neo4j for some operations, >>>>> amongst which is graph loading. >>>>> So, I try to load graphs of various sizes to Neo4j. Up to size 1M edges >>>>> the >>>>> graphs are loaded without any problem, but then this exception is thrown. >>>>> I >>>>> suspect this has to do with the transaction management in my program, >>>>> which >>>>> is handled by a BatchTxManager (a class written by me, thus highly likely >>>>> to >>>>> be the source of trouble). Does the exception ring any bells? What could I >>>>> try out in order to identify the problem? >>>>> >>>>> Best regards, >>>>> Akis >>>>> >>>>> >>>>> >>>>> Exception in thread "Lucene Merge Thread #0" >>>>> org.apache.lucene.index.MergePolicy$MergeException: >>>>> java.io.FileNotFoundException: /[some-local-path]/_4k.cfs (Too many open >>>>> files) >>>>> at >>>>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351) >>>>> at >>>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315) >>>>> Caused by: java.io.FileNotFoundException: /[some-local-path]/_4k.cfs (Too >>>>> many open files) >>>>> at java.io.RandomAccessFile.open(Native Method) >>>>> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) >>>>> at >>>>> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:78) >>>>> at >>>>> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:108) >>>>> at >>>>> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:94) >>>>> at >>>>> org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:70) >>>>> at >>>>> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70) >>>>> at >>>>> org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:111) >>>>> at >>>>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638) >>>>> at >>>>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608) >>>>> at >>>>> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686) >>>>> at >>>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979) >>>>> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614) >>>>> at >>>>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235) >>>>> at >>>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291) >>>>> Exception in thread "main" >>>>> org.neo4j.kernel.impl.transaction.TransactionFailureException: Unable to >>>>> commit transaction >>>>> at >>>>> org.neo4j.kernel.EmbeddedGraphDbImpl$TransactionImpl.finish(EmbeddedGraphDbImpl.java:331) >>>>> at >>>>> org.neo4j.util.SimpleBatchTxManager.beginReadOperation(SimpleBatchTxManager.java:94) >>>>> at >>>>> graph.implementation.neo.STSGraphNeoImpl.getUserNeo(STSGraphNeoImpl.java:463) >>>>> at >>>>> graph.implementation.neo.STSGraphNeoImpl.increaseUserTagFreq(STSGraphNeoImpl.java:824) >>>>> at >>>>> graph.benchmark.LoadGraphBenchmark.runRealBenchmark(LoadGraphBenchmark.java:433) >>>>> at >>>>> graph.benchmark.LoadGraphBenchmark.main(LoadGraphBenchmark.java:489) >>>>> Caused by: java.lang.RuntimeException: Unable to close lucene writer >>>>> org.apache.lucene.index.indexwri...@32a4eb93 >>>>> at >>>>> org.neo4j.index.lucene.LuceneDataSource.removeWriter(LuceneDataSource.java:413) >>>>> at >>>>> org.neo4j.index.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:197) >>>>> at >>>>> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:316) >>>>> at >>>>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:399) >>>>> at >>>>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) >>>>> at >>>>> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:514) >>>>> at >>>>> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:573) >>>>> at >>>>> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:543) >>>>> at >>>>> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:102) >>>>> at >>>>> org.neo4j.kernel.EmbeddedGraphDbImpl$TransactionImpl.finish(EmbeddedGraphDbImpl.java:316) >>>>> ... 5 more >>>>> Caused by: java.io.IOException: directory '/[some-local-path]/resource_id' >>>>> exists and is a directory, but cannot be listed: list() returned null >>>>> at >>>>> org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516) >>>>> at >>>>> org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531) >>>>> at >>>>> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307) >>>>> at >>>>> org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300) >>>>> at >>>>> org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192) >>>>> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183) >>>>> at >>>>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2190) >>>>> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2153) >>>>> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2117) >>>>> at >>>>> org.neo4j.index.lucene.LuceneDataSource.removeWriter(LuceneDataSource.java:409) >>>>> ... 14 more >>>>> >>>>> >>>>> _______________________________________________ >>>>> Neo mailing list >>>>> [email protected] >>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> _______________________________________________ >>> Neo mailing list >>> [email protected] >>> https://lists.neo4j.org/mailman/listinfo/user >>> >>> >> >> >> >> > > _______________________________________________ > Neo mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > -- Mattias Persson, [[email protected]] Neo Technology, www.neotechnology.com _______________________________________________ Neo mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
