Is there a good way of ensuring the IndexHits, close? Especially if I give out the IndexHits (perhaps in an IteratorWrapper) to client code?
Thanks Michael Am 31.03.2011 um 09:29 schrieb Mattias Persson: > There are two things here: > > The error you're getting is because you probably forget to close some > IndexHits somewhere, IndexHits that you haven't iterated all the way > through... could you check your code for such mistakes? > > The other one that doCommit always seems to be invoked I just tried and > cannot reproduce, so may be something on your end? could you provide a code > snippet to reproduce this? > > > Best, > Mattias > > 2011/3/30 Axel Morgner <[email protected]> > >> Hi, >> >> during stress tests, I got a "Too many open files" message (see detailed >> stack trace below). Server is a Debian Linux 2.6.26 x86_64 w/ 24 GB RAM, >> Core i7 which handles the load with ease. >> >> cat /proc/sys/fs/file-max >> 2399285 >> >> The parameter fs.file-max was already set very high, but I'll increase >> it further and see if it will happen again. >> >> fs.file-max = 6815744 >> fs.aio-max-nr = 1048576 >> >> Are there recommendations of linux kernel parameters? Or/and would you >> recommend other things to circumvent that, f.e. using multiple indexes >> instead of one? >> >> And it seems to me that finishing a Neo4j transaction will always >> trigger a LuceneTransaction.doCommit, even if there was no update on >> index (I try to avoid for the log nodes). Can this be switched off? >> >> Thanks and greetings >> >> Axel >> >> >> [1] Stack trace >> >> java.lang.RuntimeException: java.io.FileNotFoundException: >> /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too >> many open files) >> at >> >> org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:272) >> at >> >> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:318) >> at >> >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446) >> at >> >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) >> at >> >> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:516) >> at >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:621) >> at >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) >> at >> >> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) >> at >> org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) >> at >> >> org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) >> at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) >> at org.structr.core.log.LogService.flushQueue(LogService.java:101) >> at org.structr.core.log.LogService.run(LogService.java:73) >> Caused by: java.io.FileNotFoundException: >> /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too >> many open files) >> at java.io.RandomAccessFile.open(Native Method) >> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) >> at >> >> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180) >> at >> org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) >> at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:86) >> at >> >> org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66) >> at >> >> org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144) >> at >> >> org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193) >> at >> >> org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460) >> at >> >> org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479) >> at >> >> org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099) >> at >> >> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777) >> at >> >> org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752) >> at >> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1932) >> at >> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1906) >> at >> >> org.neo4j.index.impl.lucene.LuceneTransaction.applyDocuments(LuceneTransaction.java:299) >> at >> >> org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:260) >> ... 12 more >> javax.transaction.xa.XAException: Unknown >> xid[GlobalId[NEOKERNL|5131995998687892543|61442], BranchId[ 52 49 52 49 >> 52 49 ]] >> at >> >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.rollback(XaResourceManager.java:470) >> at >> >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.rollback(XaResourceHelpImpl.java:111) >> at >> >> org.neo4j.kernel.impl.transaction.TransactionImpl.doRollback(TransactionImpl.java:533) >> at >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:648) >> at >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) >> at >> >> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) >> at >> org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) >> at >> >> org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) >> at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) >> at org.structr.core.log.LogService.flushQueue(LogService.java:101) >> at org.structr.core.log.LogService.run(LogService.java:73) >> Mar 30, 2011 6:31:53 PM org.neo4j.kernel.impl.transaction.TxManager commit >> SEVERE: Unable to rollback transaction. Some resources may be commited >> others not. Neo4j kernel should be SHUTDOWN for resource maintance and >> transaction recovery ----> >> java.lang.RuntimeException: java.io.FileNotFoundException: >> /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too >> many open files) >> at >> >> org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:272) >> at >> >> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:318) >> at >> >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446) >> at >> >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) >> at >> >> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:516) >> at >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:621) >> at >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) >> at >> >> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) >> at >> org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) >> at >> >> org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) >> at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) >> at org.structr.core.log.LogService.flushQueue(LogService.java:101) >> at org.structr.core.log.LogService.run(LogService.java:73) >> Caused by: java.io.FileNotFoundException: >> /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too >> many open files) >> at java.io.RandomAccessFile.open(Native Method) >> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) >> at >> >> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180) >> at >> org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) >> at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:86) >> at >> >> org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66) >> at >> >> org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144) >> at >> >> org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193) >> at >> >> org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460) >> at >> >> org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479) >> at >> >> org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099) >> at >> >> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777) >> at >> >> org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752) >> at >> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1932) >> at >> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1906) >> at >> >> org.neo4j.index.impl.lucene.LuceneTransaction.applyDocuments(LuceneTransaction.java:299) >> at >> >> org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:260) >> ... 12 more >> org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction >> at >> org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:104) >> at >> >> org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) >> at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) >> at org.structr.core.log.LogService.flushQueue(LogService.java:101) >> at org.structr.core.log.LogService.run(LogService.java:73) >> Caused by: javax.transaction.HeuristicMixedException: Unable to rollback >> ---> error code in commit: -1 ---> error code for rollback: 0 >> at >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:662) >> at >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) >> at >> >> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) >> at >> org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) >> ... 4 more >> >> _______________________________________________ >> Neo4j mailing list >> [email protected] >> https://lists.neo4j.org/mailman/listinfo/user >> > > > > -- > Mattias Persson, [[email protected]] > Hacker, Neo Technology > www.neotechnology.com > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
