Hi,
during stress tests, I got a "Too many open files" message (see detailed
stack trace below). Server is a Debian Linux 2.6.26 x86_64 w/ 24 GB RAM,
Core i7 which handles the load with ease.
cat /proc/sys/fs/file-max
2399285
The parameter fs.file-max was already set very high, but I'll increase
it further and see if it will happen again.
fs.file-max = 6815744
fs.aio-max-nr = 1048576
Are there recommendations of linux kernel parameters? Or/and would you
recommend other things to circumvent that, f.e. using multiple indexes
instead of one?
And it seems to me that finishing a Neo4j transaction will always
trigger a LuceneTransaction.doCommit, even if there was no update on
index (I try to avoid for the log nodes). Can this be switched off?
Thanks and greetings
Axel
[1] Stack trace
java.lang.RuntimeException: java.io.FileNotFoundException:
/opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too
many open files)
at
org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:272)
at
org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:318)
at
org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446)
at
org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64)
at
org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:516)
at
org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:621)
at
org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584)
at
org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104)
at
org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85)
at
org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37)
at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048)
at org.structr.core.log.LogService.flushQueue(LogService.java:101)
at org.structr.core.log.LogService.run(LogService.java:73)
Caused by: java.io.FileNotFoundException:
/opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too
many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180)
at
org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74)
at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:86)
at
org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66)
at
org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144)
at
org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479)
at
org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777)
at
org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1932)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1906)
at
org.neo4j.index.impl.lucene.LuceneTransaction.applyDocuments(LuceneTransaction.java:299)
at
org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:260)
... 12 more
javax.transaction.xa.XAException: Unknown
xid[GlobalId[NEOKERNL|5131995998687892543|61442], BranchId[ 52 49 52 49
52 49 ]]
at
org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.rollback(XaResourceManager.java:470)
at
org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.rollback(XaResourceHelpImpl.java:111)
at
org.neo4j.kernel.impl.transaction.TransactionImpl.doRollback(TransactionImpl.java:533)
at
org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:648)
at
org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584)
at
org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104)
at
org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85)
at
org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37)
at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048)
at org.structr.core.log.LogService.flushQueue(LogService.java:101)
at org.structr.core.log.LogService.run(LogService.java:73)
Mar 30, 2011 6:31:53 PM org.neo4j.kernel.impl.transaction.TxManager commit
SEVERE: Unable to rollback transaction. Some resources may be commited
others not. Neo4j kernel should be SHUTDOWN for resource maintance and
transaction recovery ---->
java.lang.RuntimeException: java.io.FileNotFoundException:
/opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too
many open files)
at
org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:272)
at
org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:318)
at
org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446)
at
org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64)
at
org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:516)
at
org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:621)
at
org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584)
at
org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104)
at
org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85)
at
org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37)
at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048)
at org.structr.core.log.LogService.flushQueue(LogService.java:101)
at org.structr.core.log.LogService.run(LogService.java:73)
Caused by: java.io.FileNotFoundException:
/opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too
many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180)
at
org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74)
at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:86)
at
org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66)
at
org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144)
at
org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479)
at
org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777)
at
org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1932)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1906)
at
org.neo4j.index.impl.lucene.LuceneTransaction.applyDocuments(LuceneTransaction.java:299)
at
org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:260)
... 12 more
org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction
at
org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:104)
at
org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37)
at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048)
at org.structr.core.log.LogService.flushQueue(LogService.java:101)
at org.structr.core.log.LogService.run(LogService.java:73)
Caused by: javax.transaction.HeuristicMixedException: Unable to rollback
---> error code in commit: -1 ---> error code for rollback: 0
at
org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:662)
at
org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584)
at
org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104)
at
org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85)
... 4 more
_______________________________________________
Neo4j mailing list
[email protected]
https://lists.neo4j.org/mailman/listinfo/user
