Re: [Neo4j] Lucene index: Too many open files
2011/4/1 Mattias Persson
>
>
> 2011/4/1 Axel Morgner
>
>> Using 1.3-M03 here. I also try to reproduce it with an isolated test.
>> Will keep you informed about the result.
>>
>> Much appreciated Alex
>
Sorry, Axel :)
>
>
>> Axel
>>
>> On 01.04.2011 11:01, Mattias Persson wrote:
>> > 2011/4/1 Michael Hunger
>> >
>> >> What about the line in the if that doesn't execute the block, if
>> >> index-hits .hasNext() returns false?
>> >>
>> >> Does this close the underlying structures ?
>> >>
>> if (hits != null&& hits.iterator().hasNext()) {
>> > As of some milestone back it closes the iterator if there are no results
>> in
>> > it. Which version of neo4j are you running Alex?
>> >
>> >> and here 2 iterators are opened? perhaps this might be a problem, one
>> for
>> >> the if, the second for the for-each loop
>> >>
>> > Actually IndexHits is cheating where #iterator() {return this;} so
>> there's
>> > only one iterator.
>> >
>> > As a note I can say that I'm currently running a test with 30 threads,
>> > randomly modifying/querying an index where some queries are looped
>> through
>> > and some just a bit and then closed manually. Also getSingle() is used
>> > randomly. And as far as I can see, after 10 minutes of such a test
>> running
>> > the number of open files is kept it a steady low. I just cannot
>> reproduce
>> > this on the latest milestone.
>>
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>
>
>
> --
> Mattias Persson, [matt...@neotechnology.com]
> Hacker, Neo Technology
> www.neotechnology.com
>
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Lucene index: Too many open files
2011/4/1 Axel Morgner
> Using 1.3-M03 here. I also try to reproduce it with an isolated test.
> Will keep you informed about the result.
>
> Much appreciated Alex
> Axel
>
> On 01.04.2011 11:01, Mattias Persson wrote:
> > 2011/4/1 Michael Hunger
> >
> >> What about the line in the if that doesn't execute the block, if
> >> index-hits .hasNext() returns false?
> >>
> >> Does this close the underlying structures ?
> >>
> if (hits != null&& hits.iterator().hasNext()) {
> > As of some milestone back it closes the iterator if there are no results
> in
> > it. Which version of neo4j are you running Alex?
> >
> >> and here 2 iterators are opened? perhaps this might be a problem, one
> for
> >> the if, the second for the for-each loop
> >>
> > Actually IndexHits is cheating where #iterator() {return this;} so
> there's
> > only one iterator.
> >
> > As a note I can say that I'm currently running a test with 30 threads,
> > randomly modifying/querying an index where some queries are looped
> through
> > and some just a bit and then closed manually. Also getSingle() is used
> > randomly. And as far as I can see, after 10 minutes of such a test
> running
> > the number of open files is kept it a steady low. I just cannot reproduce
> > this on the latest milestone.
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Lucene index: Too many open files
Using 1.3-M03 here. I also try to reproduce it with an isolated test.
Will keep you informed about the result.
Axel
On 01.04.2011 11:01, Mattias Persson wrote:
> 2011/4/1 Michael Hunger
>
>> What about the line in the if that doesn't execute the block, if
>> index-hits .hasNext() returns false?
>>
>> Does this close the underlying structures ?
>>
if (hits != null&& hits.iterator().hasNext()) {
> As of some milestone back it closes the iterator if there are no results in
> it. Which version of neo4j are you running Alex?
>
>> and here 2 iterators are opened? perhaps this might be a problem, one for
>> the if, the second for the for-each loop
>>
> Actually IndexHits is cheating where #iterator() {return this;} so there's
> only one iterator.
>
> As a note I can say that I'm currently running a test with 30 threads,
> randomly modifying/querying an index where some queries are looped through
> and some just a bit and then closed manually. Also getSingle() is used
> randomly. And as far as I can see, after 10 minutes of such a test running
> the number of open files is kept it a steady low. I just cannot reproduce
> this on the latest milestone.
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Lucene index: Too many open files
2011/4/1 Michael Hunger
> What about the line in the if that doesn't execute the block, if
> index-hits .hasNext() returns false?
>
> Does this close the underlying structures ?
>
> >>if (hits != null && hits.iterator().hasNext()) {
>
As of some milestone back it closes the iterator if there are no results in
it. Which version of neo4j are you running Alex?
>
> and here 2 iterators are opened? perhaps this might be a problem, one for
> the if, the second for the for-each loop
>
Actually IndexHits is cheating where #iterator() {return this;} so there's
only one iterator.
As a note I can say that I'm currently running a test with 30 threads,
randomly modifying/querying an index where some queries are looped through
and some just a bit and then closed manually. Also getSingle() is used
randomly. And as far as I can see, after 10 minutes of such a test running
the number of open files is kept it a steady low. I just cannot reproduce
this on the latest milestone.
>
> Cheers
>
> Michael
>
> Am 01.04.2011 um 09:34 schrieb Mattias Persson:
>
> > It looks correct to me (even without the call to close()) since the
> entire
> > IndexHits is looped through. I'd very interested in a snippet of code
> > reproducing this. Is it reproducable just be through lots of queries to
> an
> > index, or are you doing lots of modifications as well? Many threads?
> >
> > 2011/4/1 Axel Morgner
> >
> >> On 31.03.2011 09:29, Mattias Persson wrote:
> >>> There are two things here:
> >>>
> >>> The error you're getting is because you probably forget to close some
> >>> IndexHits somewhere, IndexHits that you haven't iterated all the way
> >>> through... could you check your code for such mistakes?
> >> Hmm ... I checked the code. Iteration always goes through all hits I
> think:
> >>
> >> IndexHits hits = index.query(new QueryContext(textualQueryString));
> >>
> >> List result = nodeFactory.createNodes(hits);
> >>
> >> The createNodes-Method looks like that:
> >>
> >>public List createNodes(final Iterable hits) {
> >>List nodes = new ArrayList();
> >>if (hits != null && hits.iterator().hasNext()) {
> >>for (Node node : hits) {
> >>AbstractNode n = createNode(node);
> >>nodes.add(n);
> >>}
> >>}
> >>return nodes;
> >>}
> >>
> >> Maybe the IndexHits are not closed if thread gets interrupted?
> >>
> >> Now I added a hits.close() after the createNodes line. More stress tests
> >> will follow and reveal all flaws in my code ... ;-)
> >>
> >>> The other one that doCommit always seems to be invoked I just tried and
> >>> cannot reproduce, so may be something on your end? could you provide a
> >> code
> >>> snippet to reproduce this?
> >> Debugged it, but was not able to isolate or reproduce it. Will keep an
> >> eye on it. I'm quite sure it has to do with my asynchronous logging
> >> service, but it seems that it is kinda hard to track down ...
> >>
> >> Thanks for your help, and best regards
> >>
> >> Axel
> >>
> >> ___
> >> Neo4j mailing list
> >> User@lists.neo4j.org
> >> https://lists.neo4j.org/mailman/listinfo/user
> >>
> >
> >
> >
> > --
> > Mattias Persson, [matt...@neotechnology.com]
> > Hacker, Neo Technology
> > www.neotechnology.com
> > ___
> > Neo4j mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Lucene index: Too many open files
What about the line in the if that doesn't execute the block, if index-hits
.hasNext() returns false?
Does this close the underlying structures ?
>>if (hits != null && hits.iterator().hasNext()) {
and here 2 iterators are opened? perhaps this might be a problem, one for the
if, the second for the for-each loop
Cheers
Michael
Am 01.04.2011 um 09:34 schrieb Mattias Persson:
> It looks correct to me (even without the call to close()) since the entire
> IndexHits is looped through. I'd very interested in a snippet of code
> reproducing this. Is it reproducable just be through lots of queries to an
> index, or are you doing lots of modifications as well? Many threads?
>
> 2011/4/1 Axel Morgner
>
>> On 31.03.2011 09:29, Mattias Persson wrote:
>>> There are two things here:
>>>
>>> The error you're getting is because you probably forget to close some
>>> IndexHits somewhere, IndexHits that you haven't iterated all the way
>>> through... could you check your code for such mistakes?
>> Hmm ... I checked the code. Iteration always goes through all hits I think:
>>
>> IndexHits hits = index.query(new QueryContext(textualQueryString));
>>
>> List result = nodeFactory.createNodes(hits);
>>
>> The createNodes-Method looks like that:
>>
>>public List createNodes(final Iterable hits) {
>>List nodes = new ArrayList();
>>if (hits != null && hits.iterator().hasNext()) {
>>for (Node node : hits) {
>>AbstractNode n = createNode(node);
>>nodes.add(n);
>>}
>>}
>>return nodes;
>>}
>>
>> Maybe the IndexHits are not closed if thread gets interrupted?
>>
>> Now I added a hits.close() after the createNodes line. More stress tests
>> will follow and reveal all flaws in my code ... ;-)
>>
>>> The other one that doCommit always seems to be invoked I just tried and
>>> cannot reproduce, so may be something on your end? could you provide a
>> code
>>> snippet to reproduce this?
>> Debugged it, but was not able to isolate or reproduce it. Will keep an
>> eye on it. I'm quite sure it has to do with my asynchronous logging
>> service, but it seems that it is kinda hard to track down ...
>>
>> Thanks for your help, and best regards
>>
>> Axel
>>
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>
>
>
> --
> Mattias Persson, [matt...@neotechnology.com]
> Hacker, Neo Technology
> www.neotechnology.com
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Lucene index: Too many open files
It looks correct to me (even without the call to close()) since the entire
IndexHits is looped through. I'd very interested in a snippet of code
reproducing this. Is it reproducable just be through lots of queries to an
index, or are you doing lots of modifications as well? Many threads?
2011/4/1 Axel Morgner
> On 31.03.2011 09:29, Mattias Persson wrote:
> > There are two things here:
> >
> > The error you're getting is because you probably forget to close some
> > IndexHits somewhere, IndexHits that you haven't iterated all the way
> > through... could you check your code for such mistakes?
> Hmm ... I checked the code. Iteration always goes through all hits I think:
>
> IndexHits hits = index.query(new QueryContext(textualQueryString));
>
> List result = nodeFactory.createNodes(hits);
>
> The createNodes-Method looks like that:
>
> public List createNodes(final Iterable hits) {
> List nodes = new ArrayList();
> if (hits != null && hits.iterator().hasNext()) {
> for (Node node : hits) {
> AbstractNode n = createNode(node);
> nodes.add(n);
> }
> }
> return nodes;
> }
>
> Maybe the IndexHits are not closed if thread gets interrupted?
>
> Now I added a hits.close() after the createNodes line. More stress tests
> will follow and reveal all flaws in my code ... ;-)
>
> > The other one that doCommit always seems to be invoked I just tried and
> > cannot reproduce, so may be something on your end? could you provide a
> code
> > snippet to reproduce this?
> Debugged it, but was not able to isolate or reproduce it. Will keep an
> eye on it. I'm quite sure it has to do with my asynchronous logging
> service, but it seems that it is kinda hard to track down ...
>
> Thanks for your help, and best regards
>
> Axel
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Lucene index: Too many open files
On 31.03.2011 09:29, Mattias Persson wrote:
> There are two things here:
>
> The error you're getting is because you probably forget to close some
> IndexHits somewhere, IndexHits that you haven't iterated all the way
> through... could you check your code for such mistakes?
Hmm ... I checked the code. Iteration always goes through all hits I think:
IndexHits hits = index.query(new QueryContext(textualQueryString));
List result = nodeFactory.createNodes(hits);
The createNodes-Method looks like that:
public List createNodes(final Iterable hits) {
List nodes = new ArrayList();
if (hits != null && hits.iterator().hasNext()) {
for (Node node : hits) {
AbstractNode n = createNode(node);
nodes.add(n);
}
}
return nodes;
}
Maybe the IndexHits are not closed if thread gets interrupted?
Now I added a hits.close() after the createNodes line. More stress tests
will follow and reveal all flaws in my code ... ;-)
> The other one that doCommit always seems to be invoked I just tried and
> cannot reproduce, so may be something on your end? could you provide a code
> snippet to reproduce this?
Debugged it, but was not able to isolate or reproduce it. Will keep an
eye on it. I'm quite sure it has to do with my asynchronous logging
service, but it seems that it is kinda hard to track down ...
Thanks for your help, and best regards
Axel
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Lucene index: Too many open files
I get that too, but sometimes. I got the feeling that it relates to the number of @Transactional annotations that get nested. I might be completely wrong. But I got rid of it by revising my annotations. Alfredas On Wed, Mar 30, 2011 at 11:40 PM, rick.bullo...@burningskysoftware.com wrote: > Sounds like a bug to me. > > - Reply message - > From: "Axel Morgner" > Date: Wed, Mar 30, 2011 1:23 pm > Subject: [Neo4j] Lucene index: Too many open files > To: "Neo4j user discussions" > > Hi, > > during stress tests, I got a "Too many open files" message (see detailed > stack trace below). Server is a Debian Linux 2.6.26 x86_64 w/ 24 GB RAM, > Core i7 which handles the load with ease. > > cat /proc/sys/fs/file-max > 2399285 > > The parameter fs.file-max was already set very high, but I'll increase > it further and see if it will happen again. > > fs.file-max = 6815744 > fs.aio-max-nr = 1048576 > > Are there recommendations of linux kernel parameters? Or/and would you > recommend other things to circumvent that, f.e. using multiple indexes > instead of one? > > And it seems to me that finishing a Neo4j transaction will always > trigger a LuceneTransaction.doCommit, even if there was no update on > index (I try to avoid for the log nodes). Can this be switched off? > > Thanks and greetings > > Axel > > > [1] Stack trace > > java.lang.RuntimeException: java.io.FileNotFoundException: > /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too > many open files) > at > org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:272) > at > org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:318) > at > org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446) > at > org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) > at > org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:516) > at > org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:621) > at > org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) > at > org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) > at > org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) > at > org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) > at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) > at org.structr.core.log.LogService.flushQueue(LogService.java:101) > at org.structr.core.log.LogService.run(LogService.java:73) > Caused by: java.io.FileNotFoundException: > /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too > many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.(SimpleFSDirectory.java:180) > at > org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) > at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:86) > at > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Lucene index: Too many open files
Thanks for the info re: calling close on the indexhits object. We will wrap that logic in a try/finally wherever we use index queries. - Reply message - From: "Mattias Persson" Date: Thu, Mar 31, 2011 3:29 am Subject: [Neo4j] Lucene index: Too many open files To: "Neo4j user discussions" There are two things here: The error you're getting is because you probably forget to close some IndexHits somewhere, IndexHits that you haven't iterated all the way through... could you check your code for such mistakes? The other one that doCommit always seems to be invoked I just tried and cannot reproduce, so may be something on your end? could you provide a code snippet to reproduce this? Best, Mattias 2011/3/30 Axel Morgner > Hi, > > during stress tests, I got a "Too many open files" message (see detailed > stack trace below). Server is a Debian Linux 2.6.26 x86_64 w/ 24 GB RAM, > Core i7 which handles the load with ease. > > cat /proc/sys/fs/file-max > 2399285 > > The parameter fs.file-max was already set very high, but I'll increase > it further and see if it will happen again. > > fs.file-max = 6815744 > fs.aio-max-nr = 1048576 > > Are there recommendations of linux kernel parameters? Or/and would you > recommend other things to circumvent that, f.e. using multiple indexes > instead of one? > > And it seems to me that finishing a Neo4j transaction will always > trigger a LuceneTransaction.doCommit, even if there was no update on > index (I try to avoid for the log nodes). Can this be switched off? > > Thanks and greetings > > Axel > > > [1] Stack trace > > java.lang.RuntimeException: java.io.FileNotFoundException: > /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too > many open files) > at > > org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:272) > at > > org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:318) > at > > org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446) > at > > org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) > at > > org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:516) > at > org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:621) > at > org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) > at > > org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) > at > org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) > at > > org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) > at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) > at org.structr.core.log.LogService.flushQueue(LogService.java:101) > at org.structr.core.log.LogService.run(LogService.java:73) > Caused by: java.io.FileNotFoundException: > /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too > many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > at > > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.(SimpleFSDirectory.java:180) > at > org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) > at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:86) > at > > org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66) > at > > org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144) > at > > org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193) > at > > org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460) > at > > org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479) > at > > org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099) > at > > org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777) > at > > org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752) > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1932) > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1906) > at > > org.neo4j.index.impl.lucene.LuceneTransaction.applyDocuments(LuceneTransaction.java:299) > at > > org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:260) > ... 12
Re: [Neo4j] Lucene index: Too many open files
Is there a good way of ensuring the IndexHits, close? Especially if I give out the IndexHits (perhaps in an IteratorWrapper) to client code? Thanks Michael Am 31.03.2011 um 09:29 schrieb Mattias Persson: > There are two things here: > > The error you're getting is because you probably forget to close some > IndexHits somewhere, IndexHits that you haven't iterated all the way > through... could you check your code for such mistakes? > > The other one that doCommit always seems to be invoked I just tried and > cannot reproduce, so may be something on your end? could you provide a code > snippet to reproduce this? > > > Best, > Mattias > > 2011/3/30 Axel Morgner > >> Hi, >> >> during stress tests, I got a "Too many open files" message (see detailed >> stack trace below). Server is a Debian Linux 2.6.26 x86_64 w/ 24 GB RAM, >> Core i7 which handles the load with ease. >> >> cat /proc/sys/fs/file-max >> 2399285 >> >> The parameter fs.file-max was already set very high, but I'll increase >> it further and see if it will happen again. >> >> fs.file-max = 6815744 >> fs.aio-max-nr = 1048576 >> >> Are there recommendations of linux kernel parameters? Or/and would you >> recommend other things to circumvent that, f.e. using multiple indexes >> instead of one? >> >> And it seems to me that finishing a Neo4j transaction will always >> trigger a LuceneTransaction.doCommit, even if there was no update on >> index (I try to avoid for the log nodes). Can this be switched off? >> >> Thanks and greetings >> >> Axel >> >> >> [1] Stack trace >> >> java.lang.RuntimeException: java.io.FileNotFoundException: >> /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too >> many open files) >>at >> >> org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:272) >>at >> >> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:318) >>at >> >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446) >>at >> >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) >>at >> >> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:516) >>at >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:621) >>at >> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) >>at >> >> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) >>at >> org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) >>at >> >> org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) >>at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) >>at org.structr.core.log.LogService.flushQueue(LogService.java:101) >>at org.structr.core.log.LogService.run(LogService.java:73) >> Caused by: java.io.FileNotFoundException: >> /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too >> many open files) >>at java.io.RandomAccessFile.open(Native Method) >>at java.io.RandomAccessFile.(RandomAccessFile.java:233) >>at >> >> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.(SimpleFSDirectory.java:180) >>at >> org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) >>at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:86) >>at >> >> org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66) >>at >> >> org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144) >>at >> >> org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193) >>at >> >> org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460) >>at >> >> org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479) >>at >> >> org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099) >>at >> >> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777) >>at >> >> org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752) >>at >> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1932) >>at >> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1906) >>at >> >> org.neo4j.index.impl.lucene.LuceneTransaction.applyDocuments(LuceneTransaction.java:299) >>at >> >> org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:260) >>... 12 more >> javax.transaction.xa.XAException: Unknown >> xid[GlobalId[NEOKERNL|5131995998687892543|61442], BranchId[ 52 49 52 49 >> 52 49 ]] >>at >> >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.rollback(XaResourceManager.java:470) >>at >> >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.rollback(XaResourceHelpImpl.java:111) >
Re: [Neo4j] Lucene index: Too many open files
There are two things here: The error you're getting is because you probably forget to close some IndexHits somewhere, IndexHits that you haven't iterated all the way through... could you check your code for such mistakes? The other one that doCommit always seems to be invoked I just tried and cannot reproduce, so may be something on your end? could you provide a code snippet to reproduce this? Best, Mattias 2011/3/30 Axel Morgner > Hi, > > during stress tests, I got a "Too many open files" message (see detailed > stack trace below). Server is a Debian Linux 2.6.26 x86_64 w/ 24 GB RAM, > Core i7 which handles the load with ease. > > cat /proc/sys/fs/file-max > 2399285 > > The parameter fs.file-max was already set very high, but I'll increase > it further and see if it will happen again. > > fs.file-max = 6815744 > fs.aio-max-nr = 1048576 > > Are there recommendations of linux kernel parameters? Or/and would you > recommend other things to circumvent that, f.e. using multiple indexes > instead of one? > > And it seems to me that finishing a Neo4j transaction will always > trigger a LuceneTransaction.doCommit, even if there was no update on > index (I try to avoid for the log nodes). Can this be switched off? > > Thanks and greetings > > Axel > > > [1] Stack trace > > java.lang.RuntimeException: java.io.FileNotFoundException: > /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too > many open files) > at > > org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:272) > at > > org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:318) > at > > org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446) > at > > org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) > at > > org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:516) > at > org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:621) > at > org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) > at > > org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) > at > org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) > at > > org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) > at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) > at org.structr.core.log.LogService.flushQueue(LogService.java:101) > at org.structr.core.log.LogService.run(LogService.java:73) > Caused by: java.io.FileNotFoundException: > /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too > many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > at > > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.(SimpleFSDirectory.java:180) > at > org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) > at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:86) > at > > org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66) > at > > org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144) > at > > org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193) > at > > org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460) > at > > org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479) > at > > org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099) > at > > org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777) > at > > org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752) > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1932) > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1906) > at > > org.neo4j.index.impl.lucene.LuceneTransaction.applyDocuments(LuceneTransaction.java:299) > at > > org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:260) > ... 12 more > javax.transaction.xa.XAException: Unknown > xid[GlobalId[NEOKERNL|5131995998687892543|61442], BranchId[ 52 49 52 49 > 52 49 ]] > at > > org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.rollback(XaResourceManager.java:470) > at > > org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.rollback(XaResourceHelpImpl.java:111) > at > > org.neo4j.kernel.impl.transaction.TransactionImpl.doRollback(TransactionImpl.java:533) > at > org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:648) > at > org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) > at > > org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104
Re: [Neo4j] Lucene index: Too many open files
Sounds like a bug to me. - Reply message - From: "Axel Morgner" Date: Wed, Mar 30, 2011 1:23 pm Subject: [Neo4j] Lucene index: Too many open files To: "Neo4j user discussions" Hi, during stress tests, I got a "Too many open files" message (see detailed stack trace below). Server is a Debian Linux 2.6.26 x86_64 w/ 24 GB RAM, Core i7 which handles the load with ease. cat /proc/sys/fs/file-max 2399285 The parameter fs.file-max was already set very high, but I'll increase it further and see if it will happen again. fs.file-max = 6815744 fs.aio-max-nr = 1048576 Are there recommendations of linux kernel parameters? Or/and would you recommend other things to circumvent that, f.e. using multiple indexes instead of one? And it seems to me that finishing a Neo4j transaction will always trigger a LuceneTransaction.doCommit, even if there was no update on index (I try to avoid for the log nodes). Can this be switched off? Thanks and greetings Axel [1] Stack trace java.lang.RuntimeException: java.io.FileNotFoundException: /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too many open files) at org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:272) at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:318) at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446) at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:516) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:621) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) at org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) at org.structr.core.log.LogService.flushQueue(LogService.java:101) at org.structr.core.log.LogService.run(LogService.java:73) Caused by: java.io.FileNotFoundException: /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(RandomAccessFile.java:233) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.(SimpleFSDirectory.java:180) at org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:86) at ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Lucene index: Too many open files
Hi, during stress tests, I got a "Too many open files" message (see detailed stack trace below). Server is a Debian Linux 2.6.26 x86_64 w/ 24 GB RAM, Core i7 which handles the load with ease. cat /proc/sys/fs/file-max 2399285 The parameter fs.file-max was already set very high, but I'll increase it further and see if it will happen again. fs.file-max = 6815744 fs.aio-max-nr = 1048576 Are there recommendations of linux kernel parameters? Or/and would you recommend other things to circumvent that, f.e. using multiple indexes instead of one? And it seems to me that finishing a Neo4j transaction will always trigger a LuceneTransaction.doCommit, even if there was no update on index (I try to avoid for the log nodes). Can this be switched off? Thanks and greetings Axel [1] Stack trace java.lang.RuntimeException: java.io.FileNotFoundException: /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too many open files) at org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:272) at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:318) at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446) at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:516) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:621) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) at org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) at org.structr.core.log.LogService.flushQueue(LogService.java:101) at org.structr.core.log.LogService.run(LogService.java:73) Caused by: java.io.FileNotFoundException: /opt/structr/t5s/db/index/lucene/node/fulltextAllNodes/_ngl.fdx (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(RandomAccessFile.java:233) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.(SimpleFSDirectory.java:180) at org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:86) at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66) at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144) at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193) at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460) at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479) at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777) at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1932) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1906) at org.neo4j.index.impl.lucene.LuceneTransaction.applyDocuments(LuceneTransaction.java:299) at org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:260) ... 12 more javax.transaction.xa.XAException: Unknown xid[GlobalId[NEOKERNL|5131995998687892543|61442], BranchId[ 52 49 52 49 52 49 ]] at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.rollback(XaResourceManager.java:470) at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.rollback(XaResourceHelpImpl.java:111) at org.neo4j.kernel.impl.transaction.TransactionImpl.doRollback(TransactionImpl.java:533) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:648) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:584) at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:104) at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85) at org.structr.core.node.TransactionCommand.execute(TransactionCommand.java:37) at org.structr.core.entity.AbstractNode.commit(AbstractNode.java:1048) at org.structr.core.log.LogService.flushQueue(LogService.java:101) at org.structr.core.log.LogService.run(LogService.java:73) Mar 30, 2011 6:31:53 PM org.neo4j.kernel.impl.transaction.TxManager commit SEVERE: Unable to rollback transaction. Some resources may be commited others not. Neo4j kernel should be SHUTDOWN for resource maintance and trans
