Re: [Neo4j] Close with running transaction II
Vaccaro, can you reliably reproduce this in a unit test? that would be very helpful in doing something about it! Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Wed, Sep 7, 2011 at 8:34 PM, Vaccaro, Kristen M kvacc...@mitre.org wrote: Hi, I'm experiencing the same problem that is mentioned in a thread from a while back on this list (http://lists.neo4j.org/pipermail/user/2011-March/007746.html). This was a case where someone had a thread running when they try to close the database and keeping having a dirty log with INFO: Close invoked with 1 running transaction(s). As far as I know I close every transaction I open. The response to the other poster was that there is no way to force all transactions to end before shutting down -- which leaves me with the question: what are the risks of leaving the code as is? And what are the best practices to avoid this problem? (again, as far as I can see all transactions are ended). Thanks, Kristen -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Nuo Yan Sent: Wednesday, September 07, 2011 2:12 PM To: Neo4j user discussions Subject: Re: [Neo4j] Server couldn't start after java.lang.OutOfMemoryError According to this: http://neo4j.org/forums/#nabble-td3248377, I upgraded to 1.4.1 and the problem goes away. I guess there must be a bug in the 1.4.0 REST API that leaks classloader memory or so, and GC didn't happen as expected in PermGen. Anyways, it looks great after upgrading to 1.4.1. Memory monitoring graph shows GC happens as desired in PermGen. Thank you! On Sep 6, 2011, at 11:30 PM, Nuo Yan wrote: Hey Adriano and everyone, I finally got a chance to monitor/profile the neo4j process with VisualVM when I run my set of tests. It seems that after running the set of tests once, ~ 50M of PermGen was used. However, subsequent test runs simply multiples that, which worries me a little. After the fourth run of the tests, ~ 200M of PermGen space was used. My app accesses to neo4j completely using the REST interface. During the test run, new nodes and relationships are being frequently created. There are batch requests that performs multiple actions at once (number of actions is reasonable, really small in this case, usually a few, or tenth). There are also gremlin script executions through REST (send the gremlin query over to the server through REST API). Most of the gremlin queries are traversals and returns the vertices or edges match the constraints. Total number of nodes created during a test run is only hundreds. There are more relationships, but at most thousands. That's basically all of the things that are done against neo4j server during a test run. Given this, do you think the PermGen usage is reasonable? Or do you think something is very wrong? I worried that setting a larger PermGen would only postpone but not resolve the OutOfMemory problem. Let me know if any other info is needed. Thanks much, Nuo On Aug 29, 2011, at 6:09 AM, Adriano Henrique de Almeida wrote: Hi Nuo, any results here? Cheers 2011/8/26 Nuo Yan yan@gmail.com Hey Adriano, Thanks very much for your reply. I will try with MaxPermSize. It's on my local machine, we are evaluating Neo4j and do not have it on production yet. I'm accessing the server through the REST APIs, the app is separate from Neo. I will see if I can do some profiling on Neo when I run my tests. Thanks, Nuo On Fri, Aug 26, 2011 at 3:53 PM, Adriano Henrique de Almeida adrianoalmei...@gmail.com wrote: Hey Nuo, wrapper.java.maxmemory and Xmx won't help, since it is a PermGenSpace problem, and they only deal with HeapSpace. Did you try with: -XX:*MaxPermSize **?* Anyway, something very wrong is happening, since you have few nodes. It is happening only on production, or development also? If it is happening on development, try to isolate the situation where you're stucking the permgen. Run you app with VisualVM, for instance, or any other profiler. Aren't you dealing with classloaders on your app? Cheers Adriano Almeida 2011/8/26 Jim Webber j...@neotechnology.com Hi Nuo, if you're using Neo4j server then you can edit the value in conf/neo-wrapper.conf wrapper.java.maxmemory=1024 Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Adriano Almeida Caelum | Ensino e Inovação www.caelum.com.br ___ Neo4j mailing
Re: [Neo4j] Close with running transaction
I can't reproduce this. I tried: GraphDatabaseService db = new EmbeddedGraphDatabase( mydb ); IndexNode index = db.index().forNodes( myindex ); Thread.sleep( 5000 ); db.shutdown(); and no such message or recovery is displayed. Are you doing something else in your simple example? 2011/3/28 Axel Morgner a...@morgner.de Good evening, lately I'm experiencing the following log message when closing the database, even if no write transaction was there between startup and shutdown. In my test, I even wait 5s before closing the database and another 5s before stopping the thread - same result. INFO: Close invoked with 1 running transaction(s) This always leads to an unclean shutdown, causing a long recovery period at next startup. Not sure, but I think it shows up since I upgraded to 1.3-M03, or using the new index framework (can't say exactly). I do shutdown the database with graphDb.shutdown() since I can think. Am I missing something? How can I avoid this behaviour? Greetings Axel ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Close with running transaction
Seems that another thread is still running when I close the database, so nothing wrong with Neo4j, but with my code. :-( Can I somehow force all running transactions (even when started from other threads) to finish? Am 29.03.2011 10:23, schrieb Mattias Persson: I can't reproduce this. I tried: GraphDatabaseService db = new EmbeddedGraphDatabase( mydb ); IndexNode index = db.index().forNodes( myindex ); Thread.sleep( 5000 ); db.shutdown(); and no such message or recovery is displayed. Are you doing something else in your simple example? 2011/3/28 Axel Morgnera...@morgner.de Good evening, lately I'm experiencing the following log message when closing the database, even if no write transaction was there between startup and shutdown. In my test, I even wait 5s before closing the database and another 5s before stopping the thread - same result. INFO: Close invoked with 1 running transaction(s) This always leads to an unclean shutdown, causing a long recovery period at next startup. Not sure, but I think it shows up since I upgraded to 1.3-M03, or using the new index framework (can't say exactly). I do shutdown the database with graphDb.shutdown() since I can think. Am I missing something? How can I avoid this behaviour? Greetings Axel ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Close with running transaction
Not currently, but I know there has been some discussions some time ago about maybe adding such functionality (maybe automatically rolling back ongoing transactions nicely in shutdown). I'm not fully aware of implications that would have, but something like that could perhaps be good to have. 2011/3/29 Axel Morgner a...@morgner.de Seems that another thread is still running when I close the database, so nothing wrong with Neo4j, but with my code. :-( Can I somehow force all running transactions (even when started from other threads) to finish? Am 29.03.2011 10:23, schrieb Mattias Persson: I can't reproduce this. I tried: GraphDatabaseService db = new EmbeddedGraphDatabase( mydb ); IndexNode index = db.index().forNodes( myindex ); Thread.sleep( 5000 ); db.shutdown(); and no such message or recovery is displayed. Are you doing something else in your simple example? 2011/3/28 Axel Morgnera...@morgner.de Good evening, lately I'm experiencing the following log message when closing the database, even if no write transaction was there between startup and shutdown. In my test, I even wait 5s before closing the database and another 5s before stopping the thread - same result. INFO: Close invoked with 1 running transaction(s) This always leads to an unclean shutdown, causing a long recovery period at next startup. Not sure, but I think it shows up since I upgraded to 1.3-M03, or using the new index framework (can't say exactly). I do shutdown the database with graphDb.shutdown() since I can think. Am I missing something? How can I avoid this behaviour? Greetings Axel ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Close with running transaction
Good evening, lately I'm experiencing the following log message when closing the database, even if no write transaction was there between startup and shutdown. In my test, I even wait 5s before closing the database and another 5s before stopping the thread - same result. INFO: Close invoked with 1 running transaction(s) This always leads to an unclean shutdown, causing a long recovery period at next startup. Not sure, but I think it shows up since I upgraded to 1.3-M03, or using the new index framework (can't say exactly). I do shutdown the database with graphDb.shutdown() since I can think. Am I missing something? How can I avoid this behaviour? Greetings Axel ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
