Re: [Neo] IllegalStateException meaning
I just got this same problem, and was able to defeat by upping heap size. It was very strange - does Transaction#finish do some non-blocking work? Disclaimer: I'm using trunk neo-kernel from 2/10. Thanks, Adam On Mon, Jun 1, 2009 at 1:31 PM, Johan Svensson jo...@neotechnology.com wrote: Great that it seems to work. Please do get back to me if you run into any problems. -Johan On Mon, Jun 1, 2009 at 7:06 PM, Rob Challen rjchal...@gmail.com wrote: Thanks Johan, I increased a whole lot of memory limits and swapped my jre over from IBM Java 64 bit to a sun 32 bit. Because it was happening after 2 hours into a load I didn't get to properly debug but so far the problem seems to have disappeared, and I can't reproduce it now. I think it may be a symptom of another problem somewhere else. Will get back to you if I can find anything more. Rob. On Sun, May 31, 2009 at 3:21 PM, Johan Svensson jo...@neotechnology.comwrote: Hi Rob, I've seen this problem once before but could not reproduce the behavior once I tried. Could you check if tx.success() is called before tx.finish()? Also, could you check in the stacktrace for the source of the nested IllegalStateException. -Johan On Sun, May 31, 2009 at 12:12 AM, Rob Challen rjchal...@gmail.com wrote: I am getting the following exception when I call a Transaction.finish(). It occurs during the load of a large medical terminology dataset, and prior to this error the load progress was found to be slowing. I can;t really figure out what the exception is telling me, or where to start looking to figure out the cause. If anyone can give me any pointers to explain the meaning of the error in neo4j I would be really grateful. The .finish() is in a loop that completes without any problem up to this point. I cannot completely rule out bad input - but finding that in a 1.5M nodes graph would be a challenge in itself. Exception in thread main java.lang.RuntimeException: java.lang.IllegalStateException: Tx status is: STATUS_COMMITING at org.neo4j.api.core.EmbeddedNeo$TransactionImpl.finish(EmbeddedNeo.java:383) ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Announcing Neo4j 1.0
Congratulations! Fantastic product. On Tue, Feb 16, 2010 at 12:55 PM, Dmitri Livotov dmi...@livotov.eu wrote: Our congratulations as well ! Hope we'll also join into the graph users for both production and opensource contribution this graph year ! Best, Dmitri Johan Svensson wrote: Friends, After ten years of development, we are happy to finally announce the release of Neo4j 1.0. It's available here: http://neo4j.org/download http://dist.neo4j.org/neo4j-kernel-1.0-binary.zip http://dist.neo4j.org/neo4j-kernel-1.0-binary.tar.gz For the kernel component this release includes documentation updates together with bug fixes for all known bugs. For more information see: http://dist.neo4j.org/CHANGES.txt Also included in this release is the Neo4j index component: http://components.neo4j.org/neo4j-index/ (version 1.0) You can download the kernel and index (together with some other useful components) bundled together in the apoc package: http://dist.neo4j.org/neo4j-apoc-1.0.zip http://dist.neo4j.org/neo4j-apoc-1.0.tar.gz If you are using maven you can depend on the following (group id=org.neo4j): neo4j-apoc 1.0 or individual components: neo4j-kernel 1.0 neo4j-index 1.0 Finally, let us just offer a huge thanks to everyone on this list, on twitter and in the broader community. Without the feedback and energy and passion and interest from all of you guys, all the endless nights of staring through java.nio stacktraces would never be worth it. We truly feel that 2010 is the year of the graph. Let's change the world. -- Regards, The Neo4j team ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] Ordering of getRelationships
Hi, I was wondering if there are any guarantees about the order of relationships that come out of Node.getRelationships? In my test case (attached) it seems they come out in creation date, ascending order, until you close and re-open the database. After the database is reopened, it seems like the order is reversed. For example: 1. Create a parent 2. Add child c1 3. Add child c2 4. getRelationships on parent returns c1, c2 5. commit and finish 6. getRelationships on parent returns c1, c2 7. Shutdown and reopen 8. getRelationships on parent returns c2, c1 It's been a long week, and I'm convinced I'm messing something up here. Thanks, Adam ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Lucene Index Corruption?
Hi, Thanks for all of the information. Of course the #1 solution to this problem is to ensure all transactions are closed before shutdown is called. I am trying to implement some kind of failsafe in the case that some unforeseen problem/bug causes transactions to remain open. What do you think of an optional parameter to shutdown() to signify Do your best to rollback+finish any open transactions? Or, could you provide a code example of how to close an Iterable of TransactionImpls using TransactionManager suspend and resume? Thank you, Adam On Thu, Dec 10, 2009 at 4:46 AM, Johan Svensson jo...@neotechnology.comwrote: Hi Adam, On Wed, Dec 9, 2009 at 4:37 PM, Adam Rabung adamrab...@gmail.com wrote: 1. This iterator will just prevent duplicates from being returned from the iterator? If there's a condition (bug in my code) that causes shutdown w/ open transactions, will the Lucene indexes continue to double until they're huge? Yes, currently the index may get duplicate entries for each non clean shutdown. 2. Would it be possible to detect this situation, and rebuild the indexes? I guess this is a losing cause if the app is regularly corrupting the data. Yes, it is possible to rebuild the index if everything that needs re-index is reachable and stored in the graph. 3. Could you allow me to close transacations from different threads? Yesterday, I wrote something that tracks tx opens and closes, and could iterate through all open transactions and call finish() on them. But TransactionImpl.finish seems to assume the calling thread is the creating thread, which is not the case here. I would recommend not to do this. It is better to make sure each thread opening up a transaction manages that transaction and closes it properly. You can however change thread for a transaction using the TransactionManager's suspend() and resume() methods. 4. Better yet, expose API for me to force-finish all open transactions? I'd rather have a botched transaction than a corrupt index. I do not think that is a good solution. What if some thread detected that the running transaction has to be rolled back? If you concurrently force commit from another thread then... Something that may be possible to do is to force rollback any running transactions in shutdown and wait for any transaction that has reached the prepared/committing state to complete. 5. Is the only condition for this open transactions + a Lucene shutdown (via shutdown() OR abrubt process termination)? In further testing, it seems I can't reproduce the problem w/ a clean or dirty shutdown if all transactions are closed. Correct, this will only happen after a crash/non clean shutdown marking the logical log as dirty (meaning recovery will be run on next startup replaying the logical log). 6. I assume your iterator fix will make b11? What are the chances the root cause will be fixed in b11? Do you have a tentative release date for b11? We are planing to release in mid december/before christmas. A real fix for this problem will not make it into that release so for now I would suggest to properly close the transactions in the thread that opens them. Regards, -Johan ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] Lucene Index Corruption?
Hi, I've recently run into problems with indexes becoming corrupt after unclean shutdowns. Basically: 1. Transaction 1 writes some data 2. Transaction 2 reads some data, and is left open 3. The database is shut down, with warnings about an open transaction 4. The database is opened. Recovery executes, but it appears the Lucence indexes are doubled - that is, where we used to have key = (value1), we now have key = (value1, value1). I've attached a JUnit test case that hopefully reproduces this for you. I'm on Java 5, Mac OS 10.5, neo-1.0-b10.jar, and index-util-0.8.jar Obviously, the first step on my end is to make sure any open transactions are closed before attempting a shutdown. However, I'm able to pretty reliably reproduce this problem in a much scarier way - just killing a running Neo process via the Eclipse Console view red square process stop button. Amazingly, Eclipse doesn't properly shut down processes properly when this button is used, so I can't count on shutdown hooks: https://bugs.eclipse.org/bugs/show_bug.cgi?id=38016 What expectations should I have for corruption when a database + indexes are .shutDown() with open transactions? What expectations should I have for corruption when a database + indexes are terminated abruptly (Eclipse Console, power outage, etc)? Beyond proper transaction management, and ensuring shutDown() is called, is there anything I should be doing to help protect this data? Thanks, Adam ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] New features in trunk
Hi, I was wondering where lifecycle events fall on the roadmap? I'm thinking along the lines of Hibernate Interceptors: https://www.hibernate.org/hib_docs/v3/api/org/hibernate/Interceptor.html Thanks, Adam On Thu, Sep 3, 2009 at 11:38 AM, Rob Challenrjchal...@gmail.com wrote: I moved over to the b10 snapshot and everything seems fine so far. Recovery time was near instantaneous on a graph with no changes on it, but that had been shut down with a transaction open, which previously would have take 5 minutes to rebuild. I tried to use an EmbeddedReadOnlyNeo service within a servlet container, I updated to index-utils 0.8-SNAPSHOT and figured out that I had to use a LuceneReadOnlyIndexService, and it all works, replacing my immediate need for the RMI RemoteNeo. Neo was never the performance bottleneck in my app so difficult to assess any difference at the moment, but it certainly simplifies my code. Cheers, Rob. On Wed, Sep 2, 2009 at 2:16 PM, Johan Svensson jo...@neotechnology.comwrote: Hello, Now that the 1.0-b9 release has been out for a while we have had some time to work on some important new features that we want to be part of the final 1.0 release. The 1.0-b10-SNAPSHOT (trunk) now includes: o New algorithm to rebuild id generators during recovery after a crash. Cuts recovery time on large stores to a fraction of the time it used to take. o Read-only mode. You can now startup multiple read-only NeoServices to the same store (using EmbbededReadOnlyNeo) and have a single writer to that store. o Improved depth first traversal speed on nodes with many relationships by loading relationships on demand. This will also reduce the stress on GC when traversing over a node with many relationships since much fewer objects will be created. If you have some suitable project please help us test and try out these new features. Regards, -Johan -- Johan Svensson [jo...@neotechnology.com] Chief Technology Officer, Neo Technology www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] More complex comparisons for IndexService
I'm starting to suspect I'm using IndexService incorrectly :) I had been assuming I had one IndexService instance per neo instance. In the future, I was hoping to query that instance in many ways: indexService.getNodes(lastName, smith); //default is equals indexService.getNodes(lastName, ComparisonType.EQUALS, smith); indexService.getNodes(lastName, ComparisonType.FUZZY_MATCH, smith); indexService.getNodes(modifiedTime, ComparisonType.GREATER_THAN, 2000); With just key/value passed in the IndexService/LuceneFullText - it's unclear to me how to expose that comparison through Neo API, unless we have an IndexService instance/comparison: fuzzyMatchIndexService.getNodes(lastName, smith); greaterThanIndexService.getNodes(modifiedTime, 2000); With both approaches, I don't see how you can avoid implementing these same comparisons in-memory so that you can ensure you consider tx deleted/added/modified nodes in the search. Now let me strongly restate I think I must be using IndexService incorrectly :) On Sun, Jul 5, 2009 at 6:45 PM, Mattias Perssonmatt...@neotechnology.com wrote: 2009/7/1 Adam Rabung adamrab...@gmail.com: Hi, thanks for the quick update. I'd also need some kind of comparator (greaterThan, contains, etc) passed in to that method as well, in order to form the proper Lucene Query. The other trick is considering nodes which have been modified/added/deleted in the current Lucene transaction. Regarding the comparator, could you f.ex. use the lucene query syntax http://lucene.apache.org/java/2_4_1/queryparsersyntax.html and have the formQuery method parse that query (with the lucene QueryParser) to accomplish that? The modified/added/deleted nodes are working automatically in the LuceneFulltext* implementation so no worries there. I think exposing LuceneIndexService's xaDs to subclasses, as well txIndexed and txDeleted in LuceneTransaction would allow third parties to create their own LuceneIndexServices. OTOH, this seems like it might be worth offering in the core class. Yep, it'd be great to have it be more customizable-friendly. I'm working on the CLA stuff - thanks again everybody! By your answer here I can't really see if you decided to have a look at and try out the LuceneFulltext* classes instead of your own... what have you tried so far? I'd be really happy to hear from you if you're using it and would like to invite you to improve it furthermore if necessary! On Wed, Jul 1, 2009 at 3:47 AM, Mattias Perssonmatt...@neotechnology.com wrote: Great initiative, btw have you looked at the LuceneFulltextIndexService in the same package? It might be able to do what you're asking for. By @Overriding the formQuery method you can construct whatever queries you like based on key and search value (I just altered the signature to also take the key in addition to value for greater flexibility). Please have a look at that first and see if that helps you! Also do an 'svn up' on the index-util component since I committed it just now :) 2009/6/30 Peter Neubauer neubauer.pe...@gmail.com: Adam, we would love to get your contribution into the codebase, just contact us regarding the CLA etc and we will get you through it! /peter GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - New Energy for Data - The Graph Database. http://www.ops4j.org - New Energy for OSS Communities - Open Participation Software. http://www.oredev.org - Where Good Geeks Grok. On Tue, Jun 30, 2009 at 10:13 PM, Anders Nawrothand...@neotechnology.com wrote: Hi and welcome! I can't answer your questions, but regarding the formatting we use an Eclipse code formatter which you can download from this page: http://wiki.neo4j.org/content/Coding_Standard This page could be of some interest as well: http://wiki.neo4j.org/content/Code_Contributor%27s_Guide /anders Adam Rabung skrev: Hi, I was wondering if anyone's interested in more complex queries against Neo's Lucene (and others) indexes, like so: long twoHoursAgo = System.currentTimeMillis() - (2 * 60 * 60 * 1000); indexService.getNodes(new Comparison(modifiedTime, ComparisonType.GREATER_THAN, twoHoursAgo); I hacked this into my local IndexService/LuceneIndexService/LuceneTransaction, and it seems to work. For less than/greater than at least, the Lucene code itself is very straightforward (use a different Query class). A little trickier when considering the added/updated/deleted nodes in the current tx, but not that bad. There's _plenty_ of problems with my implementation - I applied it only to LuceneIndexService, type conversions, not too confident w/ some of the Neo-specific changes I made - but I was very encouraged by how easy it was to get a draft version running. Is there any chance at all this kind of thing could make a near-term
Re: [Neo] More complex comparisons for IndexService
Hi, thanks for the quick update. I'd also need some kind of comparator (greaterThan, contains, etc) passed in to that method as well, in order to form the proper Lucene Query. The other trick is considering nodes which have been modified/added/deleted in the current Lucene transaction. I think exposing LuceneIndexService's xaDs to subclasses, as well txIndexed and txDeleted in LuceneTransaction would allow third parties to create their own LuceneIndexServices. OTOH, this seems like it might be worth offering in the core class. I'm working on the CLA stuff - thanks again everybody! On Wed, Jul 1, 2009 at 3:47 AM, Mattias Perssonmatt...@neotechnology.com wrote: Great initiative, btw have you looked at the LuceneFulltextIndexService in the same package? It might be able to do what you're asking for. By @Overriding the formQuery method you can construct whatever queries you like based on key and search value (I just altered the signature to also take the key in addition to value for greater flexibility). Please have a look at that first and see if that helps you! Also do an 'svn up' on the index-util component since I committed it just now :) 2009/6/30 Peter Neubauer neubauer.pe...@gmail.com: Adam, we would love to get your contribution into the codebase, just contact us regarding the CLA etc and we will get you through it! /peter GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - New Energy for Data - The Graph Database. http://www.ops4j.org - New Energy for OSS Communities - Open Participation Software. http://www.oredev.org - Where Good Geeks Grok. On Tue, Jun 30, 2009 at 10:13 PM, Anders Nawrothand...@neotechnology.com wrote: Hi and welcome! I can't answer your questions, but regarding the formatting we use an Eclipse code formatter which you can download from this page: http://wiki.neo4j.org/content/Coding_Standard This page could be of some interest as well: http://wiki.neo4j.org/content/Code_Contributor%27s_Guide /anders Adam Rabung skrev: Hi, I was wondering if anyone's interested in more complex queries against Neo's Lucene (and others) indexes, like so: long twoHoursAgo = System.currentTimeMillis() - (2 * 60 * 60 * 1000); indexService.getNodes(new Comparison(modifiedTime, ComparisonType.GREATER_THAN, twoHoursAgo); I hacked this into my local IndexService/LuceneIndexService/LuceneTransaction, and it seems to work. For less than/greater than at least, the Lucene code itself is very straightforward (use a different Query class). A little trickier when considering the added/updated/deleted nodes in the current tx, but not that bad. There's _plenty_ of problems with my implementation - I applied it only to LuceneIndexService, type conversions, not too confident w/ some of the Neo-specific changes I made - but I was very encouraged by how easy it was to get a draft version running. Is there any chance at all this kind of thing could make a near-term Neo4j release? If not, would it be possible to expose internals of some the classes so that third parties could easily add custom searches? Proof of concept patch attached. Not really a patch - just 3 files - LuceneIndexService, LuceneTransaction and a new class called Comparison. Eclipse formatting ruined my ability to quickly make you a nice patch. Thanks for a great product! ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Anders Nawroth [and...@neotechnology.com] GTalk, Skype: anders.nawroth Phone: +46 737 894 163 http://twitter.com/nawroth http://blog.nawroth.com/ ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] Uses of NotFoundException
Hello, First - I am new to Neo, but very interested. After years of solving problems w/ a relational model, neo is a breath of fresh air. So far, the API has been a breeze. I recently ran into a problem where traversals were simply stopping sooner than I expected. Being a neophyte, I suspected I had chosen the wrong StopEvaluator, Directions, etc for my traversal. Even after opening up the traversal parameters completely, I was still getting only a small section of the graph. I came to realize the true problem was my custom StopEvaluator: node.getProperty(label); In my case, not all nodes have a label property. This call was causing a NotFoundException, which is caught and swallowed by AbstractTraverser.traverseToNextNode. From the comments, it looks like NotFoundException is legitimately possible if multiple threads are performing a traversal. The consequence is any problem accessing Node properties inside of a StopEvaluator results in a silent traversal failure - it looks like the nodes just aren't there. Since NotFoundException has such a distinct and unrelated meaning for AbstractTraverser, maybe a PropertyNotFoundException could be used for indicating a problem accessing Node properties? Thanks, Adam ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
