Stefan there is also the stackoverflow post: http://stackoverflow.com/questions/21238902/recover-corrupt-neo4j-database-after-server-crash-graphdb-notfoundexception
On Wed, Jan 22, 2014 at 3:30 PM, Stefan Armbruster <[email protected]> wrote: > To understand the issue let me recap: > * you're running 1.9.2 Community edition? > * without any concurrent queries you get the NotFoundException for > node 20924 constantly in a read-only cypher query? > > Further questions: > * What is the cypher statement you're executing? > * Does the NotFoundException persist after a restart of Neo4j? > * Please provide the full stacktrace in a more readable format (maybe > via pastbin.com) > * provide the snippet of data/graph.db/messages.log containing the > latest startup cycle, see https://gist.github.com/sarmbruster/8001713 > > In your description you've mentioned that you tried to reindex nodes > and "at the same point ... it crashed again, with the same error". > What was exactly the error happing while reindexing? Please send > stacktrace here as well. > > side note: if you depend on timely answers and need commercial grade > support let me know and I'll bring you in touch with a Neo Technology > sales rep. > > Cheers, > Stefan > > > 2014/1/21 <[email protected]>: >> Hi, >> >> I have a neo4j database with several million nodes and about as many >> relationships. While running a program that was adding data to it, the JVM >> seems to have crashed. When I later tried to query the database using an >> index, it opened normally and retrieved some of the nodes, but at some point >> returned the following error: >> >> Exception in thread "main" org.neo4j.graphdb.NotFoundException: Node[20924] >> not found. This can be because someone else deleted this entity while we >> were trying to read properties from it, or because of concurrent >> modification of other properties on this entity. The problem should be >> temporary. at >> org.neo4j.kernel.impl.core.Primitive.ensureFullProperties(Primitive.java:601) >> at >> org.neo4j.kernel.impl.core.Primitive.ensureFullProperties(Primitive.java:579) >> at org.neo4j.kernel.impl.core.Primitive.hasProperty(Primitive.java:309) at >> org.neo4j.kernel.impl.core.NodeImpl.hasProperty(NodeImpl.java:53) at >> org.neo4j.kernel.impl.core.NodeProxy.hasProperty(NodeProxy.java:160) at >> org.neo4j.cypher.internal.spi.gdsimpl.GDSBackedQueryContext$$anon$1.hasProperty(GDSBackedQueryContext.scala:66) >> at >> org.neo4j.cypher.internal.spi.gdsimpl.GDSBackedQueryContext$$anon$1.hasProperty(GDSBackedQueryContext.scala:48) >> at org.neo4j.cypher.internal.commands.Has.isMatch(Predicate.scala:203) at >> org.neo4j.cypher.internal.pipes.FilterPipe$$anonfun$internalCreateResults$1.apply(FilterPipe.scala:30) >> at >> org.neo4j.cypher.internal.pipes.FilterPipe$$anonfun$internalCreateResults$1.apply(FilterPipe.scala:30) >> at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:390) at >> scala.collection.Iterator$class.foreach(Iterator.scala:727) at >> scala.collection.AbstractIterator.foreach(Iterator.scala:1156) at >> org.neo4j.cypher.internal.pipes.EagerAggregationPipe.internalCreateResults(EagerAggregationPipe.scala:76) >> at >> org.neo4j.cypher.internal.pipes.PipeWithSource.createResults(Pipe.scala:69) >> at >> org.neo4j.cypher.internal.pipes.PipeWithSource.createResults(Pipe.scala:66) >> at >> org.neo4j.cypher.internal.executionplan.ExecutionPlanImpl.org$neo4j$cypher$internal$executionplan$ExecutionPlanImpl$$prepareStateAndResult(ExecutionPlanImpl.scala:164) >> at >> org.neo4j.cypher.internal.executionplan.ExecutionPlanImpl$$anonfun$getLazyReadonlyQuery$1.apply(ExecutionPlanImpl.scala:139) >> at >> org.neo4j.cypher.internal.executionplan.ExecutionPlanImpl$$anonfun$getLazyReadonlyQuery$1.apply(ExecutionPlanImpl.scala:138) >> at >> org.neo4j.cypher.internal.executionplan.ExecutionPlanImpl.execute(ExecutionPlanImpl.scala:38) >> at org.neo4j.cypher.ExecutionEngine.execute(ExecutionEngine.scala:72) at >> org.neo4j.cypher.ExecutionEngine.execute(ExecutionEngine.scala:67) at >> org.neo4j.cypher.javacompat.ExecutionEngine.execute(ExecutionEngine.java:66) >> at querygraph.BasicStatsQueries.main(BasicStatsQueries.java:54) Caused by: >> org.neo4j.kernel.impl.nioneo.store.InvalidRecordException: >> PropertyRecord[11853043] not in use at >> org.neo4j.kernel.impl.nioneo.store.PropertyStore.getRecord(PropertyStore.java:453) >> at >> org.neo4j.kernel.impl.nioneo.store.PropertyStore.getLightRecord(PropertyStore.java:306) >> at >> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getPropertyRecordChain(ReadTransaction.java:185) >> at >> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.loadProperties(ReadTransaction.java:215) >> at >> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.nodeLoadProperties(ReadTransaction.java:239) >> at >> org.neo4j.kernel.impl.persistence.PersistenceManager.loadNodeProperties(PersistenceManager.java:111) >> at >> org.neo4j.kernel.impl.core.NodeManager.loadProperties(NodeManager.java:833) >> at org.neo4j.kernel.impl.core.NodeImpl.loadProperties(NodeImpl.java:143) at >> org.neo4j.kernel.impl.core.Primitive.ensureFullProperties(Primitive.java:596) >> ... 23 more >> >> There was only one thread (at least, that I started) running the query, and >> it was all reading, not writing. And though the exception claims that it's >> temporary, this happens every time I try to query this index. I therefore >> assume it has to do with the bad shutdown. I have had database corruptions >> before from forced shutdowns before I implemented code to prevent that, but >> neo4j was always able to recover the database, though it took a while. It >> seems that this is much worse. >> >> When I looped through the index manually and added a try-catch and a >> counter, it began returning the error for every node in the index after the >> one listed above, which was about 6.6K nodes in. Does that mean that all >> these nodes are non-existent, or corrupted? That would mean a significant >> (huge) loss of data, since there should be about a million nodes in the >> index. What can I do to recover the database? >> >> As per some advice on SO, I deleted the index and tried to reindex the >> nodes, but at the same point (about 6.6K nodes in), it crashed again, with >> the same error. This database is badly needed and it's time-sensitive - is >> there anything I can do that I haven't tried yet to recover it? >> >> I am using 1.9.2 and would love to upgrade to use labels and etc., but I >> need this database right now for some time-critical work and don't have time >> to change anything major right now. >> >> Thanks a lot in advance for any help. >> >> bsg >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
