Anything more about the history? Upgrades from previous versions? Since when are you getting these errors?
Only now or also before? Which version is it? Michael On Tue, Aug 30, 2016 at 12:39 PM, Ray Chengr <[email protected]> wrote: > Hi Michael, > Sorry, I copy-pasted wrong text in my last reply two minutes ago. Please > ignore that one. > > We do insertion at rate about 250 relationships&nodes per second(!) on a > single server. The disk usage is as follows: (422GB, it's too big to send) > > [root@sjc graph.db]# du -s * /* this is in Kbytes */ > 4 index > 8 neostore > 4 neostore.counts.db.a > 4 neostore.counts.db.b > 4 neostore.id > 8 neostore.labeltokenstore.db > 4 neostore.labeltokenstore.db.id > 8 neostore.labeltokenstore.db.names > 4 neostore.labeltokenstore.db.names.id > 185148 neostore.nodestore.db > 4 neostore.nodestore.db.id > 8 neostore.nodestore.db.labels > 4 neostore.nodestore.db.labels.id > 89701060 neostore.propertystore.db > 332845360 neostore.propertystore.db.arrays > 4 neostore.propertystore.db.arrays.id > 4 neostore.propertystore.db.id > 8 neostore.propertystore.db.index > 4 neostore.propertystore.db.index.id > 8 neostore.propertystore.db.index.keys > 4 neostore.propertystore.db.index.keys.id > 1336196 neostore.propertystore.db.strings > 4 neostore.propertystore.db.strings.id > 29636 neostore.relationshipgroupstore.db > 4 neostore.relationshipgroupstore.db.id > 16814260 neostore.relationshipstore.db > 88 neostore.relationshipstore.db.id > 8 neostore.relationshiptypestore.db > 4 neostore.relationshiptypestore.db.id > 8 neostore.relationshiptypestore.db.names > 4 neostore.relationshiptypestore.db.names.id > 8 neostore.schemastore.db > 4 neostore.schemastore.db.id > 214076 neostore.transaction.db.3203 > 1302104 schema > 0 store_lock > [root@ddna-sjc graph.db]# du -s . /* also in Kbytes */ > 442428080 . > [root@ddna-sjc graph.db]# du -s -h . > 422G . > > > I started insertion and got these same messages from neo4j.log > continuously, even with the same relationship ID (?) 500676928. > > 2016-08-30 10:19:02.252+0000 ERROR Failed to commit transaction. > Relationship[500676928,used=false,source=-1,target=-1, > type=-1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,secondaryUnitId=-1, > sFirst, tFirst] not in use > org.neo4j.kernel.impl.store.InvalidRecordException: > Relationship[500676928,used=false,source=-1,target=-1, > type=-1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,secondaryUnitId=-1, > sFirst, tFirst] not in use > > at org.neo4j.kernel.impl.store.record.RecordLoad.verify( > RecordLoad.java:63) > at org.neo4j.kernel.impl.store.CommonAbstractStore.verifyAfterReading( > CommonAbstractStore.java:1263) > at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord( > CommonAbstractStore.java:1047) > at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord( > CommonAbstractStore.java:1025) > at org.neo4j.kernel.impl.transaction.state.Loaders$3. > load(Loaders.java:200) > at org.neo4j.kernel.impl.transaction.state.Loaders$3. > load(Loaders.java:190) > at org.neo4j.kernel.impl.transaction.state.RecordChanges.getOrLoad( > RecordChanges.java:72) > at org.neo4j.kernel.impl.transaction.state. > RelationshipCreator.connect(RelationshipCreator.java:203) > at org.neo4j.kernel.impl.transaction.state.RelationshipCreator. > connectRelationshipToDenseNode(RelationshipCreator.java:165) > at org.neo4j.kernel.impl.transaction.state.RelationshipCreator. > connectRelationship(RelationshipCreator.java:143) > at org.neo4j.kernel.impl.transaction.state.RelationshipCreator. > relationshipCreate(RelationshipCreator.java:67) > at org.neo4j.kernel.impl.transaction.state.TransactionRecordState. > relCreate(TransactionRecordState.java:282) > at org.neo4j.kernel.impl.storageengine.impl.recordstorage. > TransactionToRecordStateVisitor.visitCreatedRelationship( > TransactionToRecordStateVisitor.java:96) > at org.neo4j.storageengine.api.txstate.TxStateVisitor$Delegator. > visitCreatedRelationship(TxStateVisitor.java:227) > at org.neo4j.kernel.api.txstate.TransactionCountingStateVisito > r.visitCreatedRelationship(TransactionCountingStateVisitor.java:108) > at org.neo4j.kernel.impl.api.state.TxState$15.visitAddedRelationship( > TxState.java:376) > at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapt > er$1.visit(RelationshipChangeVisitorAdapter.java:125) > at org.neo4j.kernel.impl.api.state.RelationshipStateImpl. > accept(RelationshipStateImpl.java:52) > at org.neo4j.kernel.impl.api.state.TxState.relationshipVisit(TxState. > java:1539) > at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapt > er$1.visit(RelationshipChangeVisitorAdapter.java:115) > at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapt > er.visitAddedRelationship(RelationshipChangeVisitorAdapter.java:66) > at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapt > er.visitAdded(RelationshipChangeVisitorAdapter.java:90) > at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapt > er.visitAdded(RelationshipChangeVisitorAdapter.java:49) > at org.neo4j.kernel.impl.util.diffsets.SuperDiffSets.accept( > SuperDiffSets.java:66) > at org.neo4j.kernel.impl.util.diffsets.RelationshipDiffSets. > accept(RelationshipDiffSets.java:43) > at org.neo4j.kernel.impl.api.state.TxState.accept(TxState.java:265) > at org.neo4j.kernel.impl.storageengine.impl.recordstorage. > RecordStorageEngine.createCommands(RecordStorageEngine.java:312) > at org.neo4j.kernel.impl.api.KernelTransactionImplementation.commit( > KernelTransactionImplementation.java:437) > at org.neo4j.kernel.impl.api.KernelTransactionImplementation.close( > KernelTransactionImplementation.java:386) > at org.neo4j.server.rest.transactional.TransitionalTxManagementKernel > Transaction.commit(TransitionalTxManagementKernelTransaction.java:92) > at org.neo4j.server.rest.transactional.TransactionHandle. > closeContextAndCollectErrors(TransactionHandle.java:243) > at org.neo4j.server.rest.transactional.TransactionHandle.commit( > TransactionHandle.java:151) > at org.neo4j.server.rest.web.TransactionalService.lambda$ > executeStatementsAndCommit$28(TransactionalService.java:202) > at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider. > writeTo(StreamingOutputProvider.java:71) > at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider. > writeTo(StreamingOutputProvider.java:57) > at com.sun.jersey.spi.container.ContainerResponse.write( > ContainerResponse.java:302) > at com.sun.jersey.server.impl.application.WebApplicationImpl._ > handleRequest(WebApplicationImpl.java:1510) > at com.sun.jersey.server.impl.application.WebApplicationImpl. > handleRequest(WebApplicationImpl.java:1419) > at com.sun.jersey.server.impl.application.WebApplicationImpl. > handleRequest(WebApplicationImpl.java:1409) > at com.sun.jersey.spi.container.servlet.WebComponent.service( > WebComponent.java:409) > at com.sun.jersey.spi.container.servlet.ServletContainer. > service(ServletContainer.java:558) > at com.sun.jersey.spi.container.servlet.ServletContainer. > service(ServletContainer.java:733) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at org.eclipse.jetty.servlet.ServletHolder.handle( > ServletHolder.java:808) > at org.eclipse.jetty.servlet.ServletHandler$CachedChain. > doFilter(ServletHandler.java:1669) > at org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter( > CollectUserAgentFilter.java:69) > at org.eclipse.jetty.servlet.ServletHandler$CachedChain. > doFilter(ServletHandler.java:1652) > at org.eclipse.jetty.servlet.ServletHandler.doHandle( > ServletHandler.java:585) > at org.eclipse.jetty.server.session.SessionHandler. > doHandle(SessionHandler.java:221) > at org.eclipse.jetty.server.handler.ContextHandler. > doHandle(ContextHandler.java:1127) > at org.eclipse.jetty.servlet.ServletHandler.doScope( > ServletHandler.java:515) > at org.eclipse.jetty.server.session.SessionHandler. > doScope(SessionHandler.java:185) > at org.eclipse.jetty.server.handler.ContextHandler. > doScope(ContextHandler.java:1061) > at org.eclipse.jetty.server.handler.ScopedHandler.handle( > ScopedHandler.java:141) > at org.eclipse.jetty.server.handler.HandlerList.handle( > HandlerList.java:52) > at org.eclipse.jetty.server.handler.HandlerWrapper.handle( > HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:497) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) > at org.eclipse.jetty.server.HttpConnection.onFillable( > HttpConnection.java:257) > at org.eclipse.jetty.io.AbstractConnection$2.run( > AbstractConnection.java:540) > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( > QueuedThreadPool.java:635) > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run( > QueuedThreadPool.java:555) > at java.lang.Thread.run(Thread.java:745) > > On Tuesday, August 30, 2016 at 3:12:33 AM UTC-7, Michael Hunger wrote: >> >> Can you tell me more about your use of Neo4j and the history of this >> store? Were there crashes, out of disk situations or hard kills? Or any >> other unusual incidents? >> Perhaps you can even share a compressed copy with us / our engineering so >> that we can do some forensics? >> >> To recover/repair, try my store utils to transform the store to a fixed >> copy, while skipping and logging the invalid records. >> >> https://github.com/jexp/store-utils >> >> Michael >> >> On Tue, Aug 30, 2016 at 10:49 AM, Ray Chengr <[email protected]> wrote: >> >>> Hi Michael, >>> >>> This was the result of consistency check while Neo4j 3.0.3 was stopped: >>> >>> # java -cp 'lib/*:system/lib/*' org.neo4j.consistency.ConsistencyCheckTool >>> /data/databases/graph.db >>> Exception in thread "main" java.lang.IllegalArgumentException: Unknown >>> entry type 34 for version 0. At position LogPosition{logVersion=3203, >>> byteOffset=16} and entry version V2_0 >>> at org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion. >>> entryParser(LogEntryVersion.java:141) >>> at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogE >>> ntryReader.readLogEntry(VersionAwareLogEntryReader.java:84) >>> at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(Lo >>> gEntryCursor.java:53) >>> at org.neo4j.kernel.recovery.LatestCheckPointFinder.find(Latest >>> CheckPointFinder.java:77) >>> at org.neo4j.kernel.recovery.PositionToRecoverFrom.apply(Positi >>> onToRecoverFrom.java:53) >>> at org.neo4j.kernel.impl.recovery.RecoveryRequiredChecker.isRec >>> overyRequiredAt(RecoveryRequiredChecker.java:69) >>> at org.neo4j.consistency.ConsistencyCheckTool.checkDbState(Cons >>> istencyCheckTool.java:115) >>> at org.neo4j.consistency.ConsistencyCheckTool.run(ConsistencyCh >>> eckTool.java:92) >>> at org.neo4j.consistency.ConsistencyCheckTool.runConsistencyChe >>> ckTool(ConsistencyCheckTool.java:69) >>> at org.neo4j.consistency.ConsistencyCheckTool.main(ConsistencyC >>> heckTool.java:56) >>> >>> Also, neo4j.log in our Neo4j 3.0.3 Community Edition produces these >>> messages continuously while we are inserting relationships (not sure about >>> inserting nodes due to the amount of log lines): >>> >>> 2016-08-30 08:26:36.179+0000 ERROR Failed to commit transaction. >>> Relationship[499972766,used=false,source=-1,target=-1,type=- >>> 1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,secondaryUnitId=-1, >>> sFirst, tFirst] not in use >>> org.neo4j.kernel.impl.store.InvalidRecordException: >>> Relationship[499972766,used=false,source=-1,target=-1,type=- >>> 1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,secondaryUnitId=-1, >>> sFirst, tFirst] not in use >>> at org.neo4j.kernel.impl.store.record.RecordLoad.verify(RecordL >>> oad.java:63) >>> at org.neo4j.kernel.impl.store.CommonAbstractStore.verifyAfterR >>> eading(CommonAbstractStore.java:1263) >>> at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoReco >>> rd(CommonAbstractStore.java:1047) >>> at org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(Co >>> mmonAbstractStore.java:1025) >>> at org.neo4j.kernel.impl.transaction.state.Loaders$3.load( >>> Loaders.java:200) >>> at org.neo4j.kernel.impl.transaction.state.Loaders$3.load( >>> Loaders.java:190) >>> at org.neo4j.kernel.impl.transaction.state.RecordChanges. >>> getOrLoad(RecordChanges.java:72) >>> at org.neo4j.kernel.impl.transaction.state.RelationshipCreator. >>> connect(RelationshipCreator.java:203) >>> at org.neo4j.kernel.impl.transaction.state.RelationshipCreator. >>> connectRelationshipToDenseNode(RelationshipCreator.java:165) >>> at org.neo4j.kernel.impl.transaction.state.RelationshipCreator. >>> connectRelationship(RelationshipCreator.java:143) >>> at org.neo4j.kernel.impl.transaction.state.RelationshipCreator. >>> relationshipCreate(RelationshipCreator.java:67) >>> at org.neo4j.kernel.impl.transaction.state.TransactionRecordSta >>> te.relCreate(TransactionRecordState.java:282) >>> at org.neo4j.kernel.impl.storageengine.impl.recordstorage.Trans >>> actionToRecordStateVisitor.visitCreatedRelationship(Trans >>> actionToRecordStateVisitor.java:96) >>> at org.neo4j.storageengine.api.txstate.TxStateVisitor$Delegator >>> .visitCreatedRelationship(TxStateVisitor.java:227) >>> at org.neo4j.kernel.api.txstate.TransactionCountingStateVisitor >>> .visitCreatedRelationship(TransactionCountingStateVisitor.java:108) >>> at org.neo4j.kernel.impl.api.state.TxState$15.visitAddedRelatio >>> nship(TxState.java:376) >>> at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapte >>> r$1.visit(RelationshipChangeVisitorAdapter.java:125) >>> at org.neo4j.kernel.impl.api.state.RelationshipStateImpl.accept >>> (RelationshipStateImpl.java:52) >>> at org.neo4j.kernel.impl.api.state.TxState.relationshipVisit( >>> TxState.java:1539) >>> at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapte >>> r$1.visit(RelationshipChangeVisitorAdapter.java:115) >>> at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapte >>> r.visitAddedRelationship(RelationshipChangeVisitorAdapter.java:66) >>> at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapte >>> r.visitAdded(RelationshipChangeVisitorAdapter.java:90) >>> at org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapte >>> r.visitAdded(RelationshipChangeVisitorAdapter.java:49) >>> at org.neo4j.kernel.impl.util.diffsets.SuperDiffSets.accept(Sup >>> erDiffSets.java:66) >>> at org.neo4j.kernel.impl.util.diffsets.RelationshipDiffSets.acc >>> ept(RelationshipDiffSets.java:43) >>> at org.neo4j.kernel.impl.api.state.TxState.accept(TxState.java:265) >>> at org.neo4j.kernel.impl.storageengine.impl.recordstorage.Recor >>> dStorageEngine.createCommands(RecordStorageEngine.java:312) >>> at org.neo4j.kernel.impl.api.KernelTransactionImplementation. >>> commit(KernelTransactionImplementation.java:437) >>> at org.neo4j.kernel.impl.api.KernelTransactionImplementation. >>> close(KernelTransactionImplementation.java:386) >>> at org.neo4j.server.rest.transactional.TransitionalTxManagement >>> KernelTransaction.commit(TransitionalTxManagementKernelTrans >>> action.java:92) >>> at org.neo4j.server.rest.transactional.TransactionHandle.closeC >>> ontextAndCollectErrors(TransactionHandle.java:243) >>> at org.neo4j.server.rest.transactional.TransactionHandle. >>> commit(TransactionHandle.java:151) >>> at org.neo4j.server.rest.web.TransactionalService.lambda$execut >>> eStatementsAndCommit$28(TransactionalService.java:202) >>> at com.sun.jersey.core.impl.provider.entity.StreamingOutputProv >>> ider.writeTo(StreamingOutputProvider.java:71) >>> at com.sun.jersey.core.impl.provider.entity.StreamingOutputProv >>> ider.writeTo(StreamingOutputProvider.java:57) >>> at com.sun.jersey.spi.container.ContainerResponse.write(Contain >>> erResponse.java:302) >>> at com.sun.jersey.server.impl.application.WebApplicationImpl._h >>> andleRequest(WebApplicationImpl.java:1510) >>> at com.sun.jersey.server.impl.application.WebApplicationImpl.ha >>> ndleRequest(WebApplicationImpl.java:1419) >>> at com.sun.jersey.server.impl.application.WebApplicationImpl.ha >>> ndleRequest(WebApplicationImpl.java:1409) >>> at com.sun.jersey.spi.container.servlet.WebComponent.service(We >>> bComponent.java:409) >>> at com.sun.jersey.spi.container.servlet.ServletContainer.servic >>> e(ServletContainer.java:558) >>> at com.sun.jersey.spi.container.servlet.ServletContainer.servic >>> e(ServletContainer.java:733) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >>> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder >>> .java:808) >>> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte >>> r(ServletHandler.java:1669) >>> at org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(Co >>> llectUserAgentFilter.java:69) >>> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte >>> r(ServletHandler.java:1652) >>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan >>> dler.java:585) >>> at org.eclipse.jetty.server.session.SessionHandler.doHandle( >>> SessionHandler.java:221) >>> at org.eclipse.jetty.server.handler.ContextHandler.doHandle( >>> ContextHandler.java:1127) >>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand >>> ler.java:515) >>> at org.eclipse.jetty.server.session.SessionHandler.doScope( >>> SessionHandler.java:185) >>> at org.eclipse.jetty.server.handler.ContextHandler.doScope( >>> ContextHandler.java:1061) >>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped >>> Handler.java:141) >>> at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerL >>> ist.java:52) >>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl >>> erWrapper.java:97) >>> at org.eclipse.jetty.server.Server.handle(Server.java:497) >>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) >>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne >>> ction.java:257) >>> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnec >>> tion.java:540) >>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued >>> ThreadPool.java:635) >>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT >>> hreadPool.java:555) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> Is there a way to fix our Neo4j? >>> >>> Thanks, >>> Ray >>> >>> >>> On Sunday, August 28, 2016 at 3:11:39 PM UTC-7, Michael Hunger wrote: >>>> >>>> Can you run a consistency check on your store? >>>> >>>> https://neo4j.com/developer/kb/how-do-i-run-consistency-check-manually/ >>>> >>>> Thx >>>> >>>> Am 29.08.2016 um 00:02 schrieb Ray Chengr <[email protected]>: >>>> >>>> On our Neo4j 3.0.3, I tried to delete relationships and nodes from the >>>> browser by: >>>> >>>> match (n:node_type_A {day: 17016}) detach delete(n) >>>> >>>> I got this error: >>>> >>>> Relationship[482224340,used=false,source=-1,target=-1,type=-1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1, >>>> secondaryUnitId=-1, sFirst, tFirst] not in use >>>> >>>> >>>> match (n:node_type_A {day: 17016})-[r]->(m) with n, r limit 1000 >>>> delete n, r return count(n) as deletedRelsCount >>>> >>>> also gave me exactly the same error message. 482224340 may be the ID of >>>> a relationship. I guess there is an inconsistency of this particular >>>> relationship within Neo4j. >>>> >>>> Is there a way (or tool) to delete this "abnormal-shaped" relationship >>>> so that I can delete the rest of relationships? >>>> Thanks! >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Neo4j" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Neo4j" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
