Hi Michael,
Sorry, I copy-pasted wrong text in my last reply two minutes ago. Please
ignore that one.
We do insertion at rate about 250 relationships&nodes per second(!) on a
single server. The disk usage is as follows: (422GB, it's too big to send)
[root@sjc graph.db]# du -s * /* this is in Kbytes */
4 index
8 neostore
4 neostore.counts.db.a
4 neostore.counts.db.b
4 neostore.id
8 neostore.labeltokenstore.db
4 neostore.labeltokenstore.db.id
8 neostore.labeltokenstore.db.names
4 neostore.labeltokenstore.db.names.id
185148 neostore.nodestore.db
4 neostore.nodestore.db.id
8 neostore.nodestore.db.labels
4 neostore.nodestore.db.labels.id
89701060 neostore.propertystore.db
332845360 neostore.propertystore.db.arrays
4 neostore.propertystore.db.arrays.id
4 neostore.propertystore.db.id
8 neostore.propertystore.db.index
4 neostore.propertystore.db.index.id
8 neostore.propertystore.db.index.keys
4 neostore.propertystore.db.index.keys.id
1336196 neostore.propertystore.db.strings
4 neostore.propertystore.db.strings.id
29636 neostore.relationshipgroupstore.db
4 neostore.relationshipgroupstore.db.id
16814260 neostore.relationshipstore.db
88 neostore.relationshipstore.db.id
8 neostore.relationshiptypestore.db
4 neostore.relationshiptypestore.db.id
8 neostore.relationshiptypestore.db.names
4 neostore.relationshiptypestore.db.names.id
8 neostore.schemastore.db
4 neostore.schemastore.db.id
214076 neostore.transaction.db.3203
1302104 schema
0 store_lock
[root@ddna-sjc graph.db]# du -s . /* also in Kbytes */
442428080 .
[root@ddna-sjc graph.db]# du -s -h .
422G .
I started insertion and got these same messages from neo4j.log
continuously, even with the same relationship ID (?) 500676928.
2016-08-30 10:19:02.252+0000 ERROR Failed to commit transaction.
Relationship[500676928,used=false,source=-1,target=-1,type=-1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,secondaryUnitId=-1,
sFirst, tFirst] not in use
org.neo4j.kernel.impl.store.InvalidRecordException:
Relationship[500676928,used=false,source=-1,target=-1,type=-1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,secondaryUnitId=-1,
sFirst, tFirst] not in use
at
org.neo4j.kernel.impl.store.record.RecordLoad.verify(RecordLoad.java:63)
at
org.neo4j.kernel.impl.store.CommonAbstractStore.verifyAfterReading(CommonAbstractStore.java:1263)
at
org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1047)
at
org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1025)
at
org.neo4j.kernel.impl.transaction.state.Loaders$3.load(Loaders.java:200)
at
org.neo4j.kernel.impl.transaction.state.Loaders$3.load(Loaders.java:190)
at
org.neo4j.kernel.impl.transaction.state.RecordChanges.getOrLoad(RecordChanges.java:72)
at
org.neo4j.kernel.impl.transaction.state.RelationshipCreator.connect(RelationshipCreator.java:203)
at
org.neo4j.kernel.impl.transaction.state.RelationshipCreator.connectRelationshipToDenseNode(RelationshipCreator.java:165)
at
org.neo4j.kernel.impl.transaction.state.RelationshipCreator.connectRelationship(RelationshipCreator.java:143)
at
org.neo4j.kernel.impl.transaction.state.RelationshipCreator.relationshipCreate(RelationshipCreator.java:67)
at
org.neo4j.kernel.impl.transaction.state.TransactionRecordState.relCreate(TransactionRecordState.java:282)
at
org.neo4j.kernel.impl.storageengine.impl.recordstorage.TransactionToRecordStateVisitor.visitCreatedRelationship(TransactionToRecordStateVisitor.java:96)
at
org.neo4j.storageengine.api.txstate.TxStateVisitor$Delegator.visitCreatedRelationship(TxStateVisitor.java:227)
at
org.neo4j.kernel.api.txstate.TransactionCountingStateVisitor.visitCreatedRelationship(TransactionCountingStateVisitor.java:108)
at
org.neo4j.kernel.impl.api.state.TxState$15.visitAddedRelationship(TxState.java:376)
at
org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter$1.visit(RelationshipChangeVisitorAdapter.java:125)
at
org.neo4j.kernel.impl.api.state.RelationshipStateImpl.accept(RelationshipStateImpl.java:52)
at
org.neo4j.kernel.impl.api.state.TxState.relationshipVisit(TxState.java:1539)
at
org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter$1.visit(RelationshipChangeVisitorAdapter.java:115)
at
org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter.visitAddedRelationship(RelationshipChangeVisitorAdapter.java:66)
at
org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter.visitAdded(RelationshipChangeVisitorAdapter.java:90)
at
org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter.visitAdded(RelationshipChangeVisitorAdapter.java:49)
at
org.neo4j.kernel.impl.util.diffsets.SuperDiffSets.accept(SuperDiffSets.java:66)
at
org.neo4j.kernel.impl.util.diffsets.RelationshipDiffSets.accept(RelationshipDiffSets.java:43)
at org.neo4j.kernel.impl.api.state.TxState.accept(TxState.java:265)
at
org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.createCommands(RecordStorageEngine.java:312)
at
org.neo4j.kernel.impl.api.KernelTransactionImplementation.commit(KernelTransactionImplementation.java:437)
at
org.neo4j.kernel.impl.api.KernelTransactionImplementation.close(KernelTransactionImplementation.java:386)
at
org.neo4j.server.rest.transactional.TransitionalTxManagementKernelTransaction.commit(TransitionalTxManagementKernelTransaction.java:92)
at
org.neo4j.server.rest.transactional.TransactionHandle.closeContextAndCollectErrors(TransactionHandle.java:243)
at
org.neo4j.server.rest.transactional.TransactionHandle.commit(TransactionHandle.java:151)
at
org.neo4j.server.rest.web.TransactionalService.lambda$executeStatementsAndCommit$28(TransactionalService.java:202)
at
com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
at
com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
at
com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at
org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(CollectUserAgentFilter.java:69)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
On Tuesday, August 30, 2016 at 3:12:33 AM UTC-7, Michael Hunger wrote:
>
> Can you tell me more about your use of Neo4j and the history of this
> store? Were there crashes, out of disk situations or hard kills? Or any
> other unusual incidents?
> Perhaps you can even share a compressed copy with us / our engineering so
> that we can do some forensics?
>
> To recover/repair, try my store utils to transform the store to a fixed
> copy, while skipping and logging the invalid records.
>
> https://github.com/jexp/store-utils
>
> Michael
>
> On Tue, Aug 30, 2016 at 10:49 AM, Ray Chengr <[email protected]
> <javascript:>> wrote:
>
>> Hi Michael,
>>
>> This was the result of consistency check while Neo4j 3.0.3 was stopped:
>>
>> # java -cp 'lib/*:system/lib/*'
>> org.neo4j.consistency.ConsistencyCheckTool /data/databases/graph.db
>> Exception in thread "main" java.lang.IllegalArgumentException: Unknown
>> entry type 34 for version 0. At position LogPosition{logVersion=3203,
>> byteOffset=16} and entry version V2_0
>> at
>> org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion.entryParser(LogEntryVersion.java:141)
>> at
>> org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:84)
>> at
>> org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:53)
>> at
>> org.neo4j.kernel.recovery.LatestCheckPointFinder.find(LatestCheckPointFinder.java:77)
>> at
>> org.neo4j.kernel.recovery.PositionToRecoverFrom.apply(PositionToRecoverFrom.java:53)
>> at
>> org.neo4j.kernel.impl.recovery.RecoveryRequiredChecker.isRecoveryRequiredAt(RecoveryRequiredChecker.java:69)
>> at
>> org.neo4j.consistency.ConsistencyCheckTool.checkDbState(ConsistencyCheckTool.java:115)
>> at
>> org.neo4j.consistency.ConsistencyCheckTool.run(ConsistencyCheckTool.java:92)
>> at
>> org.neo4j.consistency.ConsistencyCheckTool.runConsistencyCheckTool(ConsistencyCheckTool.java:69)
>> at
>> org.neo4j.consistency.ConsistencyCheckTool.main(ConsistencyCheckTool.java:56)
>>
>> Also, neo4j.log in our Neo4j 3.0.3 Community Edition produces these
>> messages continuously while we are inserting relationships (not sure about
>> inserting nodes due to the amount of log lines):
>>
>> 2016-08-30 08:26:36.179+0000 ERROR Failed to commit transaction.
>> Relationship[499972766,used=false,source=-1,target=-1,type=-1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,secondaryUnitId=-1,
>>
>> sFirst, tFirst] not in use
>> org.neo4j.kernel.impl.store.InvalidRecordException:
>> Relationship[499972766,used=false,source=-1,target=-1,type=-1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,secondaryUnitId=-1,
>>
>> sFirst, tFirst] not in use
>> at
>> org.neo4j.kernel.impl.store.record.RecordLoad.verify(RecordLoad.java:63)
>> at
>> org.neo4j.kernel.impl.store.CommonAbstractStore.verifyAfterReading(CommonAbstractStore.java:1263)
>> at
>> org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1047)
>> at
>> org.neo4j.kernel.impl.store.CommonAbstractStore.getRecord(CommonAbstractStore.java:1025)
>> at
>> org.neo4j.kernel.impl.transaction.state.Loaders$3.load(Loaders.java:200)
>> at
>> org.neo4j.kernel.impl.transaction.state.Loaders$3.load(Loaders.java:190)
>> at
>> org.neo4j.kernel.impl.transaction.state.RecordChanges.getOrLoad(RecordChanges.java:72)
>> at
>> org.neo4j.kernel.impl.transaction.state.RelationshipCreator.connect(RelationshipCreator.java:203)
>> at
>> org.neo4j.kernel.impl.transaction.state.RelationshipCreator.connectRelationshipToDenseNode(RelationshipCreator.java:165)
>> at
>> org.neo4j.kernel.impl.transaction.state.RelationshipCreator.connectRelationship(RelationshipCreator.java:143)
>> at
>> org.neo4j.kernel.impl.transaction.state.RelationshipCreator.relationshipCreate(RelationshipCreator.java:67)
>> at
>> org.neo4j.kernel.impl.transaction.state.TransactionRecordState.relCreate(TransactionRecordState.java:282)
>> at
>> org.neo4j.kernel.impl.storageengine.impl.recordstorage.TransactionToRecordStateVisitor.visitCreatedRelationship(TransactionToRecordStateVisitor.java:96)
>> at
>> org.neo4j.storageengine.api.txstate.TxStateVisitor$Delegator.visitCreatedRelationship(TxStateVisitor.java:227)
>> at
>> org.neo4j.kernel.api.txstate.TransactionCountingStateVisitor.visitCreatedRelationship(TransactionCountingStateVisitor.java:108)
>> at
>> org.neo4j.kernel.impl.api.state.TxState$15.visitAddedRelationship(TxState.java:376)
>> at
>> org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter$1.visit(RelationshipChangeVisitorAdapter.java:125)
>> at
>> org.neo4j.kernel.impl.api.state.RelationshipStateImpl.accept(RelationshipStateImpl.java:52)
>> at
>> org.neo4j.kernel.impl.api.state.TxState.relationshipVisit(TxState.java:1539)
>> at
>> org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter$1.visit(RelationshipChangeVisitorAdapter.java:115)
>> at
>> org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter.visitAddedRelationship(RelationshipChangeVisitorAdapter.java:66)
>> at
>> org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter.visitAdded(RelationshipChangeVisitorAdapter.java:90)
>> at
>> org.neo4j.kernel.api.txstate.RelationshipChangeVisitorAdapter.visitAdded(RelationshipChangeVisitorAdapter.java:49)
>> at
>> org.neo4j.kernel.impl.util.diffsets.SuperDiffSets.accept(SuperDiffSets.java:66)
>> at
>> org.neo4j.kernel.impl.util.diffsets.RelationshipDiffSets.accept(RelationshipDiffSets.java:43)
>> at org.neo4j.kernel.impl.api.state.TxState.accept(TxState.java:265)
>> at
>> org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.createCommands(RecordStorageEngine.java:312)
>> at
>> org.neo4j.kernel.impl.api.KernelTransactionImplementation.commit(KernelTransactionImplementation.java:437)
>> at
>> org.neo4j.kernel.impl.api.KernelTransactionImplementation.close(KernelTransactionImplementation.java:386)
>> at
>> org.neo4j.server.rest.transactional.TransitionalTxManagementKernelTransaction.commit(TransitionalTxManagementKernelTransaction.java:92)
>> at
>> org.neo4j.server.rest.transactional.TransactionHandle.closeContextAndCollectErrors(TransactionHandle.java:243)
>> at
>> org.neo4j.server.rest.transactional.TransactionHandle.commit(TransactionHandle.java:151)
>> at
>> org.neo4j.server.rest.web.TransactionalService.lambda$executeStatementsAndCommit$28(TransactionalService.java:202)
>> at
>> com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
>> at
>> com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
>> at
>> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302)
>> at
>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510)
>> at
>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
>> at
>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
>> at
>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
>> at
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
>> at
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>> at
>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
>> at
>> org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(CollectUserAgentFilter.java:69)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> at
>> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> at org.eclipse.jetty.server.Server.handle(Server.java:497)
>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> at org.eclipse.jetty.io
>> .AbstractConnection$2.run(AbstractConnection.java:540)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Is there a way to fix our Neo4j?
>>
>> Thanks,
>> Ray
>>
>>
>> On Sunday, August 28, 2016 at 3:11:39 PM UTC-7, Michael Hunger wrote:
>>>
>>> Can you run a consistency check on your store?
>>>
>>> https://neo4j.com/developer/kb/how-do-i-run-consistency-check-manually/
>>>
>>> Thx
>>>
>>> Am 29.08.2016 um 00:02 schrieb Ray Chengr <[email protected]>:
>>>
>>> On our Neo4j 3.0.3, I tried to delete relationships and nodes from the
>>> browser by:
>>>
>>> match (n:node_type_A {day: 17016}) detach delete(n)
>>>
>>> I got this error:
>>>
>>> Relationship[482224340,used=false,source=-1,target=-1,type=-1,sCount=1,sNext=-1,tCount=1,tNext=-1,prop=-1,
>>> secondaryUnitId=-1, sFirst, tFirst] not in use
>>>
>>>
>>> match (n:node_type_A {day: 17016})-[r]->(m) with n, r limit 1000 delete
>>> n, r return count(n) as deletedRelsCount
>>>
>>> also gave me exactly the same error message. 482224340 may be the ID of
>>> a relationship. I guess there is an inconsistency of this particular
>>> relationship within Neo4j.
>>>
>>> Is there a way (or tool) to delete this "abnormal-shaped" relationship
>>> so that I can delete the rest of relationships?
>>> Thanks!
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Neo4j" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "Neo4j" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
