Yep, makes sense, in general it is sensible to stay in the 30k range so you get the best utilization between tx-commit disk-flushes and memory usage of large tx. See: http://jexp.de/blog/2013/05/on-importing-data-in-neo4j-blog-series/
Am 14.07.2014 um 18:59 schrieb José Cornado <[email protected]>: > FYI, > > I ran a little, crude experiment. I increased the -Xmx (max heap size) option > from 512 to 1024 mb while incrementing the number of nodes added to the > graph. It crashed around 78K nodes. > > So a crude guideline to use is (78k - 30K)/512MB. In case you want to > minimize the number of transactions used. > > > On Mon, Jul 14, 2014 at 9:45 AM, José Cornado <[email protected]> wrote: > I am not importing. I am creating. These were very simple (one value == one > node) It can be more complex than that. Where can I find docs about the > memory usage and setup? > > > On Mon, Jul 14, 2014 at 9:35 AM, Michael Hunger > <[email protected]> wrote: > In my experience a it depends on what you do, I had good results with > tx-sizes from 1k to 30k > > But if you need to import millions at once, you might want to look into > batch-insertion, see http://neo4j.org/develop/import > > Cheers, > > Michael > > Am 14.07.2014 um 17:32 schrieb José Cornado <[email protected]>: > >> The data is fed to an engine that is able to process work in the millions. >> The number of relations is really low compared to the number of nodes so the >> cost of node per transaction is the driver. >> >> Thanks! >> >> >> On Mon, Jul 14, 2014 at 9:27 AM, José Cornado <[email protected]> wrote: >> that I know. What is the cost per element (node/rels) in a transaction? few >> kb? >> >> >> On Mon, Jul 14, 2014 at 9:24 AM, Michael Hunger >> <[email protected]> wrote: >> Make sure your tx is not too big for your memory, according to your messages >> log your JVM only runs with a few megabytes of memory. >> >> Try to limit your tx-size to 10k or 20k elements (nodes and rels) restarting >> the tx after calling >> >> if (count++ == 10000) { >> tx.success(); tx.close(); >> tx = db.beginTx(); >> count = 0; >> } >> >> Cheers, >> >> Michael >> >> Am 14.07.2014 um 17:21 schrieb José Cornado <[email protected]>: >> >>> I managed to reproduce the same exception. >>> >>> Trace and log file are included. I will put together a test case in a few >>> >>> >>> >>> (org.neo4j.graphdb.TransactionFailureException) >>> org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction >>> >>> >>> Caused by: org.neo4j.graphdb.TransactionFailureException: commit threw >>> exception >>> >>> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:498) >>> >>> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:397) >>> >>> at >>> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:122) >>> >>> at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:124) >>> >>> at endTransaction(...) >>> >>> ... 31 more >>> >>> Caused by: javax.transaction.xa.XAException >>> >>> at >>> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:553) >>> >>> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:460) >>> >>> ... 38 more >>> >>> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded >>> >>> at org.apache.lucene.util.BytesRef.<init>(BytesRef.java:77) >>> >>> at org.apache.lucene.store.DataOutput.writeString(DataOutput.java:111) >>> >>> at org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:212) >>> >>> at >>> org.apache.lucene.index.StoredFieldsWriterPerThread.addField(StoredFieldsWriterPerThread.java:58) >>> >>> at >>> org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:265) >>> >>> at >>> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:766) >>> >>> at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060) >>> >>> at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2034) >>> >>> at >>> org.neo4j.kernel.api.impl.index.LuceneIndexAccessor.add(LuceneIndexAccessor.java:151) >>> >>> at >>> org.neo4j.kernel.api.impl.index.LuceneIndexAccessor$LuceneIndexUpdater.process(LuceneIndexAccessor.java:186) >>> >>> at >>> org.neo4j.kernel.impl.api.index.FlippableIndexProxy$LockingIndexUpdater.process(FlippableIndexProxy.java:337) >>> >>> at >>> org.neo4j.kernel.impl.api.index.ContractCheckingIndexProxy$1.process(ContractCheckingIndexProxy.java:102) >>> >>> at >>> org.neo4j.kernel.impl.api.index.IndexingService.processUpdateIfIndexExists(IndexingService.java:411) >>> >>> at >>> org.neo4j.kernel.impl.api.index.IndexingService.applyUpdates(IndexingService.java:359) >>> >>> at >>> org.neo4j.kernel.impl.api.index.IndexingService.updateIndexes(IndexingService.java:310) >>> >>> at >>> org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.applyCommit(WriteTransaction.java:817) >>> >>> at >>> org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.doCommit(WriteTransaction.java:751) >>> >>> at >>> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:322) >>> >>> at >>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commitWriteTx(XaResourceManager.java:530) >>> >>> at >>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446) >>> >>> at >>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) >>> >>> at >>> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:545) >>> >>> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:460) >>> >>> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:397) >>> >>> at >>> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:122) >>> >>> >>> at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:124) >>> >>> -> endTransaction(...) >>> >>> >>> >>> On Mon, Jul 14, 2014 at 7:18 AM, José Cornado <[email protected]> >>> wrote: >>> It is crashing before it gets where the original exception occurred. >>> >>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>> >>> at >>> org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement(KernelTransactionImplementation.java:182) >>> >>> at >>> org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement(KernelTransactionImplementation.java:63) >>> >>> at >>> org.neo4j.kernel.impl.core.ThreadToStatementContextBridge.instance(ThreadToStatementContextBridge.java:47) >>> >>> at org.neo4j.kernel.impl.core.NodeProxy.addLabel(NodeProxy.java:468) >>> >>> at MY LOGIC >>> >>> at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:220) >>> >>> at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84) >>> >>> at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4166) >>> >>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1466) >>> >>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1489) >>> >>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1474) >>> >>> at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1279) >>> >>> at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4012) >>> >>> at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3651) >>> >>> at >>> org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1113) >>> >>> at >>> org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332) >>> >>> at >>> org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:997) >>> >>> at >>> org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:138) >>> >>> at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:610) >>> >>> at >>> org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332) >>> >>> at >>> org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:567) >>> >>> at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150) >>> >>> at >>> org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:124) >>> >>> at >>> org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196) >>> >>> at >>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110) >>> >>> at >>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79) >>> >>> at >>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:354) >>> >>> >>> >>> On Mon, Jul 14, 2014 at 6:52 AM, José Cornado <[email protected]> >>> wrote: >>> Let .me reproduce it again. I was able to wrap 29,000 in a transaction >>> (side note) >>> >>> >>> On Mon, Jul 14, 2014 at 12:19 AM, Michael Hunger >>> <[email protected]> wrote: >>> Can you share your code and the full exception? As well as the >>> graph.db/messages.log file >>> >>> Thx >>> >>> Sent from mobile device >>> >>> Am 13.07.2014 um 19:34 schrieb José Cornado <[email protected]>: >>> >>>> Hello! >>>> >>>> I am running int the following: >>>> >>>> After inserting around 30,000 nodes to a graph, I close the operation with >>>> transaction success. But the neo returns: >>>> >>>> Unable to commit transaction. >>>> >>>> Is there a hard limit on the size of a transaction? Doing it on a per node >>>> basis is too slow. >>>> >>>> Thanks a lot!!! >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Neo4j" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Neo4j" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >>> -- >>> José Cornado >>> >>> -- >>> >>> home: http://www.efekctive.com >>> blog: http://blogging.efekctive.com >>> ---------------------- >>> >>> Everything has been said before, but since nobody listens we have to keep >>> going back and beginning all over again. >>> >>> Andre Gide >>> >>> >>> >>> -- >>> José Cornado >>> >>> -- >>> >>> home: http://www.efekctive.com >>> blog: http://blogging.efekctive.com >>> ---------------------- >>> >>> Everything has been said before, but since nobody listens we have to keep >>> going back and beginning all over again. >>> >>> Andre Gide >>> >>> >>> >>> -- >>> José Cornado >>> >>> -- >>> >>> home: http://www.efekctive.com >>> blog: http://blogging.efekctive.com >>> ---------------------- >>> >>> Everything has been said before, but since nobody listens we have to keep >>> going back and beginning all over again. >>> >>> Andre Gide >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Neo4j" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> <messages.log> >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Neo4j" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> José Cornado >> >> -- >> >> home: http://www.efekctive.com >> blog: http://blogging.efekctive.com >> ---------------------- >> >> Everything has been said before, but since nobody listens we have to keep >> going back and beginning all over again. >> >> Andre Gide >> >> >> >> -- >> José Cornado >> >> -- >> >> home: http://www.efekctive.com >> blog: http://blogging.efekctive.com >> ---------------------- >> >> Everything has been said before, but since nobody listens we have to keep >> going back and beginning all over again. >> >> Andre Gide >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to a topic in the Google > Groups "Neo4j" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. > > > > -- > José Cornado > > -- > > home: http://www.efekctive.com > blog: http://blogging.efekctive.com > ---------------------- > > Everything has been said before, but since nobody listens we have to keep > going back and beginning all over again. > > Andre Gide > > > > -- > José Cornado > > -- > > home: http://www.efekctive.com > blog: http://blogging.efekctive.com > ---------------------- > > Everything has been said before, but since nobody listens we have to keep > going back and beginning all over again. > > Andre Gide > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
