FYI, I ran a little, crude experiment. I increased the -Xmx (max heap size) option from 512 to 1024 mb while incrementing the number of nodes added to the graph. It crashed around 78K nodes.
So a crude guideline to use is (78k - 30K)/512MB. In case you want to minimize the number of transactions used. On Mon, Jul 14, 2014 at 9:45 AM, José Cornado <[email protected]> wrote: > I am not importing. I am creating. These were very simple (one value == > one node) It can be more complex than that. Where can I find docs about the > memory usage and setup? > > > On Mon, Jul 14, 2014 at 9:35 AM, Michael Hunger < > [email protected]> wrote: > >> In my experience a it depends on what you do, I had good results with >> tx-sizes from 1k to 30k >> >> But if you need to import millions at once, you might want to look into >> batch-insertion, see http://neo4j.org/develop/import >> >> Cheers, >> >> Michael >> >> Am 14.07.2014 um 17:32 schrieb José Cornado <[email protected]>: >> >> The data is fed to an engine that is able to process work in the >> millions. The number of relations is really low compared to the number of >> nodes so the cost of node per transaction is the driver. >> >> Thanks! >> >> >> On Mon, Jul 14, 2014 at 9:27 AM, José Cornado <[email protected]> >> wrote: >> >>> that I know. What is the cost per element (node/rels) in a transaction? >>> few kb? >>> >>> >>> On Mon, Jul 14, 2014 at 9:24 AM, Michael Hunger < >>> [email protected]> wrote: >>> >>>> Make sure your tx is not too big for your memory, according to your >>>> messages log your JVM only runs with a few megabytes of memory. >>>> >>>> Try to limit your tx-size to 10k or 20k elements (nodes and rels) >>>> restarting the tx after calling >>>> >>>> if (count++ == 10000) { >>>> tx.success(); tx.close(); >>>> tx = db.beginTx(); >>>> count = 0; >>>> } >>>> >>>> Cheers, >>>> >>>> Michael >>>> >>>> Am 14.07.2014 um 17:21 schrieb José Cornado <[email protected]>: >>>> >>>> I managed to reproduce the same exception. >>>> >>>> Trace and log file are included. I will put together a test case in a >>>> few >>>> >>>> >>>> (org.neo4j.graphdb.TransactionFailureException) >>>> org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction >>>> >>>> >>>> Caused by: org.neo4j.graphdb.TransactionFailureException: commit threw >>>> exception >>>> >>>> at org.neo4j.kernel.impl.transaction.TxManager.commit( >>>> TxManager.java:498) >>>> >>>> at org.neo4j.kernel.impl.transaction.TxManager.commit( >>>> TxManager.java:397) >>>> >>>> at org.neo4j.kernel.impl.transaction.TransactionImpl.commit( >>>> TransactionImpl.java:122) >>>> >>>> at org.neo4j.kernel.TopLevelTransaction.close( >>>> TopLevelTransaction.java:124) >>>> >>>> at endTransaction(...) >>>> >>>> ... 31 more >>>> >>>> Caused by: javax.transaction.xa.XAException >>>> >>>> at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit( >>>> TransactionImpl.java:553) >>>> >>>> at org.neo4j.kernel.impl.transaction.TxManager.commit( >>>> TxManager.java:460) >>>> >>>> ... 38 more >>>> >>>> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded >>>> >>>> at org.apache.lucene.util.BytesRef.<init>(BytesRef.java:77) >>>> >>>> at org.apache.lucene.store.DataOutput.writeString(DataOutput.java:111) >>>> >>>> at org.apache.lucene.index.FieldsWriter.writeField( >>>> FieldsWriter.java:212) >>>> >>>> at org.apache.lucene.index.StoredFieldsWriterPerThread.addField( >>>> StoredFieldsWriterPerThread.java:58) >>>> >>>> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument( >>>> DocFieldProcessorPerThread.java:265) >>>> >>>> at org.apache.lucene.index.DocumentsWriter.updateDocument( >>>> DocumentsWriter.java:766) >>>> >>>> at org.apache.lucene.index.IndexWriter.addDocument( >>>> IndexWriter.java:2060) >>>> >>>> at org.apache.lucene.index.IndexWriter.addDocument( >>>> IndexWriter.java:2034) >>>> >>>> at org.neo4j.kernel.api.impl.index.LuceneIndexAccessor.add( >>>> LuceneIndexAccessor.java:151) >>>> >>>> at >>>> org.neo4j.kernel.api.impl.index.LuceneIndexAccessor$LuceneIndexUpdater.process( >>>> LuceneIndexAccessor.java:186) >>>> >>>> at >>>> org.neo4j.kernel.impl.api.index.FlippableIndexProxy$LockingIndexUpdater.process( >>>> FlippableIndexProxy.java:337) >>>> >>>> at org.neo4j.kernel.impl.api.index.ContractCheckingIndexProxy$1.process( >>>> ContractCheckingIndexProxy.java:102) >>>> >>>> at >>>> org.neo4j.kernel.impl.api.index.IndexingService.processUpdateIfIndexExists( >>>> IndexingService.java:411) >>>> >>>> at org.neo4j.kernel.impl.api.index.IndexingService.applyUpdates( >>>> IndexingService.java:359) >>>> >>>> at org.neo4j.kernel.impl.api.index.IndexingService.updateIndexes( >>>> IndexingService.java:310) >>>> >>>> at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.applyCommit( >>>> WriteTransaction.java:817) >>>> >>>> at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.doCommit( >>>> WriteTransaction.java:751) >>>> >>>> at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit( >>>> XaTransaction.java:322) >>>> >>>> at >>>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commitWriteTx( >>>> XaResourceManager.java:530) >>>> >>>> at >>>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit( >>>> XaResourceManager.java:446) >>>> >>>> at >>>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit( >>>> XaResourceHelpImpl.java:64) >>>> >>>> at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit( >>>> TransactionImpl.java:545) >>>> >>>> at org.neo4j.kernel.impl.transaction.TxManager.commit( >>>> TxManager.java:460) >>>> >>>> at org.neo4j.kernel.impl.transaction.TxManager.commit( >>>> TxManager.java:397) >>>> >>>> at org.neo4j.kernel.impl.transaction.TransactionImpl.commit( >>>> TransactionImpl.java:122) >>>> >>>> at org.neo4j.kernel.TopLevelTransaction.close( >>>> TopLevelTransaction.java:124) >>>> >>>> -> endTransaction(...) >>>> >>>> >>>> On Mon, Jul 14, 2014 at 7:18 AM, José Cornado <[email protected]> >>>> wrote: >>>> >>>>> It is crashing before it gets where the original exception occurred. >>>>> >>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>>> >>>>> at >>>>> org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement( >>>>> KernelTransactionImplementation.java:182) >>>>> >>>>> at >>>>> org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement( >>>>> KernelTransactionImplementation.java:63) >>>>> >>>>> at org.neo4j.kernel.impl.core.ThreadToStatementContextBridge.instance( >>>>> ThreadToStatementContextBridge.java:47) >>>>> >>>>> at org.neo4j.kernel.impl.core.NodeProxy.addLabel(NodeProxy.java:468) >>>>> >>>>> at MY LOGIC >>>>> >>>>> at org.eclipse.swt.widgets.TypedListener.handleEvent( >>>>> TypedListener.java:220) >>>>> >>>>> at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84) >>>>> >>>>> at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4166) >>>>> >>>>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1466) >>>>> >>>>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1489) >>>>> >>>>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1474) >>>>> >>>>> at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1279) >>>>> >>>>> at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4012 >>>>> ) >>>>> >>>>> at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3651) >>>>> >>>>> at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run( >>>>> PartRenderingEngine.java:1113) >>>>> >>>>> at org.eclipse.core.databinding.observable.Realm.runWithDefault( >>>>> Realm.java:332) >>>>> >>>>> at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run( >>>>> PartRenderingEngine.java:997) >>>>> >>>>> at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI( >>>>> E4Workbench.java:138) >>>>> >>>>> at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:610) >>>>> >>>>> at org.eclipse.core.databinding.observable.Realm.runWithDefault( >>>>> Realm.java:332) >>>>> >>>>> at org.eclipse.ui.internal.Workbench.createAndRunWorkbench( >>>>> Workbench.java:567) >>>>> >>>>> at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150 >>>>> ) >>>>> >>>>> at org.eclipse.ui.internal.ide.application.IDEApplication.start( >>>>> IDEApplication.java:124) >>>>> >>>>> at org.eclipse.equinox.internal.app.EclipseAppHandle.run( >>>>> EclipseAppHandle.java:196) >>>>> >>>>> at >>>>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication( >>>>> EclipseAppLauncher.java:110) >>>>> >>>>> at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start( >>>>> EclipseAppLauncher.java:79) >>>>> >>>>> at org.eclipse.core.runtime.adaptor.EclipseStarter.run( >>>>> EclipseStarter.java:354) >>>>> >>>>> >>>>> On Mon, Jul 14, 2014 at 6:52 AM, José Cornado <[email protected]> >>>>> wrote: >>>>> >>>>>> Let .me reproduce it again. I was able to wrap 29,000 in a >>>>>> transaction (side note) >>>>>> >>>>>> >>>>>> On Mon, Jul 14, 2014 at 12:19 AM, Michael Hunger < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Can you share your code and the full exception? As well as the >>>>>>> graph.db/messages.log file >>>>>>> >>>>>>> Thx >>>>>>> >>>>>>> Sent from mobile device >>>>>>> >>>>>>> Am 13.07.2014 um 19:34 schrieb José Cornado <[email protected] >>>>>>> >: >>>>>>> >>>>>>> Hello! >>>>>>> >>>>>>> I am running int the following: >>>>>>> >>>>>>> After inserting around 30,000 nodes to a graph, I close the >>>>>>> operation with transaction success. But the neo returns: >>>>>>> >>>>>>> Unable to commit transaction. >>>>>>> >>>>>>> Is there a hard limit on the size of a transaction? Doing it on a >>>>>>> per node basis is too slow. >>>>>>> >>>>>>> Thanks a lot!!! >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Neo4j" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "Neo4j" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> José Cornado >>>>>> >>>>>> -- >>>>>> >>>>>> home: http://www.efekctive.com >>>>>> blog: http://blogging.efekctive.com >>>>>> ---------------------- >>>>>> >>>>>> Everything has been said before, but since nobody listens we have to >>>>>> keep going back and beginning all over again. >>>>>> >>>>>> Andre Gide >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> José Cornado >>>>> >>>>> -- >>>>> >>>>> home: http://www.efekctive.com >>>>> blog: http://blogging.efekctive.com >>>>> ---------------------- >>>>> >>>>> Everything has been said before, but since nobody listens we have to >>>>> keep going back and beginning all over again. >>>>> >>>>> Andre Gide >>>>> >>>> >>>> >>>> >>>> -- >>>> José Cornado >>>> >>>> -- >>>> >>>> home: http://www.efekctive.com >>>> blog: http://blogging.efekctive.com >>>> ---------------------- >>>> >>>> Everything has been said before, but since nobody listens we have to >>>> keep going back and beginning all over again. >>>> >>>> Andre Gide >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Neo4j" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> <messages.log> >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Neo4j" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> José Cornado >>> >>> -- >>> >>> home: http://www.efekctive.com >>> blog: http://blogging.efekctive.com >>> ---------------------- >>> >>> Everything has been said before, but since nobody listens we have to >>> keep going back and beginning all over again. >>> >>> Andre Gide >>> >> >> >> >> -- >> José Cornado >> >> -- >> >> home: http://www.efekctive.com >> blog: http://blogging.efekctive.com >> ---------------------- >> >> Everything has been said before, but since nobody listens we have to keep >> going back and beginning all over again. >> >> Andre Gide >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Neo4j" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > José Cornado > > -- > > home: http://www.efekctive.com > blog: http://blogging.efekctive.com > ---------------------- > > Everything has been said before, but since nobody listens we have to keep > going back and beginning all over again. > > Andre Gide > -- José Cornado -- home: http://www.efekctive.com blog: http://blogging.efekctive.com ---------------------- Everything has been said before, but since nobody listens we have to keep going back and beginning all over again. Andre Gide -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
