ok.
On Mon, Jul 14, 2014 at 11:43 AM, Michael Hunger < [email protected]> wrote: > Yep, makes sense, in general it is sensible to stay in the 30k range so > you get the best utilization between tx-commit disk-flushes and memory > usage of large tx. > See: > http://jexp.de/blog/2013/05/on-importing-data-in-neo4j-blog-series/ > > > Am 14.07.2014 um 18:59 schrieb José Cornado <[email protected]>: > > FYI, > > I ran a little, crude experiment. I increased the -Xmx (max heap size) > option from 512 to 1024 mb while incrementing the number of nodes added to > the graph. It crashed around 78K nodes. > > So a crude guideline to use is (78k - 30K)/512MB. In case you want to > minimize the number of transactions used. > > > On Mon, Jul 14, 2014 at 9:45 AM, José Cornado <[email protected]> > wrote: > >> I am not importing. I am creating. These were very simple (one value == >> one node) It can be more complex than that. Where can I find docs about the >> memory usage and setup? >> >> >> On Mon, Jul 14, 2014 at 9:35 AM, Michael Hunger < >> [email protected]> wrote: >> >>> In my experience a it depends on what you do, I had good results with >>> tx-sizes from 1k to 30k >>> >>> But if you need to import millions at once, you might want to look into >>> batch-insertion, see http://neo4j.org/develop/import >>> >>> Cheers, >>> >>> Michael >>> >>> Am 14.07.2014 um 17:32 schrieb José Cornado <[email protected]>: >>> >>> The data is fed to an engine that is able to process work in the >>> millions. The number of relations is really low compared to the number of >>> nodes so the cost of node per transaction is the driver. >>> >>> Thanks! >>> >>> >>> On Mon, Jul 14, 2014 at 9:27 AM, José Cornado <[email protected]> >>> wrote: >>> >>>> that I know. What is the cost per element (node/rels) in a transaction? >>>> few kb? >>>> >>>> >>>> On Mon, Jul 14, 2014 at 9:24 AM, Michael Hunger < >>>> [email protected]> wrote: >>>> >>>>> Make sure your tx is not too big for your memory, according to your >>>>> messages log your JVM only runs with a few megabytes of memory. >>>>> >>>>> Try to limit your tx-size to 10k or 20k elements (nodes and rels) >>>>> restarting the tx after calling >>>>> >>>>> if (count++ == 10000) { >>>>> tx.success(); tx.close(); >>>>> tx = db.beginTx(); >>>>> count = 0; >>>>> } >>>>> >>>>> Cheers, >>>>> >>>>> Michael >>>>> >>>>> Am 14.07.2014 um 17:21 schrieb José Cornado <[email protected]>: >>>>> >>>>> I managed to reproduce the same exception. >>>>> >>>>> Trace and log file are included. I will put together a test case in a >>>>> few >>>>> >>>>> >>>>> (org.neo4j.graphdb.TransactionFailureException) >>>>> org.neo4j.graphdb.TransactionFailureException: Unable to commit >>>>> transaction >>>>> >>>>> >>>>> Caused by: org.neo4j.graphdb.TransactionFailureException: commit >>>>> threw exception >>>>> >>>>> at org.neo4j.kernel.impl.transaction.TxManager.commit( >>>>> TxManager.java:498) >>>>> >>>>> at org.neo4j.kernel.impl.transaction.TxManager.commit( >>>>> TxManager.java:397) >>>>> >>>>> at org.neo4j.kernel.impl.transaction.TransactionImpl.commit( >>>>> TransactionImpl.java:122) >>>>> >>>>> at org.neo4j.kernel.TopLevelTransaction.close( >>>>> TopLevelTransaction.java:124) >>>>> >>>>> at endTransaction(...) >>>>> >>>>> ... 31 more >>>>> >>>>> Caused by: javax.transaction.xa.XAException >>>>> >>>>> at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit( >>>>> TransactionImpl.java:553) >>>>> >>>>> at org.neo4j.kernel.impl.transaction.TxManager.commit( >>>>> TxManager.java:460) >>>>> >>>>> ... 38 more >>>>> >>>>> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded >>>>> >>>>> at org.apache.lucene.util.BytesRef.<init>(BytesRef.java:77) >>>>> >>>>> at org.apache.lucene.store.DataOutput.writeString(DataOutput.java:111) >>>>> >>>>> at org.apache.lucene.index.FieldsWriter.writeField( >>>>> FieldsWriter.java:212) >>>>> >>>>> at org.apache.lucene.index.StoredFieldsWriterPerThread.addField( >>>>> StoredFieldsWriterPerThread.java:58) >>>>> >>>>> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument( >>>>> DocFieldProcessorPerThread.java:265) >>>>> >>>>> at org.apache.lucene.index.DocumentsWriter.updateDocument( >>>>> DocumentsWriter.java:766) >>>>> >>>>> at org.apache.lucene.index.IndexWriter.addDocument( >>>>> IndexWriter.java:2060) >>>>> >>>>> at org.apache.lucene.index.IndexWriter.addDocument( >>>>> IndexWriter.java:2034) >>>>> >>>>> at org.neo4j.kernel.api.impl.index.LuceneIndexAccessor.add( >>>>> LuceneIndexAccessor.java:151) >>>>> >>>>> at >>>>> org.neo4j.kernel.api.impl.index.LuceneIndexAccessor$LuceneIndexUpdater.process( >>>>> LuceneIndexAccessor.java:186) >>>>> >>>>> at >>>>> org.neo4j.kernel.impl.api.index.FlippableIndexProxy$LockingIndexUpdater.process( >>>>> FlippableIndexProxy.java:337) >>>>> >>>>> at >>>>> org.neo4j.kernel.impl.api.index.ContractCheckingIndexProxy$1.process( >>>>> ContractCheckingIndexProxy.java:102) >>>>> >>>>> at >>>>> org.neo4j.kernel.impl.api.index.IndexingService.processUpdateIfIndexExists( >>>>> IndexingService.java:411) >>>>> >>>>> at org.neo4j.kernel.impl.api.index.IndexingService.applyUpdates( >>>>> IndexingService.java:359) >>>>> >>>>> at org.neo4j.kernel.impl.api.index.IndexingService.updateIndexes( >>>>> IndexingService.java:310) >>>>> >>>>> at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.applyCommit( >>>>> WriteTransaction.java:817) >>>>> >>>>> at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.doCommit( >>>>> WriteTransaction.java:751) >>>>> >>>>> at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit( >>>>> XaTransaction.java:322) >>>>> >>>>> at >>>>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commitWriteTx( >>>>> XaResourceManager.java:530) >>>>> >>>>> at >>>>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit( >>>>> XaResourceManager.java:446) >>>>> >>>>> at >>>>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit( >>>>> XaResourceHelpImpl.java:64) >>>>> >>>>> at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit( >>>>> TransactionImpl.java:545) >>>>> >>>>> at org.neo4j.kernel.impl.transaction.TxManager.commit( >>>>> TxManager.java:460) >>>>> >>>>> at org.neo4j.kernel.impl.transaction.TxManager.commit( >>>>> TxManager.java:397) >>>>> >>>>> at org.neo4j.kernel.impl.transaction.TransactionImpl.commit( >>>>> TransactionImpl.java:122) >>>>> >>>>> at org.neo4j.kernel.TopLevelTransaction.close( >>>>> TopLevelTransaction.java:124) >>>>> >>>>> -> endTransaction(...) >>>>> >>>>> >>>>> On Mon, Jul 14, 2014 at 7:18 AM, José Cornado <[email protected]> >>>>> wrote: >>>>> >>>>>> It is crashing before it gets where the original exception occurred. >>>>>> >>>>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>>>> >>>>>> at >>>>>> org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement( >>>>>> KernelTransactionImplementation.java:182) >>>>>> >>>>>> at >>>>>> org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement( >>>>>> KernelTransactionImplementation.java:63) >>>>>> >>>>>> at org.neo4j.kernel.impl.core.ThreadToStatementContextBridge.instance( >>>>>> ThreadToStatementContextBridge.java:47) >>>>>> >>>>>> at org.neo4j.kernel.impl.core.NodeProxy.addLabel(NodeProxy.java:468) >>>>>> >>>>>> at MY LOGIC >>>>>> >>>>>> at org.eclipse.swt.widgets.TypedListener.handleEvent( >>>>>> TypedListener.java:220) >>>>>> >>>>>> at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84) >>>>>> >>>>>> at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4166) >>>>>> >>>>>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1466) >>>>>> >>>>>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1489) >>>>>> >>>>>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1474) >>>>>> >>>>>> at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1279) >>>>>> >>>>>> at org.eclipse.swt.widgets.Display.runDeferredEvents( >>>>>> Display.java:4012) >>>>>> >>>>>> at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3651) >>>>>> >>>>>> at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run( >>>>>> PartRenderingEngine.java:1113) >>>>>> >>>>>> at org.eclipse.core.databinding.observable.Realm.runWithDefault( >>>>>> Realm.java:332) >>>>>> >>>>>> at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run( >>>>>> PartRenderingEngine.java:997) >>>>>> >>>>>> at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI( >>>>>> E4Workbench.java:138) >>>>>> >>>>>> at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:610) >>>>>> >>>>>> at org.eclipse.core.databinding.observable.Realm.runWithDefault( >>>>>> Realm.java:332) >>>>>> >>>>>> at org.eclipse.ui.internal.Workbench.createAndRunWorkbench( >>>>>> Workbench.java:567) >>>>>> >>>>>> at org.eclipse.ui.PlatformUI.createAndRunWorkbench( >>>>>> PlatformUI.java:150) >>>>>> >>>>>> at org.eclipse.ui.internal.ide.application.IDEApplication.start( >>>>>> IDEApplication.java:124) >>>>>> >>>>>> at org.eclipse.equinox.internal.app.EclipseAppHandle.run( >>>>>> EclipseAppHandle.java:196) >>>>>> >>>>>> at >>>>>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication( >>>>>> EclipseAppLauncher.java:110) >>>>>> >>>>>> at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start( >>>>>> EclipseAppLauncher.java:79) >>>>>> >>>>>> at org.eclipse.core.runtime.adaptor.EclipseStarter.run( >>>>>> EclipseStarter.java:354) >>>>>> >>>>>> >>>>>> On Mon, Jul 14, 2014 at 6:52 AM, José Cornado <[email protected] >>>>>> > wrote: >>>>>> >>>>>>> Let .me reproduce it again. I was able to wrap 29,000 in a >>>>>>> transaction (side note) >>>>>>> >>>>>>> >>>>>>> On Mon, Jul 14, 2014 at 12:19 AM, Michael Hunger < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Can you share your code and the full exception? As well as the >>>>>>>> graph.db/messages.log file >>>>>>>> >>>>>>>> Thx >>>>>>>> >>>>>>>> Sent from mobile device >>>>>>>> >>>>>>>> Am 13.07.2014 um 19:34 schrieb José Cornado <[email protected] >>>>>>>> >: >>>>>>>> >>>>>>>> Hello! >>>>>>>> >>>>>>>> I am running int the following: >>>>>>>> >>>>>>>> After inserting around 30,000 nodes to a graph, I close the >>>>>>>> operation with transaction success. But the neo returns: >>>>>>>> >>>>>>>> Unable to commit transaction. >>>>>>>> >>>>>>>> Is there a hard limit on the size of a transaction? Doing it on a >>>>>>>> per node basis is too slow. >>>>>>>> >>>>>>>> Thanks a lot!!! >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Neo4j" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "Neo4j" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> José Cornado >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> home: http://www.efekctive.com >>>>>>> blog: http://blogging.efekctive.com >>>>>>> ---------------------- >>>>>>> >>>>>>> Everything has been said before, but since nobody listens we have to >>>>>>> keep going back and beginning all over again. >>>>>>> >>>>>>> Andre Gide >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> José Cornado >>>>>> >>>>>> -- >>>>>> >>>>>> home: http://www.efekctive.com >>>>>> blog: http://blogging.efekctive.com >>>>>> ---------------------- >>>>>> >>>>>> Everything has been said before, but since nobody listens we have to >>>>>> keep going back and beginning all over again. >>>>>> >>>>>> Andre Gide >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> José Cornado >>>>> >>>>> -- >>>>> >>>>> home: http://www.efekctive.com >>>>> blog: http://blogging.efekctive.com >>>>> ---------------------- >>>>> >>>>> Everything has been said before, but since nobody listens we have to >>>>> keep going back and beginning all over again. >>>>> >>>>> Andre Gide >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Neo4j" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> <messages.log> >>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Neo4j" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> José Cornado >>>> >>>> -- >>>> >>>> home: http://www.efekctive.com >>>> blog: http://blogging.efekctive.com >>>> ---------------------- >>>> >>>> Everything has been said before, but since nobody listens we have to >>>> keep going back and beginning all over again. >>>> >>>> Andre Gide >>>> >>> >>> >>> >>> -- >>> José Cornado >>> >>> -- >>> >>> home: http://www.efekctive.com >>> blog: http://blogging.efekctive.com >>> ---------------------- >>> >>> Everything has been said before, but since nobody listens we have to >>> keep going back and beginning all over again. >>> >>> Andre Gide >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Neo4j" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Neo4j" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> José Cornado >> >> -- >> >> home: http://www.efekctive.com >> blog: http://blogging.efekctive.com >> ---------------------- >> >> Everything has been said before, but since nobody listens we have to keep >> going back and beginning all over again. >> >> Andre Gide >> > > > > -- > José Cornado > > -- > > home: http://www.efekctive.com > blog: http://blogging.efekctive.com > ---------------------- > > Everything has been said before, but since nobody listens we have to keep > going back and beginning all over again. > > Andre Gide > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Neo4j" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- José Cornado -- home: http://www.efekctive.com blog: http://blogging.efekctive.com ---------------------- Everything has been said before, but since nobody listens we have to keep going back and beginning all over again. Andre Gide -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
