The data is fed to an engine that is able to process work in the millions. The number of relations is really low compared to the number of nodes so the cost of node per transaction is the driver.
Thanks! On Mon, Jul 14, 2014 at 9:27 AM, José Cornado <[email protected]> wrote: > that I know. What is the cost per element (node/rels) in a transaction? > few kb? > > > On Mon, Jul 14, 2014 at 9:24 AM, Michael Hunger < > [email protected]> wrote: > >> Make sure your tx is not too big for your memory, according to your >> messages log your JVM only runs with a few megabytes of memory. >> >> Try to limit your tx-size to 10k or 20k elements (nodes and rels) >> restarting the tx after calling >> >> if (count++ == 10000) { >> tx.success(); tx.close(); >> tx = db.beginTx(); >> count = 0; >> } >> >> Cheers, >> >> Michael >> >> Am 14.07.2014 um 17:21 schrieb José Cornado <[email protected]>: >> >> I managed to reproduce the same exception. >> >> Trace and log file are included. I will put together a test case in a few >> >> >> (org.neo4j.graphdb.TransactionFailureException) >> org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction >> >> >> Caused by: org.neo4j.graphdb.TransactionFailureException: commit threw >> exception >> >> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:498) >> >> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:397) >> >> at org.neo4j.kernel.impl.transaction.TransactionImpl.commit( >> TransactionImpl.java:122) >> >> at org.neo4j.kernel.TopLevelTransaction.close( >> TopLevelTransaction.java:124) >> >> at endTransaction(...) >> >> ... 31 more >> >> Caused by: javax.transaction.xa.XAException >> >> at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit( >> TransactionImpl.java:553) >> >> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:460) >> >> ... 38 more >> >> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> at org.apache.lucene.util.BytesRef.<init>(BytesRef.java:77) >> >> at org.apache.lucene.store.DataOutput.writeString(DataOutput.java:111) >> >> at org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:212) >> >> at org.apache.lucene.index.StoredFieldsWriterPerThread.addField( >> StoredFieldsWriterPerThread.java:58) >> >> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument( >> DocFieldProcessorPerThread.java:265) >> >> at org.apache.lucene.index.DocumentsWriter.updateDocument( >> DocumentsWriter.java:766) >> >> at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060) >> >> at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2034) >> >> at org.neo4j.kernel.api.impl.index.LuceneIndexAccessor.add( >> LuceneIndexAccessor.java:151) >> >> at >> org.neo4j.kernel.api.impl.index.LuceneIndexAccessor$LuceneIndexUpdater.process( >> LuceneIndexAccessor.java:186) >> >> at >> org.neo4j.kernel.impl.api.index.FlippableIndexProxy$LockingIndexUpdater.process( >> FlippableIndexProxy.java:337) >> >> at org.neo4j.kernel.impl.api.index.ContractCheckingIndexProxy$1.process( >> ContractCheckingIndexProxy.java:102) >> >> at >> org.neo4j.kernel.impl.api.index.IndexingService.processUpdateIfIndexExists( >> IndexingService.java:411) >> >> at org.neo4j.kernel.impl.api.index.IndexingService.applyUpdates( >> IndexingService.java:359) >> >> at org.neo4j.kernel.impl.api.index.IndexingService.updateIndexes( >> IndexingService.java:310) >> >> at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.applyCommit( >> WriteTransaction.java:817) >> >> at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.doCommit( >> WriteTransaction.java:751) >> >> at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit( >> XaTransaction.java:322) >> >> at >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commitWriteTx( >> XaResourceManager.java:530) >> >> at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit( >> XaResourceManager.java:446) >> >> at >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit( >> XaResourceHelpImpl.java:64) >> >> at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit( >> TransactionImpl.java:545) >> >> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:460) >> >> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:397) >> >> at org.neo4j.kernel.impl.transaction.TransactionImpl.commit( >> TransactionImpl.java:122) >> >> at org.neo4j.kernel.TopLevelTransaction.close( >> TopLevelTransaction.java:124) >> >> -> endTransaction(...) >> >> >> On Mon, Jul 14, 2014 at 7:18 AM, José Cornado <[email protected]> >> wrote: >> >>> It is crashing before it gets where the original exception occurred. >>> >>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>> >>> at >>> org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement( >>> KernelTransactionImplementation.java:182) >>> >>> at >>> org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement( >>> KernelTransactionImplementation.java:63) >>> >>> at org.neo4j.kernel.impl.core.ThreadToStatementContextBridge.instance( >>> ThreadToStatementContextBridge.java:47) >>> >>> at org.neo4j.kernel.impl.core.NodeProxy.addLabel(NodeProxy.java:468) >>> >>> at MY LOGIC >>> >>> at org.eclipse.swt.widgets.TypedListener.handleEvent( >>> TypedListener.java:220) >>> >>> at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84) >>> >>> at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4166) >>> >>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1466) >>> >>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1489) >>> >>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1474) >>> >>> at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1279) >>> >>> at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4012) >>> >>> at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3651) >>> >>> at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run( >>> PartRenderingEngine.java:1113) >>> >>> at org.eclipse.core.databinding.observable.Realm.runWithDefault( >>> Realm.java:332) >>> >>> at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run( >>> PartRenderingEngine.java:997) >>> >>> at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI( >>> E4Workbench.java:138) >>> >>> at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:610) >>> >>> at org.eclipse.core.databinding.observable.Realm.runWithDefault( >>> Realm.java:332) >>> >>> at org.eclipse.ui.internal.Workbench.createAndRunWorkbench( >>> Workbench.java:567) >>> >>> at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150) >>> >>> at org.eclipse.ui.internal.ide.application.IDEApplication.start( >>> IDEApplication.java:124) >>> >>> at org.eclipse.equinox.internal.app.EclipseAppHandle.run( >>> EclipseAppHandle.java:196) >>> >>> at >>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication( >>> EclipseAppLauncher.java:110) >>> >>> at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start( >>> EclipseAppLauncher.java:79) >>> >>> at org.eclipse.core.runtime.adaptor.EclipseStarter.run( >>> EclipseStarter.java:354) >>> >>> >>> On Mon, Jul 14, 2014 at 6:52 AM, José Cornado <[email protected]> >>> wrote: >>> >>>> Let .me reproduce it again. I was able to wrap 29,000 in a transaction >>>> (side note) >>>> >>>> >>>> On Mon, Jul 14, 2014 at 12:19 AM, Michael Hunger < >>>> [email protected]> wrote: >>>> >>>>> Can you share your code and the full exception? As well as the >>>>> graph.db/messages.log file >>>>> >>>>> Thx >>>>> >>>>> Sent from mobile device >>>>> >>>>> Am 13.07.2014 um 19:34 schrieb José Cornado <[email protected]>: >>>>> >>>>> Hello! >>>>> >>>>> I am running int the following: >>>>> >>>>> After inserting around 30,000 nodes to a graph, I close the operation >>>>> with transaction success. But the neo returns: >>>>> >>>>> Unable to commit transaction. >>>>> >>>>> Is there a hard limit on the size of a transaction? Doing it on a per >>>>> node basis is too slow. >>>>> >>>>> Thanks a lot!!! >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Neo4j" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Neo4j" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> José Cornado >>>> >>>> -- >>>> >>>> home: http://www.efekctive.com >>>> blog: http://blogging.efekctive.com >>>> ---------------------- >>>> >>>> Everything has been said before, but since nobody listens we have to >>>> keep going back and beginning all over again. >>>> >>>> Andre Gide >>>> >>> >>> >>> >>> -- >>> José Cornado >>> >>> -- >>> >>> home: http://www.efekctive.com >>> blog: http://blogging.efekctive.com >>> ---------------------- >>> >>> Everything has been said before, but since nobody listens we have to >>> keep going back and beginning all over again. >>> >>> Andre Gide >>> >> >> >> >> -- >> José Cornado >> >> -- >> >> home: http://www.efekctive.com >> blog: http://blogging.efekctive.com >> ---------------------- >> >> Everything has been said before, but since nobody listens we have to keep >> going back and beginning all over again. >> >> Andre Gide >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> <messages.log> >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Neo4j" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > José Cornado > > -- > > home: http://www.efekctive.com > blog: http://blogging.efekctive.com > ---------------------- > > Everything has been said before, but since nobody listens we have to keep > going back and beginning all over again. > > Andre Gide > -- José Cornado -- home: http://www.efekctive.com blog: http://blogging.efekctive.com ---------------------- Everything has been said before, but since nobody listens we have to keep going back and beginning all over again. Andre Gide -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
