In my experience a it depends on what you do, I had good results with tx-sizes from 1k to 30k
But if you need to import millions at once, you might want to look into batch-insertion, see http://neo4j.org/develop/import Cheers, Michael Am 14.07.2014 um 17:32 schrieb José Cornado <[email protected]>: > The data is fed to an engine that is able to process work in the millions. > The number of relations is really low compared to the number of nodes so the > cost of node per transaction is the driver. > > Thanks! > > > On Mon, Jul 14, 2014 at 9:27 AM, José Cornado <[email protected]> wrote: > that I know. What is the cost per element (node/rels) in a transaction? few > kb? > > > On Mon, Jul 14, 2014 at 9:24 AM, Michael Hunger > <[email protected]> wrote: > Make sure your tx is not too big for your memory, according to your messages > log your JVM only runs with a few megabytes of memory. > > Try to limit your tx-size to 10k or 20k elements (nodes and rels) restarting > the tx after calling > > if (count++ == 10000) { > tx.success(); tx.close(); > tx = db.beginTx(); > count = 0; > } > > Cheers, > > Michael > > Am 14.07.2014 um 17:21 schrieb José Cornado <[email protected]>: > >> I managed to reproduce the same exception. >> >> Trace and log file are included. I will put together a test case in a few >> >> >> >> (org.neo4j.graphdb.TransactionFailureException) >> org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction >> >> >> Caused by: org.neo4j.graphdb.TransactionFailureException: commit threw >> exception >> >> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:498) >> >> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:397) >> >> at >> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:122) >> >> at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:124) >> >> at endTransaction(...) >> >> ... 31 more >> >> Caused by: javax.transaction.xa.XAException >> >> at >> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:553) >> >> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:460) >> >> ... 38 more >> >> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> at org.apache.lucene.util.BytesRef.<init>(BytesRef.java:77) >> >> at org.apache.lucene.store.DataOutput.writeString(DataOutput.java:111) >> >> at org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:212) >> >> at >> org.apache.lucene.index.StoredFieldsWriterPerThread.addField(StoredFieldsWriterPerThread.java:58) >> >> at >> org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:265) >> >> at >> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:766) >> >> at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060) >> >> at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2034) >> >> at >> org.neo4j.kernel.api.impl.index.LuceneIndexAccessor.add(LuceneIndexAccessor.java:151) >> >> at >> org.neo4j.kernel.api.impl.index.LuceneIndexAccessor$LuceneIndexUpdater.process(LuceneIndexAccessor.java:186) >> >> at >> org.neo4j.kernel.impl.api.index.FlippableIndexProxy$LockingIndexUpdater.process(FlippableIndexProxy.java:337) >> >> at >> org.neo4j.kernel.impl.api.index.ContractCheckingIndexProxy$1.process(ContractCheckingIndexProxy.java:102) >> >> at >> org.neo4j.kernel.impl.api.index.IndexingService.processUpdateIfIndexExists(IndexingService.java:411) >> >> at >> org.neo4j.kernel.impl.api.index.IndexingService.applyUpdates(IndexingService.java:359) >> >> at >> org.neo4j.kernel.impl.api.index.IndexingService.updateIndexes(IndexingService.java:310) >> >> at >> org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.applyCommit(WriteTransaction.java:817) >> >> at >> org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.doCommit(WriteTransaction.java:751) >> >> at >> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:322) >> >> at >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commitWriteTx(XaResourceManager.java:530) >> >> at >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:446) >> >> at >> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) >> >> at >> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:545) >> >> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:460) >> >> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:397) >> >> at >> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:122) >> >> >> at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:124) >> >> -> endTransaction(...) >> >> >> >> On Mon, Jul 14, 2014 at 7:18 AM, José Cornado <[email protected]> wrote: >> It is crashing before it gets where the original exception occurred. >> >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> at >> org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement(KernelTransactionImplementation.java:182) >> >> at >> org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement(KernelTransactionImplementation.java:63) >> >> at >> org.neo4j.kernel.impl.core.ThreadToStatementContextBridge.instance(ThreadToStatementContextBridge.java:47) >> >> at org.neo4j.kernel.impl.core.NodeProxy.addLabel(NodeProxy.java:468) >> >> at MY LOGIC >> >> at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:220) >> >> at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84) >> >> at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4166) >> >> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1466) >> >> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1489) >> >> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1474) >> >> at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1279) >> >> at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4012) >> >> at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3651) >> >> at >> org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1113) >> >> at >> org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332) >> >> at >> org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:997) >> >> at >> org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:138) >> >> at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:610) >> >> at >> org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332) >> >> at >> org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:567) >> >> at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150) >> >> at >> org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:124) >> >> at >> org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196) >> >> at >> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110) >> >> at >> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79) >> >> at >> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:354) >> >> >> >> On Mon, Jul 14, 2014 at 6:52 AM, José Cornado <[email protected]> wrote: >> Let .me reproduce it again. I was able to wrap 29,000 in a transaction (side >> note) >> >> >> On Mon, Jul 14, 2014 at 12:19 AM, Michael Hunger >> <[email protected]> wrote: >> Can you share your code and the full exception? As well as the >> graph.db/messages.log file >> >> Thx >> >> Sent from mobile device >> >> Am 13.07.2014 um 19:34 schrieb José Cornado <[email protected]>: >> >>> Hello! >>> >>> I am running int the following: >>> >>> After inserting around 30,000 nodes to a graph, I close the operation with >>> transaction success. But the neo returns: >>> >>> Unable to commit transaction. >>> >>> Is there a hard limit on the size of a transaction? Doing it on a per node >>> basis is too slow. >>> >>> Thanks a lot!!! >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Neo4j" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Neo4j" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> José Cornado >> >> -- >> >> home: http://www.efekctive.com >> blog: http://blogging.efekctive.com >> ---------------------- >> >> Everything has been said before, but since nobody listens we have to keep >> going back and beginning all over again. >> >> Andre Gide >> >> >> >> -- >> José Cornado >> >> -- >> >> home: http://www.efekctive.com >> blog: http://blogging.efekctive.com >> ---------------------- >> >> Everything has been said before, but since nobody listens we have to keep >> going back and beginning all over again. >> >> Andre Gide >> >> >> >> -- >> José Cornado >> >> -- >> >> home: http://www.efekctive.com >> blog: http://blogging.efekctive.com >> ---------------------- >> >> Everything has been said before, but since nobody listens we have to keep >> going back and beginning all over again. >> >> Andre Gide >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> <messages.log> > > > -- > You received this message because you are subscribed to a topic in the Google > Groups "Neo4j" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. > > > > -- > José Cornado > > -- > > home: http://www.efekctive.com > blog: http://blogging.efekctive.com > ---------------------- > > Everything has been said before, but since nobody listens we have to keep > going back and beginning all over again. > > Andre Gide > > > > -- > José Cornado > > -- > > home: http://www.efekctive.com > blog: http://blogging.efekctive.com > ---------------------- > > Everything has been said before, but since nobody listens we have to keep > going back and beginning all over again. > > Andre Gide > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
