I am not importing. I am creating. These were very simple (one value == one node) It can be more complex than that. Where can I find docs about the memory usage and setup?
On Mon, Jul 14, 2014 at 9:35 AM, Michael Hunger < [email protected]> wrote: > In my experience a it depends on what you do, I had good results with > tx-sizes from 1k to 30k > > But if you need to import millions at once, you might want to look into > batch-insertion, see http://neo4j.org/develop/import > > Cheers, > > Michael > > Am 14.07.2014 um 17:32 schrieb José Cornado <[email protected]>: > > The data is fed to an engine that is able to process work in the millions. > The number of relations is really low compared to the number of nodes so > the cost of node per transaction is the driver. > > Thanks! > > > On Mon, Jul 14, 2014 at 9:27 AM, José Cornado <[email protected]> > wrote: > >> that I know. What is the cost per element (node/rels) in a transaction? >> few kb? >> >> >> On Mon, Jul 14, 2014 at 9:24 AM, Michael Hunger < >> [email protected]> wrote: >> >>> Make sure your tx is not too big for your memory, according to your >>> messages log your JVM only runs with a few megabytes of memory. >>> >>> Try to limit your tx-size to 10k or 20k elements (nodes and rels) >>> restarting the tx after calling >>> >>> if (count++ == 10000) { >>> tx.success(); tx.close(); >>> tx = db.beginTx(); >>> count = 0; >>> } >>> >>> Cheers, >>> >>> Michael >>> >>> Am 14.07.2014 um 17:21 schrieb José Cornado <[email protected]>: >>> >>> I managed to reproduce the same exception. >>> >>> Trace and log file are included. I will put together a test case in a few >>> >>> >>> (org.neo4j.graphdb.TransactionFailureException) >>> org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction >>> >>> >>> Caused by: org.neo4j.graphdb.TransactionFailureException: commit threw >>> exception >>> >>> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:498 >>> ) >>> >>> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:397 >>> ) >>> >>> at org.neo4j.kernel.impl.transaction.TransactionImpl.commit( >>> TransactionImpl.java:122) >>> >>> at org.neo4j.kernel.TopLevelTransaction.close( >>> TopLevelTransaction.java:124) >>> >>> at endTransaction(...) >>> >>> ... 31 more >>> >>> Caused by: javax.transaction.xa.XAException >>> >>> at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit( >>> TransactionImpl.java:553) >>> >>> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:460 >>> ) >>> >>> ... 38 more >>> >>> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded >>> >>> at org.apache.lucene.util.BytesRef.<init>(BytesRef.java:77) >>> >>> at org.apache.lucene.store.DataOutput.writeString(DataOutput.java:111) >>> >>> at org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:212 >>> ) >>> >>> at org.apache.lucene.index.StoredFieldsWriterPerThread.addField( >>> StoredFieldsWriterPerThread.java:58) >>> >>> at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument( >>> DocFieldProcessorPerThread.java:265) >>> >>> at org.apache.lucene.index.DocumentsWriter.updateDocument( >>> DocumentsWriter.java:766) >>> >>> at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060 >>> ) >>> >>> at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2034 >>> ) >>> >>> at org.neo4j.kernel.api.impl.index.LuceneIndexAccessor.add( >>> LuceneIndexAccessor.java:151) >>> >>> at >>> org.neo4j.kernel.api.impl.index.LuceneIndexAccessor$LuceneIndexUpdater.process( >>> LuceneIndexAccessor.java:186) >>> >>> at >>> org.neo4j.kernel.impl.api.index.FlippableIndexProxy$LockingIndexUpdater.process( >>> FlippableIndexProxy.java:337) >>> >>> at org.neo4j.kernel.impl.api.index.ContractCheckingIndexProxy$1.process( >>> ContractCheckingIndexProxy.java:102) >>> >>> at >>> org.neo4j.kernel.impl.api.index.IndexingService.processUpdateIfIndexExists( >>> IndexingService.java:411) >>> >>> at org.neo4j.kernel.impl.api.index.IndexingService.applyUpdates( >>> IndexingService.java:359) >>> >>> at org.neo4j.kernel.impl.api.index.IndexingService.updateIndexes( >>> IndexingService.java:310) >>> >>> at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.applyCommit( >>> WriteTransaction.java:817) >>> >>> at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.doCommit( >>> WriteTransaction.java:751) >>> >>> at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit( >>> XaTransaction.java:322) >>> >>> at >>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commitWriteTx( >>> XaResourceManager.java:530) >>> >>> at >>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit( >>> XaResourceManager.java:446) >>> >>> at >>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit( >>> XaResourceHelpImpl.java:64) >>> >>> at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit( >>> TransactionImpl.java:545) >>> >>> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:460 >>> ) >>> >>> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:397 >>> ) >>> >>> at org.neo4j.kernel.impl.transaction.TransactionImpl.commit( >>> TransactionImpl.java:122) >>> >>> at org.neo4j.kernel.TopLevelTransaction.close( >>> TopLevelTransaction.java:124) >>> >>> -> endTransaction(...) >>> >>> >>> On Mon, Jul 14, 2014 at 7:18 AM, José Cornado <[email protected]> >>> wrote: >>> >>>> It is crashing before it gets where the original exception occurred. >>>> >>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>> >>>> at >>>> org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement( >>>> KernelTransactionImplementation.java:182) >>>> >>>> at >>>> org.neo4j.kernel.impl.api.KernelTransactionImplementation.acquireStatement( >>>> KernelTransactionImplementation.java:63) >>>> >>>> at org.neo4j.kernel.impl.core.ThreadToStatementContextBridge.instance( >>>> ThreadToStatementContextBridge.java:47) >>>> >>>> at org.neo4j.kernel.impl.core.NodeProxy.addLabel(NodeProxy.java:468) >>>> >>>> at MY LOGIC >>>> >>>> at org.eclipse.swt.widgets.TypedListener.handleEvent( >>>> TypedListener.java:220) >>>> >>>> at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84) >>>> >>>> at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4166) >>>> >>>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1466) >>>> >>>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1489) >>>> >>>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1474) >>>> >>>> at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1279) >>>> >>>> at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4012) >>>> >>>> at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3651) >>>> >>>> at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run( >>>> PartRenderingEngine.java:1113) >>>> >>>> at org.eclipse.core.databinding.observable.Realm.runWithDefault( >>>> Realm.java:332) >>>> >>>> at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run( >>>> PartRenderingEngine.java:997) >>>> >>>> at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI( >>>> E4Workbench.java:138) >>>> >>>> at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:610) >>>> >>>> at org.eclipse.core.databinding.observable.Realm.runWithDefault( >>>> Realm.java:332) >>>> >>>> at org.eclipse.ui.internal.Workbench.createAndRunWorkbench( >>>> Workbench.java:567) >>>> >>>> at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150) >>>> >>>> at org.eclipse.ui.internal.ide.application.IDEApplication.start( >>>> IDEApplication.java:124) >>>> >>>> at org.eclipse.equinox.internal.app.EclipseAppHandle.run( >>>> EclipseAppHandle.java:196) >>>> >>>> at >>>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication( >>>> EclipseAppLauncher.java:110) >>>> >>>> at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start( >>>> EclipseAppLauncher.java:79) >>>> >>>> at org.eclipse.core.runtime.adaptor.EclipseStarter.run( >>>> EclipseStarter.java:354) >>>> >>>> >>>> On Mon, Jul 14, 2014 at 6:52 AM, José Cornado <[email protected]> >>>> wrote: >>>> >>>>> Let .me reproduce it again. I was able to wrap 29,000 in a transaction >>>>> (side note) >>>>> >>>>> >>>>> On Mon, Jul 14, 2014 at 12:19 AM, Michael Hunger < >>>>> [email protected]> wrote: >>>>> >>>>>> Can you share your code and the full exception? As well as the >>>>>> graph.db/messages.log file >>>>>> >>>>>> Thx >>>>>> >>>>>> Sent from mobile device >>>>>> >>>>>> Am 13.07.2014 um 19:34 schrieb José Cornado <[email protected]>: >>>>>> >>>>>> Hello! >>>>>> >>>>>> I am running int the following: >>>>>> >>>>>> After inserting around 30,000 nodes to a graph, I close the operation >>>>>> with transaction success. But the neo returns: >>>>>> >>>>>> Unable to commit transaction. >>>>>> >>>>>> Is there a hard limit on the size of a transaction? Doing it on a per >>>>>> node basis is too slow. >>>>>> >>>>>> Thanks a lot!!! >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Neo4j" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "Neo4j" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> José Cornado >>>>> >>>>> -- >>>>> >>>>> home: http://www.efekctive.com >>>>> blog: http://blogging.efekctive.com >>>>> ---------------------- >>>>> >>>>> Everything has been said before, but since nobody listens we have to >>>>> keep going back and beginning all over again. >>>>> >>>>> Andre Gide >>>>> >>>> >>>> >>>> >>>> -- >>>> José Cornado >>>> >>>> -- >>>> >>>> home: http://www.efekctive.com >>>> blog: http://blogging.efekctive.com >>>> ---------------------- >>>> >>>> Everything has been said before, but since nobody listens we have to >>>> keep going back and beginning all over again. >>>> >>>> Andre Gide >>>> >>> >>> >>> >>> -- >>> José Cornado >>> >>> -- >>> >>> home: http://www.efekctive.com >>> blog: http://blogging.efekctive.com >>> ---------------------- >>> >>> Everything has been said before, but since nobody listens we have to >>> keep going back and beginning all over again. >>> >>> Andre Gide >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Neo4j" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> <messages.log> >>> >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Neo4j" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> José Cornado >> >> -- >> >> home: http://www.efekctive.com >> blog: http://blogging.efekctive.com >> ---------------------- >> >> Everything has been said before, but since nobody listens we have to keep >> going back and beginning all over again. >> >> Andre Gide >> > > > > -- > José Cornado > > -- > > home: http://www.efekctive.com > blog: http://blogging.efekctive.com > ---------------------- > > Everything has been said before, but since nobody listens we have to keep > going back and beginning all over again. > > Andre Gide > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Neo4j" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/neo4j/ABfjS4yZeJM/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- José Cornado -- home: http://www.efekctive.com blog: http://blogging.efekctive.com ---------------------- Everything has been said before, but since nobody listens we have to keep going back and beginning all over again. Andre Gide -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
