It also seems like you are updating a single node per transaction. That's not efficient. You can do thousands of updates in a single transaction.
Sent from my iPhone > On Nov 5, 2015, at 21:34, Michael Hunger <[email protected]> > wrote: > > Also Klemens, > > it is a bit confusing that you share Java code but refer to server config? > > Which Neo4j version do you use? The config seem to be for 2.1.x > For 2.2.x there is dbms.pagecache.memory=8G > I'd use cache_type=none > > Where and how do you run this code? > > Usually neo4j can update stuff in a multi-threaded-way really quickly. > > I can update 10M nodes in 20 seconds with 24 threads on a 6 core machine. > > You probably also want to batch it up a bit, i.e. doing 1000 to 10k nodes per > transaction and job. > > If you have 8 cores you should use a threadpool of size 16 or 32 > > Michael > > >> Am 16.10.2015 um 08:41 schrieb Klemens Engelbrechtsmüller >> <[email protected]>: >> >> Hello! I use the neo4j java core api and want to update 10 million nodes. I >> thought it will be better to do it with multithreading but the performance >> is not that good (35 minutes for setting properties). >> >> To explain: Each node "Person" has at least one relation "POINTSREL" to a >> "Point" node, which has the property "Points". I want to sum up the points >> from the "Point" node and set it as property to the "Person" node. >> >> >> >> Here is my code: >> >> >> >> Transaction transaction = service.beginTx(); >> ResourceIterator<Node> iterator = service.findNodes(Labels.person); >> transaction.success(); >> transaction.close(); >> >> ExecutorService executor = Executors.newFixedThreadPool(5); >> >> while(iterator.hasNext()){ >> executor.execute(new MyJob(iterator.next())); >> } >> >> //wait until all threads are done >> executor.shutdown(); >> >> try { >> executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS); >> } catch (InterruptedException e) { >> e.printStackTrace(); >> } >> >> >> And here the runnable class >> >> >> >> private class MyJob implements Runnable { >> >> private Node node; >> >> /* collect useful parameters in the constructor */ >> public MyJob(Node node) { >> this.node = node; >> } >> >> public void run() { >> Transaction transaction = service.beginTx(); >> Iterable<org.neo4j.graphdb.Relationship> rel = >> this.node.getRelationships(RelationType.POINTSREL, Direction.OUTGOING); >> >> double sum = 0; >> for(org.neo4j.graphdb.Relationship entry : rel){ >> try{ >> sum += (Double)entry.getEndNode().getProperty("Points"); >> } catch(Exception e){ >> e.printStackTrace(); >> } >> } >> this.node. double sum = 0; for(org.neo4j.graphdb.Relationship entry >> : rel){ try{ sum += (Double)entry.getEndNode().getProperty("Points"); } >> catch(Exception e){ e.printStackTrace(); } } this.node.setProperty("Sum", >> sum); >> >> transaction.success(); >> transaction.close(); >> } >> } >> >> >> Is there a better (faster) way to do that? >> >> >> >> About my setting: AWS Instance with 8 CPUs and 32GB ram >> >> >> >> neo4j-wrapper.conf >> >> >> >> # Java Heap Size: by default the Java heap size is dynamically >> # calculated based on available system resources. >> # Uncomment these lines to set specific initial and maximum >> # heap size in MB. >> wrapper.java.initmemory=16000 >> wrapper.java.maxmemory=16000 >> >> >> neo4j.properties >> >> >> >> # The type of cache to use for nodes and relationships. >> cache_type=soft >> cache.memory_ratio=30.0 >> neostore.nodestore.db.mapped_memory=2G >> neostore.relationshipstore.db.mapped_memory=7G >> neostore.propertystore.db.mapped_memory=2G >> neostore.propertystore.db.strings.mapped_memory=2G >> neostore.propertystore.db.arrays.mapped_memory=512M >> >> >> Do you have any ideas? Thank you >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
