HI Klemens, sorry I missed your email. Which version are you running?
You should include up to 1000 - 10k nodes in your tx and not do one by one. If you are on 2.2+ you have to set dbms.pagecache.memory=12G instead of the mmio settings. Why do you compute the sum twice? You might even be faster separating the compute and update. I.e. putting the sums into a double array keyed by the node-id and then just having the threads in your pool filling that array and other threads pulling out the values But if its only 10M you might also just use cypher (provided you keep the 16G heap) And batch it into 500k node-updates MATCH (p:Person) WHERE not exists(p.sum) WITH p LIMIT 500000 MATCH (p)-[POINTSREL]->(point) WITH p, sum(point.points) as sum SET p.sum = points; you can also just use skip + limit to page through the set of people and execute the cypher concurrently. Michael On Fri, Oct 16, 2015 at 8:41 AM, Klemens Engelbrechtsmüller < [email protected]> wrote: > Hello! I use the neo4j java core api and want to update 10 million nodes. > I thought it will be better to do it with multithreading but the > performance is not that good (35 minutes for setting properties). > > To explain: Each node "Person" has at least one relation "POINTSREL" to a > "Point" node, which has the property "Points". I want to sum up the points > from the "Point" node and set it as property to the "Person" node. > > > Here is my code: > > > Transaction transaction = service.beginTx();ResourceIterator<Node> iterator = > service.findNodes(Labels.person); > transaction.success(); > transaction.close(); > ExecutorService executor = Executors.newFixedThreadPool(5); > while(iterator.hasNext()){ > executor.execute(new MyJob(iterator.next()));} > //wait until all threads are done > executor.shutdown(); > try { > executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);} catch > (InterruptedException e) { > e.printStackTrace();} > > > And here the runnable class > > > private class MyJob implements Runnable { > > private Node node; > > /* collect useful parameters in the constructor */ > public MyJob(Node node) { > this.node = node; > } > > public void run() { > Transaction transaction = service.beginTx(); > Iterable<org.neo4j.graphdb.Relationship> rel = > this.node.getRelationships(RelationType.POINTSREL, Direction.OUTGOING); > > double sum = 0; > for(org.neo4j.graphdb.Relationship entry : rel){ > try{ > sum += (Double)entry.getEndNode().getProperty("Points"); > } catch(Exception e){ > e.printStackTrace(); > } > } > this.node. double sum = 0; for(org.neo4j.graphdb.Relationship entry : > rel){ try{ sum += (Double)entry.getEndNode().getProperty("Points"); } > catch(Exception e){ e.printStackTrace(); } } this.node.setProperty("Sum", > sum); > > transaction.success(); > transaction.close(); > }} > > > Is there a better (faster) way to do that? > > > About my setting: AWS Instance with 8 CPUs and 32GB ram > > > neo4j-wrapper.conf > > > # Java Heap Size: by default the Java heap size is dynamically# calculated > based on available system resources.# Uncomment these lines to set specific > initial and maximum# heap size in MB. > wrapper.java.initmemory=16000 > wrapper.java.maxmemory=16000 > > > neo4j.properties > > > # The type of cache to use for nodes and relationships. > cache_type=soft > cache.memory_ratio=30.0 > neostore.nodestore.db.mapped_memory=2G > neostore.relationshipstore.db.mapped_memory=7G > neostore.propertystore.db.mapped_memory=2G > neostore.propertystore.db.strings.mapped_memory=2G > neostore.propertystore.db.arrays.mapped_memory=512M > > > > Do you have any ideas? Thank you > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
