Filter in cypher what you return, and yes only return the props needed
For the batching you can also send in a list of maps on which you can store
additional props or information.
For concurrent execution neither guarantees uniqueness of rels but you execute
one after another, so you have a guarantee with merge and create unique
Where did you find that answer on SO?
Von meinem iPhone gesendet
> Am 13.05.2016 um 13:31 schrieb Cherie Pun <cherie.cy....@gmail.com>:
>
> Hi,
>
> Thanks for the quick reply. I returned the relationship because the property
> of the transitive relationship that I am trying to create depends on the
> property of the original two relationship. Should I have just returned the
> properties and node id to reduce data? There are two different types of
> transitive relationship I want to create: 1) assign the property of bc to ac
> 2) assign the product of the trust strength of ab and bc to ac.
>
> The tx-handling code was written according to answers from stack overflow as
> I don't have much knowledge in batch executing cypher queries. Thanks for
> correcting it.
>
> I have read that merge does not guarantee uniqueness of relationship, so when
> should I use create unique and when I should I use merge? Thanks.
>
> Kind regards,
> Cherie
>
> On 13 May, 2016 12:16 pm, "'Michael Hunger' via Neo4j"
> <neo4j@googlegroups.com> wrote:
>
> Your tx-handling code looks wrong.
>
> You should execute the whole batch of creating the connections in the other
> thread, Just grabbing the transaction there is very dangerous as the tx will
> be used that's attached to the current thread instead.
>
> I also recommend batching in cypher, that makes it much easier:
> Just send in a batch (e.g. 100k) list of pairs of id's as parameters and then
> do
>
> pool.submit( () ->
> Iterables.<Long>singleOrNull(db.execute("
> UNWIND {data} as pair
> MATCH (a),(b) WHERE id(a) = pair[0] and id(b) = pair[1]
> MERGE (a)-[:TRUST]->(b) RETURN count(*) as c;
> ", batchOfIds()).columnAs("c"));
> )).get()
>
> If your return ab, bc then that's relationships? but in the other query you
> match nodes?
>
> You should only return id(a),id(c) to minimize the data created.
>
>
> > Am 13.05.2016 um 12:49 schrieb Cherie Pun <cherie.cy....@gmail.com>:
> >
> > Hi,
> >
> > Thanks for your reply. By actual code do you mean all my files that are
> > related? I have attached the main part of the code that does the querying
> > underneath the stacktrace. The other files just defined strings for the
> > property names and enum type for relationship and node.
> >
> > The query that I ran first was :
> > private final String FIND_TRANSITIVE_RELATION_QUERY =
> > "MATCH (a:" + TwitterLabels.USERS + ") " +
> > "- [ab:" + TwitterRelationships.TRUST + "] -> " +
> > "(b:" + TwitterLabels.USERS + ") " +
> > "- [bc:" + TwitterRelationships.TRUST + "] -> " +
> > "(c:" + TwitterLabels.USERS + ")\n" +
> > "WHERE a <> c\n" +
> > "return ab, bc";
> > Iterating through the result from the query, I then perform a another query
> > that create a unique trust relationship between node a and c:
> > private final String CREATE_UNIQUE_TRUST_RELATION_QUERY =
> > "MATCH (a:" + TwitterLabels.USERS + "),(b:" + TwitterLabels.USERS +
> > ")\n" +
> > "WHERE id(a) = {startNode} AND id(b) = {endNode} \n" +
> > "CREATE UNIQUE (a) - [r:" + TwitterRelationships.TRUST + "]
> > -> (b) \n" +
> > "return r";
> >
> > The second query is being executed on another thread, and the thread
> > commits the transaction for every 1000 queries.
> >
> > This executes fine until it processed 200,000 result and crashes because of
> > gc limit reached. I did a heap dump and found out that the
> > relationshipproxy objects were taking up most of the space.
> >
> > I hope that gives you sufficient information, please let me know if you
> > need more explanation of my code.
> >
> > Kind regards,
> > Cherie
> >
> >
> > Can you share your actual code and queries
> >
> > Von meinem iPhone gesendet
> >
> > Am 12.05.2016 um 18:52 schrieb Cherie Pun <cherie.cy....@gmail.com>:
> >
> > > I am trying to iterate through the results from the first query and
> > > execute an extra cypher query for each result to either create or find
> > > the existing relationship. However I encountered GC overhead error and
> > > the stack trace is as followed. I thought running the queries in batch of
> > > 100 will solve the memory problem but it seems that either it's using up
> > > the memory too quickly or not releasing the memory quick enough. It broke
> > > down after processing 200000 queries after I increased the batch size to
> > > 1000. Is there a way to optimise my query to reduce the memory used, or
> > > is there an alternative way to achieve the same result? Thanks.
> > >
> > > Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> > > exceeded
> > > at
> > > scala.collection.mutable.OpenHashMap.foreachUndeletedEntry(OpenHashMap.scala:226)
> > > at scala.collection.mutable.OpenHashMap.foreach(OpenHashMap.scala:219)
> > > at
> > > org.neo4j.cypher.internal.compiler.v2_3.ExecutionContext.foreach(ExecutionContext.scala:46)
> > > at
> > > scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:155)
> > > at
> > > org.neo4j.cypher.internal.compiler.v2_3.ExecutionContext.foldLeft(ExecutionContext.scala:33)
> > > at
> > > org.neo4j.cypher.internal.frontend.v2_3.helpers.Eagerly$.mapToBuilder(Eagerly.scala:44)
> > > at
> > > org.neo4j.cypher.internal.frontend.v2_3.helpers.Eagerly$.immutableMapValues(Eagerly.scala:37)
> > > at
> > > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator$$anonfun$next$1.apply(ResultIterator.scala:76)
> > > at
> > > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator$$anonfun$next$1.apply(ResultIterator.scala:72)
> > > at
> > > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator$$anonfun$failIfThrows$1.apply(ResultIterator.scala:121)
> > > at
> > > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator.decoratedCypherException(ResultIterator.scala:130)
> > > at
> > > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator.failIfThrows(ResultIterator.scala:119)
> > > at
> > > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator.next(ResultIterator.scala:72)
> > > at
> > > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator.next(ResultIterator.scala:50)
> > > at
> > > org.neo4j.cypher.internal.compiler.v2_3.PipeExecutionResult.next(PipeExecutionResult.scala:77)
> > > at
> > > org.neo4j.cypher.internal.compiler.v2_3.PipeExecutionResult$$anon$2.next(PipeExecutionResult.scala:70)
> > > at
> > > org.neo4j.cypher.internal.compiler.v2_3.PipeExecutionResult$$anon$2.next(PipeExecutionResult.scala:68)
> > > at
> > > org.neo4j.cypher.internal.compatibility.ExecutionResultWrapperFor2_3$$anon$1$$anonfun$next$1.apply(CompatibilityFor2_3.scala:234)
> > > at
> > > org.neo4j.cypher.internal.compatibility.ExecutionResultWrapperFor2_3$$anon$1$$anonfun$next$1.apply(CompatibilityFor2_3.scala:234)
> > > at
> > > org.neo4j.cypher.internal.compatibility.exceptionHandlerFor2_3$.runSafely(CompatibilityFor2_3.scala:116)
> > > at
> > > org.neo4j.cypher.internal.compatibility.ExecutionResultWrapperFor2_3$$anon$1.next(CompatibilityFor2_3.scala:234)
> > > at
> > > org.neo4j.cypher.internal.compatibility.ExecutionResultWrapperFor2_3$$anon$1.next(CompatibilityFor2_3.scala:229)
> > > at
> > > org.neo4j.cypher.javacompat.ExecutionResult.next(ExecutionResult.java:233)
> > > at
> > > org.neo4j.cypher.javacompat.ExecutionResult.next(ExecutionResult.java:55)
> > > at java.util.Iterator.forEachRemaining(Iterator.java:116)
> > > at
> > > Neo4j.TrustCalculator.iterateAndExecuteBatchedInSeparateThread(TrustCalculator.java:239)
> > > at
> > > Neo4j.TrustCalculator.insertSimpleTransitiveTrust(TrustCalculator.java:127)
> > > at Neo4j.TrustCalculator.main(TrustCalculator.java:265)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:483)
> > >
> > > private final String CREATE_UNIQUE_TRUST_RELATION_QUERY =
> > > "MATCH (a:" + TwitterLabels.USERS + "),(b:" + TwitterLabels.USERS
> > > + ")\n" +
> > > "WHERE id(a) = {startNode} AND id(b) = {endNode} \n" +
> > > "CREATE UNIQUE (a) - [r:" + TwitterRelationships.TRUST +
> > > "] -> (b) \n" +
> > > "return r";
> > >
> > > private final String FIND_TRANSITIVE_RELATION_QUERY =
> > > "MATCH (a:" + TwitterLabels.USERS + ") " +
> > > "- [ab:" + TwitterRelationships.TRUST + "] -> " +
> > > "(b:" + TwitterLabels.USERS + ") " +
> > > "- [bc:" + TwitterRelationships.TRUST + "] -> " +
> > > "(c:" + TwitterLabels.USERS + ")\n" +
> > > "WHERE a <> c\n" +
> > > "return ab, bc";
> > >
> > > private ExecutorService createSinglePool() {
> > > return Executors.newSingleThreadExecutor();
> > > }
> > >
> > > private void iterateAndExecuteBatchedInSeparateThread(int batchsize,
> > > Iterator iterator, Consumer consumer) {
> > > final int[] opsCount = {0};
> > > final int[] batchesCount = {0};
> > > ExecutorService executorService = createSinglePool();
> > > try {
> > > final Transaction[] workerTransaction =
> > > {executorService.submit(() -> graphDb.beginTx()).get()};
> > >
> > > iterator.forEachRemaining(t -> executorService.submit(() -> {
> > > consumer.accept(t);
> > > if ((++opsCount[0]) % batchsize == 0) {
> > > batchesCount[0]++;
> > > workerTransaction[0].success();
> > > workerTransaction[0].close();
> > > workerTransaction[0] = graphDb.beginTx();
> > > }
> > > }));
> > > executorService.submit(() -> {
> > > workerTransaction[0].success();
> > > workerTransaction[0].close();
> > > }).get();
> > >
> > > } catch (InterruptedException | ExecutionException e) {
> > > throw new RuntimeException(e);
> > > }
> > > }
> > >
> > >
> > > public void insertSimpleTransitiveTrust(){
> > > try ( Transaction tx = graphDb.beginTx() )
> > > {
> > > Result results = graphDb.execute( FIND_TRANSITIVE_RELATION_QUERY,
> > > new HashMap<String, Object>() );
> > >
> > > iterateAndExecuteBatchedInSeparateThread(1000, results, result->
> > > insertOneSimpleTransitiveTrust((Map<String, Object>) result));
> > >
> > > tx.success();
> > > }
> > > }
> > >
> > > private void insertOneSimpleTransitiveTrust(Map<String, Object> map){
> > > Object object = map.get("ab");
> > > Relationship trustAb = (Relationship) object;
> > > object = map.get("bc");
> > > Relationship trustBc = (Relationship) object;
> > > if(trustAb.hasProperty(TwitterProperties.CONVERSATIONAL_TRUST) &&
> > > trustBc.hasProperty(TwitterProperties.CONVERSATIONAL_TRUST)){
> > > Node nodeA = trustAb.getStartNode();
> > > Node nodeB = trustAb.getEndNode();
> > > Node nodeC = trustBc.getEndNode();
> > >
> > > Map<String, Object> params = new HashMap<String, Object>();
> > > params.put("startNode", nodeA.getId());
> > > params.put("endNode", nodeC.getId());
> > > Result results = graphDb.execute(
> > > CREATE_UNIQUE_TRUST_RELATION_QUERY, params );
> > > if(results.hasNext()){
> > > Map<String, Object> result = results.next();
> > > object = result.get("r");
> > > Relationship transitiveTrust = (Relationship) object;
> > >
> > > transitiveTrust.setProperty(TwitterProperties.SIMPLE_TRANSITIVE_TRUST,
> > > trustBc.getProperty(TwitterProperties.CONVERSATIONAL_TRUST));
> > > }
> > > }
> > > }
> > >
> > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "Neo4j" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to neo4j+unsubscr...@googlegroups.com.
> > > For more options, visit https://groups.google.com/d/optout.
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "Neo4j" group.
> > To unsubscribe from this topic, visit
> > https://groups.google.com/d/topic/neo4j/sjFDz496AwA/unsubscribe.
> > To unsubscribe from this group and all its topics, send an email to
> > neo4j+unsubscr...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Neo4j" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to neo4j+unsubscr...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google
> Groups "Neo4j" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/neo4j/sjFDz496AwA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> neo4j+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> --
> You received this message because you are subscribed to the Google Groups
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to neo4j+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
