It was in the original post ;) 2.0.0-M06 but I'll have a look. On Tuesday, December 17, 2013 9:45:41 AM UTC, Michael Hunger wrote: > > What version did you have before > > Please check the rc1 blog post for manual index cleanup on upgrade > > Sent from mobile device > > Am 17.12.2013 um 10:21 schrieb JDS <[email protected] <javascript:>>: > > Hi Michael, > > That's what I thought but I didn't want to assume, any further tuning you > think I should do with hpc or just leave the defaults? Query below: > > neo4j-sh (?)$ MATCH (n) where id(n) = 462370 return n; > +---+ > | n | > +---+ > +---+ > 0 row > 387 ms > > No stack traces in messages.log (already looked, should have mentioned > that), just the startup information. > > On Tuesday, December 17, 2013 9:16:51 AM UTC, Michael Hunger wrote: >> >> Sorry, my bad. The gcr cache was renamed to hpc in 2.0 so use >> cache_type=hpc >> >> Can you check graph.db/messages.log or data/logs/* for an exception >> related to the missing node? >> >> EntityNotFoundException: Node with id 462370 >> >> >> can you also try this for me? >> >> MATCH (n) where id(n) = 462370 return n >> >> Thanks >> >> Michael >> >> Am 17.12.2013 um 10:11 schrieb JDS <[email protected]>: >> >> Hello Michael, >> >> Thanks for the advice. I've upgraded to 2.0.0 enterprise and set all the >> settings: >> >> neostore.nodestore.db.mapped_memory=50M >> neostore.relationshipstore.db.mapped_memory=756M >> neostore.propertystore.db.mapped_memory=100M >> neostore.propertystore.db.strings.mapped_memory=324M >> neostore.propertystore.db.arrays.mapped_memory=50M >> cache_type=gcr >> >> wrapper.java.initmemory=1536 >> wrapper.java.maxmemory=3072 >> >> I have 1 new question and 1 problem/question: >> >> 1) Why is gcr not documented at >> http://docs.neo4j.org/chunked/stable/configuration-caches.html#_object_cache >> 2) I can't run the query as I'm getting an exception with a missing node >> of some sort (do I need to rebuild my indexes?) >> >> neo4j-sh (?)$ PROFILE MATCH >> (nt:NHS_TRUST)-[r1:HAS_NHS_TRUST_LOCATION]->(n:NHS_TRUST_LOCATION)<-[r2:IS_NHS_TRUST_LOCATION_INCIDENT]-(i:INCIDENT)-[r3:IS_NHS_TRUST_INCIDENT]->(nt) >> >> USING SCAN nt:NHS_TRUST USING SCAN n:NHS_TRUST_LOCATION RETURN nt.nameAS >> nhs_trust, type(r1), n.location_level_01 AS nhs_trust_location, >> type(r2), COUNT(i.incident_id) AS incident_count ORDER BY nhs_trust, >> nhs_trust_location; >> EntityNotFoundException: Node with id 462370 >> >> On Tuesday, December 17, 2013 8:10:01 AM UTC, Michael Hunger wrote: >>> >>> Update Neo4j to 2.0 >>> there have been huge improvements in that area. >>> >>> Please report back after that, then we can continue to improve your >>> query. >>> >>> Still it shouldn't be that slow, you could also increase the rel-store >>> mmio to 500M. >>> >>> And perhaps use cache_type=gcr when you are using Enterprise anyway. >>> >>> Something that might help a tiny bit as well (test it after the update), >>> is to leave off the labels except the first if your relationships already >>> make the node-types unique. >>> >>> Cheers >>> >>> Michael >>> >>> Am 17.12.2013 um 08:58 schrieb JDS <[email protected]>: >>> >>> I'm looking at two queries. They both return the same results, but the >>> 1st one took an hour to run (when I tried to run it again recently it >>> didn't finish after 8 hours), chomps the cpu, and ups the heap usage, while >>> the other took less than 10 seconds and is barely noticeable. Note that I >>> am loading incident data while these queries are running. What I'm trying >>> to figure out is if there's a way I can improve my relationships, queries, >>> or configuration to have the same results without having to use scan (i.e. >>> am I missing something obvious) or is it just better to continue using >>> scan? Would I also be right in thinking I should increase my >>> neostore.relationshipstore.db.mapped_memory >>> setting to gain better performance (looking at jconsole it looks like my >>> current heap size is never exceeding 2G and is usually closer to about >>> 1.7G, so I could tune down my init/max and give more memory to cache as >>> well as other processes)? Also, another strange thing, watching jconsole, >>> if I run the second query it seems to trigger a massive release of the heap >>> if I run it while the first query has been running (which I'm assuming is >>> causing the heap usage to increase), see attached screenshot for the >>> jconsole capture. I saw a post saying that PROFILE documentation was on >>> it's way, is that done now? >>> >>> Neo4j Instance: Neo4j 2.0.0 M06 Enterprise >>> >>> Server: VM with 2 x Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz & 8G's RAM >>> running 2.6.32-358.18.1.el6.x86_64 >>> >>> Config Details: >>> wrapper.java.initmemory=3584 >>> wrapper.java.maxmemory=5632 >>> >>> neostore.nodestore.db.mapped_memory=50M >>> neostore.relationshipstore.db.mapped_memory=100M >>> neostore.propertystore.db.mapped_memory=180M >>> neostore.propertystore.db.strings.mapped_memory=260M >>> neostore.propertystore.db.arrays.mapped_memory=260M >>> >>> -rw-r--r--. 1 root root 63 Dec 16 20:35 >>> /opt/neo4j/data/graph.db/neostore >>> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.id >>> -rw-r--r--. 1 root root 50 Dec 15 20:50 >>> /opt/neo4j/data/graph.db/neostore.labeltokenstore.db >>> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.labeltokenstore.db.id >>> -rw-r--r--. 1 root root 418 Dec 15 20:50 >>> /opt/neo4j/data/graph.db/neostore.labeltokenstore.db.names >>> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.labeltokenstore.db.names.id >>> -rw-r--r--. 1 root root 10902528 Dec 16 20:35 >>> /opt/neo4j/data/graph.db/neostore.nodestore.db >>> -rw-r--r--. 1 root root 1681 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.nodestore.db.id >>> -rw-r--r--. 1 root root 68 Dec 15 20:50 >>> /opt/neo4j/data/graph.db/neostore.nodestore.db.labels >>> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.nodestore.db.labels.id >>> -rw-r--r--. 1 root root 64591400 Dec 16 19:42 >>> /opt/neo4j/data/graph.db/neostore.propertystore.db >>> -rw-r--r--. 1 root root 128 Dec 15 20:50 >>> /opt/neo4j/data/graph.db/neostore.propertystore.db.arrays >>> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.propertystore.db.arrays.id >>> -rw-r--r--. 1 root root 1817 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.propertystore.db.id >>> -rw-r--r--. 1 root root 153 Dec 15 20:50 >>> /opt/neo4j/data/graph.db/neostore.propertystore.db.index >>> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.propertystore.db.index.id >>> -rw-r--r--. 1 root root 684 Dec 15 20:50 >>> /opt/neo4j/data/graph.db/neostore.propertystore.db.index.keys >>> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.propertystore.db.index.keys.id >>> -rw-r--r--. 1 root root 244170752 Dec 16 19:42 >>> /opt/neo4j/data/graph.db/neostore.propertystore.db.strings >>> -rw-r--r--. 1 root root 73 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.propertystore.db.strings.id >>> -rw-r--r--. 1 root root 524220279 Dec 16 20:35 >>> /opt/neo4j/data/graph.db/neostore.relationshipstore.db >>> -rw-r--r--. 1 root root 84203681 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.relationshipstore.db.id >>> -rw-r--r--. 1 root root 45 Dec 15 20:50 >>> /opt/neo4j/data/graph.db/neostore.relationshiptypestore.db >>> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.relationshiptypestore.db.id >>> -rw-r--r--. 1 root root 380 Dec 15 20:50 >>> /opt/neo4j/data/graph.db/neostore.relationshiptypestore.db.names >>> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.relationshiptypestore.db.names.id >>> -rw-r--r--. 1 root root 1600 Dec 15 20:50 >>> /opt/neo4j/data/graph.db/neostore.schemastore.db >>> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >>> neostore.schemastore.db.id >>> >>> Query without USING SCAN with PROFILE: >>> >>> PROFILE MATCH >>> (nt:NHS_TRUST)-[r1:HAS_NHS_TRUST_LOCATION]->(n:NHS_TRUST_LOCATION)<-[r2:IS_NHS_TRUST_LOCATION_INCIDENT]-(i:INCIDENT)-[r3:IS_NHS_TRUST_INCIDENT]->(nt) >>> >>> RETURN nt.name AS nhs_trust, type(r1), n.location_level_01 AS >>> nhs_trust_location, type(r2), COUNT(i.incident_id) AS incident_count ORDER >>> BY nhs_trust, nhs_trust_location; >>> >>> (See attached screenshot for profile because now I can't even get it to >>> finish running after 8 hours) >>> >>> Query with USING SCAN & PROFILE: >>> >>> PROFILE MATCH >>> (nt:NHS_TRUST)-[r1:HAS_NHS_TRUST_LOCATION]->(n:NHS_TRUST_LOCATION)<-[r2:IS_NHS_TRUST_LOCATION_INCIDENT]-(i:INCIDENT)-[r3:IS_NHS_TRUST_INCIDENT]->(nt) >>> >>> USING SCAN nt:NHS_TRUST USING SCAN n:NHS_TRUST_LOCATION RETURNnt.name AS >>> nhs_trust, type(r1), n.location_level_01 AS nhs_trust_location, type(r2), >>> COUNT(i.incident_id) AS incident_count ORDER BY nhs_trust, >>> nhs_trust_location; >>> >>> ColumnFilter(symKeys=["type(r1)", "type(r2)", "nhs_trust_location", " >>> INTERNAL_AGGREGATEf02dc4cd-b519-4ab3-87a7-0aab228c7373", "nhs_trust"], >>> returnItemNames=["nhs_trust", "type(r1)", "nhs_trust_location", "type(r2)", >>> "incident_count"], _rows=48, _db_hits=0) >>> Sort(descr=["SortItem(Cached(nhs_trust of type Any),true)", >>> "SortItem(Cached(nhs_trust_location of type Any),true)"], _rows=48, >>> _db_hits=0) >>> EagerAggregation(keys=["Cached(nhs_trust of type Any)", >>> "Cached(type(r1) of type String)", "Cached(nhs_trust_location of type >>> Any)", "Cached(type(r2) of type String)"], aggregates=["( >>> >>> INTERNAL_AGGREGATEf02dc4cd-b519-4ab3-87a7-0aab228c7373,Count(Product(i,incident_id(5),true)))"], >>> >>> _rows=48, _db_hits=172287) >>> Extract(symKeys=["n", "i", "r1", "r2", "r3", "nt"], >>> exprKeys=["nhs_trust", "type(r1)", "nhs_trust_location", "type(r2)"], >>> _rows=172287, _db_hits=344574) >>> Filter(pred="(hasLabel(i:INCIDENT(3)) AND >>> hasLabel(i:INCIDENT(3)))", _rows=172287, _db_hits=0) >>> PatternMatch(g="(i)-['r3']-(nt),(i)-['r2']-(n),(nt)-['r1']-(n)", >>> _rows=172287, _db_hits=0) >>> Filter(pred="(hasLabel(n:NHS_TRUST_LOCATION(2)) AND >>> hasLabel(n:NHS_TRUST_LOCATION(2)))", _rows=312, _db_hits=0) >>> NodeByLabel(label="NHS_TRUST_LOCATION", identifier="n", >>> _rows=312, _db_hits=0) >>> Filter(pred="hasLabel(nt:NHS_TRUST(1))", _rows=26, >>> _db_hits=0) >>> NodeByLabel(label="NHS_TRUST", identifier="nt", >>> _rows=26, _db_hits=0) >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Neo4j" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> <Screenshot_2013-12-16-14-49-06.png><jconsole.png> >>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/groups/opt_out. > >
-- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
