Hi Michael, That's what I thought but I didn't want to assume, any further tuning you think I should do with hpc or just leave the defaults? Query below:
neo4j-sh (?)$ MATCH (n) where id(n) = 462370 return n; +---+ | n | +---+ +---+ 0 row 387 ms No stack traces in messages.log (already looked, should have mentioned that), just the startup information. On Tuesday, December 17, 2013 9:16:51 AM UTC, Michael Hunger wrote: > > Sorry, my bad. The gcr cache was renamed to hpc in 2.0 so use > cache_type=hpc > > Can you check graph.db/messages.log or data/logs/* for an exception > related to the missing node? > > EntityNotFoundException: Node with id 462370 > > > can you also try this for me? > > MATCH (n) where id(n) = 462370 return n > > Thanks > > Michael > > Am 17.12.2013 um 10:11 schrieb JDS <[email protected] <javascript:>>: > > Hello Michael, > > Thanks for the advice. I've upgraded to 2.0.0 enterprise and set all the > settings: > > neostore.nodestore.db.mapped_memory=50M > neostore.relationshipstore.db.mapped_memory=756M > neostore.propertystore.db.mapped_memory=100M > neostore.propertystore.db.strings.mapped_memory=324M > neostore.propertystore.db.arrays.mapped_memory=50M > cache_type=gcr > > wrapper.java.initmemory=1536 > wrapper.java.maxmemory=3072 > > I have 1 new question and 1 problem/question: > > 1) Why is gcr not documented at > http://docs.neo4j.org/chunked/stable/configuration-caches.html#_object_cache > 2) I can't run the query as I'm getting an exception with a missing node > of some sort (do I need to rebuild my indexes?) > > neo4j-sh (?)$ PROFILE MATCH > (nt:NHS_TRUST)-[r1:HAS_NHS_TRUST_LOCATION]->(n:NHS_TRUST_LOCATION)<-[r2:IS_NHS_TRUST_LOCATION_INCIDENT]-(i:INCIDENT)-[r3:IS_NHS_TRUST_INCIDENT]->(nt) > > USING SCAN nt:NHS_TRUST USING SCAN n:NHS_TRUST_LOCATION RETURN nt.name AS > nhs_trust, type(r1), n.location_level_01 AS nhs_trust_location, type(r2), > COUNT(i.incident_id) AS incident_count ORDER BY nhs_trust, > nhs_trust_location; > EntityNotFoundException: Node with id 462370 > > On Tuesday, December 17, 2013 8:10:01 AM UTC, Michael Hunger wrote: >> >> Update Neo4j to 2.0 >> there have been huge improvements in that area. >> >> Please report back after that, then we can continue to improve your query. >> >> Still it shouldn't be that slow, you could also increase the rel-store >> mmio to 500M. >> >> And perhaps use cache_type=gcr when you are using Enterprise anyway. >> >> Something that might help a tiny bit as well (test it after the update), >> is to leave off the labels except the first if your relationships already >> make the node-types unique. >> >> Cheers >> >> Michael >> >> Am 17.12.2013 um 08:58 schrieb JDS <[email protected]>: >> >> I'm looking at two queries. They both return the same results, but the >> 1st one took an hour to run (when I tried to run it again recently it >> didn't finish after 8 hours), chomps the cpu, and ups the heap usage, while >> the other took less than 10 seconds and is barely noticeable. Note that I >> am loading incident data while these queries are running. What I'm trying >> to figure out is if there's a way I can improve my relationships, queries, >> or configuration to have the same results without having to use scan (i.e. >> am I missing something obvious) or is it just better to continue using >> scan? Would I also be right in thinking I should increase my >> neostore.relationshipstore.db.mapped_memory >> setting to gain better performance (looking at jconsole it looks like my >> current heap size is never exceeding 2G and is usually closer to about >> 1.7G, so I could tune down my init/max and give more memory to cache as >> well as other processes)? Also, another strange thing, watching jconsole, >> if I run the second query it seems to trigger a massive release of the heap >> if I run it while the first query has been running (which I'm assuming is >> causing the heap usage to increase), see attached screenshot for the >> jconsole capture. I saw a post saying that PROFILE documentation was on >> it's way, is that done now? >> >> Neo4j Instance: Neo4j 2.0.0 M06 Enterprise >> >> Server: VM with 2 x Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz & 8G's RAM >> running 2.6.32-358.18.1.el6.x86_64 >> >> Config Details: >> wrapper.java.initmemory=3584 >> wrapper.java.maxmemory=5632 >> >> neostore.nodestore.db.mapped_memory=50M >> neostore.relationshipstore.db.mapped_memory=100M >> neostore.propertystore.db.mapped_memory=180M >> neostore.propertystore.db.strings.mapped_memory=260M >> neostore.propertystore.db.arrays.mapped_memory=260M >> >> -rw-r--r--. 1 root root 63 Dec 16 20:35 >> /opt/neo4j/data/graph.db/neostore >> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.id >> -rw-r--r--. 1 root root 50 Dec 15 20:50 >> /opt/neo4j/data/graph.db/neostore.labeltokenstore.db >> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.labeltokenstore.db.id >> -rw-r--r--. 1 root root 418 Dec 15 20:50 >> /opt/neo4j/data/graph.db/neostore.labeltokenstore.db.names >> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.labeltokenstore.db.names.id >> -rw-r--r--. 1 root root 10902528 Dec 16 20:35 >> /opt/neo4j/data/graph.db/neostore.nodestore.db >> -rw-r--r--. 1 root root 1681 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.nodestore.db.id >> -rw-r--r--. 1 root root 68 Dec 15 20:50 >> /opt/neo4j/data/graph.db/neostore.nodestore.db.labels >> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.nodestore.db.labels.id >> -rw-r--r--. 1 root root 64591400 Dec 16 19:42 >> /opt/neo4j/data/graph.db/neostore.propertystore.db >> -rw-r--r--. 1 root root 128 Dec 15 20:50 >> /opt/neo4j/data/graph.db/neostore.propertystore.db.arrays >> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.propertystore.db.arrays.id >> -rw-r--r--. 1 root root 1817 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.propertystore.db.id >> -rw-r--r--. 1 root root 153 Dec 15 20:50 >> /opt/neo4j/data/graph.db/neostore.propertystore.db.index >> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.propertystore.db.index.id >> -rw-r--r--. 1 root root 684 Dec 15 20:50 >> /opt/neo4j/data/graph.db/neostore.propertystore.db.index.keys >> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.propertystore.db.index.keys.id >> -rw-r--r--. 1 root root 244170752 Dec 16 19:42 >> /opt/neo4j/data/graph.db/neostore.propertystore.db.strings >> -rw-r--r--. 1 root root 73 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.propertystore.db.strings.id >> -rw-r--r--. 1 root root 524220279 Dec 16 20:35 >> /opt/neo4j/data/graph.db/neostore.relationshipstore.db >> -rw-r--r--. 1 root root 84203681 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.relationshipstore.db.id >> -rw-r--r--. 1 root root 45 Dec 15 20:50 >> /opt/neo4j/data/graph.db/neostore.relationshiptypestore.db >> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.relationshiptypestore.db.id >> -rw-r--r--. 1 root root 380 Dec 15 20:50 >> /opt/neo4j/data/graph.db/neostore.relationshiptypestore.db.names >> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.relationshiptypestore.db.names.id >> -rw-r--r--. 1 root root 1600 Dec 15 20:50 >> /opt/neo4j/data/graph.db/neostore.schemastore.db >> -rw-r--r--. 1 root root 9 Dec 15 20:50 /opt/neo4j/data/graph.db/ >> neostore.schemastore.db.id >> >> Query without USING SCAN with PROFILE: >> >> PROFILE MATCH >> (nt:NHS_TRUST)-[r1:HAS_NHS_TRUST_LOCATION]->(n:NHS_TRUST_LOCATION)<-[r2:IS_NHS_TRUST_LOCATION_INCIDENT]-(i:INCIDENT)-[r3:IS_NHS_TRUST_INCIDENT]->(nt) >> >> RETURN nt.name AS nhs_trust, type(r1), n.location_level_01 AS >> nhs_trust_location, type(r2), COUNT(i.incident_id) AS incident_count ORDER >> BY nhs_trust, nhs_trust_location; >> >> (See attached screenshot for profile because now I can't even get it to >> finish running after 8 hours) >> >> Query with USING SCAN & PROFILE: >> >> PROFILE MATCH >> (nt:NHS_TRUST)-[r1:HAS_NHS_TRUST_LOCATION]->(n:NHS_TRUST_LOCATION)<-[r2:IS_NHS_TRUST_LOCATION_INCIDENT]-(i:INCIDENT)-[r3:IS_NHS_TRUST_INCIDENT]->(nt) >> >> USING SCAN nt:NHS_TRUST USING SCAN n:NHS_TRUST_LOCATION RETURNnt.name AS >> nhs_trust, type(r1), n.location_level_01 AS nhs_trust_location, type(r2), >> COUNT(i.incident_id) AS incident_count ORDER BY nhs_trust, >> nhs_trust_location; >> >> ColumnFilter(symKeys=["type(r1)", "type(r2)", "nhs_trust_location", " >> INTERNAL_AGGREGATEf02dc4cd-b519-4ab3-87a7-0aab228c7373", "nhs_trust"], >> returnItemNames=["nhs_trust", "type(r1)", "nhs_trust_location", "type(r2)", >> "incident_count"], _rows=48, _db_hits=0) >> Sort(descr=["SortItem(Cached(nhs_trust of type Any),true)", >> "SortItem(Cached(nhs_trust_location of type Any),true)"], _rows=48, >> _db_hits=0) >> EagerAggregation(keys=["Cached(nhs_trust of type Any)", >> "Cached(type(r1) of type String)", "Cached(nhs_trust_location of type >> Any)", "Cached(type(r2) of type String)"], aggregates=["( >> >> INTERNAL_AGGREGATEf02dc4cd-b519-4ab3-87a7-0aab228c7373,Count(Product(i,incident_id(5),true)))"], >> >> _rows=48, _db_hits=172287) >> Extract(symKeys=["n", "i", "r1", "r2", "r3", "nt"], >> exprKeys=["nhs_trust", "type(r1)", "nhs_trust_location", "type(r2)"], >> _rows=172287, _db_hits=344574) >> Filter(pred="(hasLabel(i:INCIDENT(3)) AND >> hasLabel(i:INCIDENT(3)))", _rows=172287, _db_hits=0) >> PatternMatch(g="(i)-['r3']-(nt),(i)-['r2']-(n),(nt)-['r1']-(n)", >> _rows=172287, _db_hits=0) >> Filter(pred="(hasLabel(n:NHS_TRUST_LOCATION(2)) AND >> hasLabel(n:NHS_TRUST_LOCATION(2)))", _rows=312, _db_hits=0) >> NodeByLabel(label="NHS_TRUST_LOCATION", identifier="n", >> _rows=312, _db_hits=0) >> Filter(pred="hasLabel(nt:NHS_TRUST(1))", _rows=26, >> _db_hits=0) >> NodeByLabel(label="NHS_TRUST", identifier="nt", _rows=26, >> _db_hits=0) >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> <Screenshot_2013-12-16-14-49-06.png><jconsole.png> >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
