Set heap to at most 16-24G should be fine, And switch to G1GC with -XX:+UseG1GC instead of CMS.
CMS is not so good for larger heaps. Depending on your store file-sizes (nodes + props + rels) set the pagecache to the sum of values, so in your case 35G Which edition of Neo4j are you using? Michael > Am 16.09.2015 um 15:12 schrieb Singleton Chiu <[email protected]>: > > I'm using the community version, > I have searched about my situation, and turned the gc log on. > And I found that the REST API got stuck during the Full GC. > > 29184.923: [GC 24510626K->23288671K(24643840K), 0.0907810 secs] > 29185.932: [GC 24515487K->23310086K(24643840K), 0.1171710 secs] > 29186.976: [GC 24536867K->23309125K(24643840K), 0.0917860 secs] > 29187.988: [GC 24535807K->23322112K(24643840K), 0.1089120 secs] > 29189.018: [GC 24548925K->23333210K(24643840K), 0.1097290 secs] > 29190.054: [GC 24560025K->23350298K(24643840K), 0.1037740 secs] > 29204.010: [Full GC 24573732K->23319822K(24643840K), 134.0548940 secs] > 29338.275: [GC 23363153K(24643840K), 0.1082740 secs] > 29367.432: [Full GC 24643839K->23328404K(24643840K), 133.3548790 secs] > 29500.901: [GC 23366365K(24643840K), 0.1045400 secs] > 29529.261: [Full GC 24643839K->23336254K(24643840K), 134.2712890 secs] > 29663.733: [GC 23367649K(24643840K), 0.1222970 secs] > 29691.674: [Full GC 24643839K->23343854K(24643840K), 133.6376480 secs] > 29825.511: [GC 23369329K(24643840K), 0.1298200 secs] > 29854.807: [Full GC 24643839K->23351726K(24643840K), 134.3462990 secs] > 29989.364: [GC 23395476K(24643840K), 0.1952480 secs] > 30018.853: [Full GC 24643839K->23359633K(24643840K), 133.7108280 secs] > 30152.764: [GC 23395952K(24643840K), 0.1885450 secs] > 30181.939: [Full GC > > Exactly get stuck at the first Full GC. > > I thought my problem is the jvm setting, > > There are totally 96 GB mem, and my usage scenarios are lots of > node,property,rel write and read, sometimes run a heavy time consuming > cypher. > > I think the heap size is not enough, so I change my setting into: > > wrapper.java.additional=-XX:+UseConcMarkSweepGC > wrapper.java.additional=-XX:+CMSClassUnloadingEnabled > wrapper.java.additional=-XX:-OmitStackTraceInFastThrow > wrapper.java.additional=-XX:hashCode=5 > > wrapper.java.initmemory=50000 > wrapper.java.maxmemory=50000 > > dbms.pagecache.memory=30g > > Total Store Size 31.88 GiB > > Is there a guide to set these settings? > > The server now running for 6 hrs normally, with RES:26g, VIRT:61g > > If it goes Full GC continuously or REST API get stuck again, > I'll come back to you with the packed info. > > > Many Thanks! > > > On Wednesday, September 16, 2015 at 8:44:52 AM UTC+8, Michael Hunger wrote: > It should definitely recover > > do you run neo4j community or enterprise? > > what are your settings? > > perhaps you can pack together > > conf/* > data/log/* > data/graph.db/messsages.log > > for some analytics > > it can also be that the heap memory is fully utilized by the 2nd level cache > so it tries to make room by clearing memory, which can take a second or two > but then should be back > > so what you could also do is to enable gc-logging in neo4j-wrapper.conf and > then also send the gc-logs together with the other information. > > Am 16.09.2015 um 02:37 schrieb Singleton Chiu <[email protected] > <javascript:>>: > > Thanks for you quick response, I have already restart the server, I need to > wait the problem situation, coz I can't re-product it. > > I have checked the py2neo log: > > [2015-09-16 07:27:19] [INFO] [28681:MainThread] [http:310 send()] > GET > http://10.181.205.198:8474/db/data/ <http://10.181.205.198:8474/db/data/> > [2015-09-16 07:27:37] [INFO] [28681:MainThread] [http:306 send()] ~ > Reconnecting (timeout) > > > Normally, it response 200 OK within 1 second. > Not only this REST API, but all encounter time out for hours, not one moment. > > And I'm sure the IP and port are certainly right, coz It have run for long > time, and after restart the neo4j server, > I get the log: > [2015-09-16 07:47:16] [INFO] [7557:MainThread] [http:310 send()] > GET > http://10.181.205.198:8474/db/data/ <http://10.181.205.198:8474/db/data/> > [2015-09-16 07:47:16] [INFO] [7557:MainThread] [http:511 __init__()] < 200 OK > [785] > > > How's that possible after server running normal for hours (have response > millions of read/write REST API normally), > and suddenly goes time out? and seems not recover to normal... > > On Wednesday, September 16, 2015 at 8:22:40 AM UTC+8, Michael Hunger wrote: > Hmm, > > 1. first of all you should use the / at the end otherwise you also measure > redirect time. > 2. you should use accept header of application/json > as you don't want to measure html generation. > > Please try that and report back, for me i does > > time curl -L http://neo.jexp.de:12474/db/data/ > <http://neo.jexp.de:12474/db/data/> > { > "extensions" : { }, > "node" : "http://neo.jexp.de:12474/db/data/node > <http://neo.jexp.de:12474/db/data/node>", > "node_index" : "http://neo.jexp.de:12474/db/data/index/node > <http://neo.jexp.de:12474/db/data/index/node>", > "relationship_index" : "http://neo.jexp.de:12474/db/data/index/relationship > <http://neo.jexp.de:12474/db/data/index/relationship>", > "extensions_info" : "http://neo.jexp.de:12474/db/data/ext > <http://neo.jexp.de:12474/db/data/ext>", > "relationship_types" : "http://neo.jexp.de:12474/db/data/relationship/types > <http://neo.jexp.de:12474/db/data/relationship/types>", > "batch" : "http://neo.jexp.de:12474/db/data/batch > <http://neo.jexp.de:12474/db/data/batch>", > "cypher" : "http://neo.jexp.de:12474/db/data/cypher > <http://neo.jexp.de:12474/db/data/cypher>", > "indexes" : "http://neo.jexp.de:12474/db/data/schema/index > <http://neo.jexp.de:12474/db/data/schema/index>", > "constraints" : "http://neo.jexp.de:12474/db/data/schema/constraint > <http://neo.jexp.de:12474/db/data/schema/constraint>", > "transaction" : "http://neo.jexp.de:12474/db/data/transaction > <http://neo.jexp.de:12474/db/data/transaction>", > "node_labels" : "http://neo.jexp.de:12474/db/data/labels > <http://neo.jexp.de:12474/db/data/labels>", > "neo4j_version" : "2.3-SNAPSHOT" > } > real 0m0.055s > user 0m0.004s > sys 0m0.004s > > > Am 16.09.2015 um 02:12 schrieb Singleton Chiu <[email protected] <>>: > > Just Now, I encounter the same situation, here's the more specific info, > I would very much appreciate for your time. > > neo4j v2.2.3 java1.7 on linux 64bit 96GB mem > > node nums > Node ID 9058830 > Property ID 80189638 > Relationship ID 37459428 > > > config: > org.neo4j.server.webserver.address=0.0.0.0 > org.neo4j.server.webserver.port=8474 > remote_shell_port=1390 > > > > Problem: REST API suddenly very slow response, situation for hours. When I > restart the server, API return to normal. > > $time wget localhost:8474/db/data > --2015-09-16 07:36:09-- http <http://localhost:8474/db/data>: > <http://localhost:8474/db/data>//localhost:8474/db/data > <http://localhost:8474/db/data> > Resolving localhost... 127.0.0.1 > Connecting to localhost|127.0.0.1|:8474... connected. > HTTP request sent, awaiting response... 302 Found > Location: http <http://localhost:8474/db/data/>: > <http://localhost:8474/db/data/>//localhost:8474/db/data/ > <http://localhost:8474/db/data/> [following] > --2015-09-16 07:36:43-- http <http://localhost:8474/db/data/>: > <http://localhost:8474/db/data/>//localhost:8474/db/data/ > <http://localhost:8474/db/data/> > Reusing existing connection to localhost:8474. > HTTP request sent, awaiting response... 200 OK > Length: 730 [application/json] > Saving to: `index.html.1' > > 100%[===========================================================================================================================================>] > 730 --.-K/s in 0s > > 2015-09-16 07:36:43 (60.4 MB/s) - `index.html.1' saved [730/730] > > real 0m34.547s > user 0m0.001s > sys 0m0.002s > > > See, it took 34s to get /db/data/ > > But, shell works just fine! > > > > $bin/neo4j-shell -port 1390 > Welcome to the Neo4j Shell! Enter 'help' for a list of commands > NOTE: Remote Neo4j graph database service 'shell' at port 1390 > > neo4j-sh (?)$ match (n:entity) return n limit 1; > +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > | n > > | > +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > | Node[5161]{xxxxxxxx} > +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > 1 row > 19 ms > > Here's network IO: > ... > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
