I'm using the community version, I have searched about my situation, and turned the gc log on. And I found that the REST API got stuck during the Full GC.
29184.923: [GC 24510626K->23288671K(24643840K), 0.0907810 secs] 29185.932: [GC 24515487K->23310086K(24643840K), 0.1171710 secs] 29186.976: [GC 24536867K->23309125K(24643840K), 0.0917860 secs] 29187.988: [GC 24535807K->23322112K(24643840K), 0.1089120 secs] 29189.018: [GC 24548925K->23333210K(24643840K), 0.1097290 secs] 29190.054: [GC 24560025K->23350298K(24643840K), 0.1037740 secs] 29204.010: [Full GC 24573732K->23319822K(24643840K), 134.0548940 secs] 29338.275: [GC 23363153K(24643840K), 0.1082740 secs] 29367.432: [Full GC 24643839K->23328404K(24643840K), 133.3548790 secs] 29500.901: [GC 23366365K(24643840K), 0.1045400 secs] 29529.261: [Full GC 24643839K->23336254K(24643840K), 134.2712890 secs] 29663.733: [GC 23367649K(24643840K), 0.1222970 secs] 29691.674: [Full GC 24643839K->23343854K(24643840K), 133.6376480 secs] 29825.511: [GC 23369329K(24643840K), 0.1298200 secs] 29854.807: [Full GC 24643839K->23351726K(24643840K), 134.3462990 secs] 29989.364: [GC 23395476K(24643840K), 0.1952480 secs] 30018.853: [Full GC 24643839K->23359633K(24643840K), 133.7108280 secs] 30152.764: [GC 23395952K(24643840K), 0.1885450 secs] 30181.939: [Full GC Exactly get stuck at the first Full GC. I thought my problem is the jvm setting, There are totally 96 GB mem, and my usage scenarios are lots of node,property,rel write and read, sometimes run a heavy time consuming cypher. I think the heap size is not enough, so I change my setting into: wrapper.java.additional=-XX:+UseConcMarkSweepGC wrapper.java.additional=-XX:+CMSClassUnloadingEnabled wrapper.java.additional=-XX:-OmitStackTraceInFastThrow wrapper.java.additional=-XX:hashCode=5 wrapper.java.initmemory=50000 wrapper.java.maxmemory=50000 dbms.pagecache.memory=30g Total Store Size 31.88 GiB Is there a guide to set these settings? The server now running for 6 hrs normally, with RES:26g, VIRT:61g *If it goes Full GC continuously or REST API get stuck again, * *I'll come back to you with the packed info.* Many Thanks! On Wednesday, September 16, 2015 at 8:44:52 AM UTC+8, Michael Hunger wrote: > > It should definitely recover > > do you run neo4j community or enterprise? > > what are your settings? > > perhaps you can pack together > > conf/* > data/log/* > data/graph.db/messsages.log > > for some analytics > > it can also be that the heap memory is fully utilized by the 2nd level > cache so it tries to make room by clearing memory, which can take a second > or two but then should be back > > so what you could also do is to enable gc-logging in neo4j-wrapper.conf > and then also send the gc-logs together with the other information. > > Am 16.09.2015 um 02:37 schrieb Singleton Chiu <[email protected] > <javascript:>>: > > Thanks for you quick response, I have already restart the server, I need > to wait the problem situation, coz I can't re-product it. > > I have checked the py2neo log: > > [2015-09-16 07:27:19] [INFO] [28681:MainThread] [http:310 send()] > GET > http://10.181.205.198:8474/db/data/ > [2015-09-16 07:27:37] [INFO] [28681:MainThread] [http:306 send()] ~ > Reconnecting (timeout) > > > Normally, it response 200 OK within 1 second. > Not only this REST API, but all encounter time out for hours, not one > moment. > > And I'm sure the IP and port are certainly right, coz It have run for long > time, and after restart the neo4j server, > I get the log: > [2015-09-16 07:47:16] [INFO] [7557:MainThread] [http:310 send()] > GET > http://10.181.205.198:8474/db/data/ > [2015-09-16 07:47:16] [INFO] [7557:MainThread] [http:511 __init__()] < 200 > OK [785] > > > How's that possible after server running normal for hours (have response > millions of read/write REST API normally), > and suddenly goes time out? and seems not recover to normal... > > On Wednesday, September 16, 2015 at 8:22:40 AM UTC+8, Michael Hunger wrote: > > Hmm, > > 1. first of all you should use the / at the end otherwise you also measure > redirect time. > 2. you should use accept header of application/json > as you don't want to measure html generation. > > Please try that and report back, for me i does > > time curl -L http://neo.jexp.de:12474/db/data/ > { > "extensions" : { }, > "node" : "http://neo.jexp.de:12474/db/data/node";, > "node_index" : "http://neo.jexp.de:12474/db/data/index/node";, > "relationship_index" : " > http://neo.jexp.de:12474/db/data/index/relationship";, > "extensions_info" : "http://neo.jexp.de:12474/db/data/ext";, > "relationship_types" : " > http://neo.jexp.de:12474/db/data/relationship/types";, > "batch" : "http://neo.jexp.de:12474/db/data/batch";, > "cypher" : "http://neo.jexp.de:12474/db/data/cypher";, > "indexes" : "http://neo.jexp.de:12474/db/data/schema/index";, > "constraints" : "http://neo.jexp.de:12474/db/data/schema/constraint";, > "transaction" : "http://neo.jexp.de:12474/db/data/transaction";, > "node_labels" : "http://neo.jexp.de:12474/db/data/labels";, > "neo4j_version" : "2.3-SNAPSHOT" > } > real 0m0.055s > user 0m0.004s > sys 0m0.004s > > > Am 16.09.2015 um 02:12 schrieb Singleton Chiu <[email protected]>: > > Just Now, I encounter the same situation, here's the more specific info, > I would very much appreciate for your time. > > neo4j v2.2.3 java1.7 on linux 64bit 96GB mem > > node nums > Node ID 9058830 > Property ID 80189638 > Relationship ID 37459428 > > > config: > org.neo4j.server.webserver.address=0.0.0.0 > org.neo4j.server.webserver.port=8474 > remote_shell_port=1390 > > > > Problem: REST API suddenly very slow response, situation for hours. When I > restart the server, API return to normal. > > $time wget localhost:8474/db/data > --2015-09-16 07:36:09-- http <http://localhost:8474/db/data>: > <http://localhost:8474/db/data>//localhost:8474/db/data > <http://localhost:8474/db/data> > Resolving localhost... 127.0.0.1 > Connecting to localhost|127.0.0.1|:8474... connected. > HTTP request sent, awaiting response... 302 Found > Location: http <http://localhost:8474/db/data/>: > <http://localhost:8474/db/data/>//localhost:8474/db/data/ > <http://localhost:8474/db/data/> [following] > --2015-09-16 07:36:43-- http <http://localhost:8474/db/data/>: > <http://localhost:8474/db/data/>//localhost:8474/db/data/ > <http://localhost:8474/db/data/> > Reusing existing connection to localhost:8474. > HTTP request sent, awaiting response... 200 OK > Length: 730 [application/json] > Saving to: `index.html.1' > > 100%[===========================================================================================================================================>] > > 730 --.-K/s in 0s > > 2015-09-16 07:36:43 (60.4 MB/s) - `index.html.1' saved [730/730] > > real 0m34.547s > user 0m0.001s > sys 0m0.002s > > > See, it took 34s to get /db/data/ > > But, shell works just fine! > > > > $bin/neo4j-shell -port 1390 > Welcome to the Neo4j Shell! Enter 'help' for a list of commands > NOTE: Remote Neo4j graph database service 'shell' at port 1390 > > neo4j-sh (?)$ match (n:entity) return n limit 1; > > +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > | n > > | > > +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > | Node[5161]{xxxxxxxx} > > +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > 1 row > 19 ms > > Here's network IO: > > ... -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
