Marc, Re smaller index sizes - it's the stored field compression that didn't exist in 3.x. See https://issues.apache.org/jira/browse/SOLR-4375
Otis -- Solr & ElasticSearch Support http://sematext.com/ On Thu, Apr 11, 2013 at 10:53 AM, Marc Des Garets <marc.desgar...@192.com> wrote: > Same config. I compared both, some defaults changed like ramBufferSize > which I've set like in 3.5 (same with other things). > > It becomes even more strange to me. Now I have changed the jvm settings > to this: > -d64 -server -Xms40g -Xmx40g -XX:+UseG1GC -XX:NewRatio=6 > -XX:SurvivorRatio=2 -XX:G1ReservePercent=10 -XX:MaxGCPauseMillis=100 > -XX:InitiatingHeapOccupancyPercent=30 -XX:PermSize=728m -XX:MaxPermSize=728m > > So the Eden space is just 6Gb, survivor space is still weird (80Mb) and > full 100% of time, and old gen is 34Gb. > > I now get GCs of just 0.07 sec every 30sec/1mn. Very regular like this: > [GC pause (young) 16214M->10447M(40960M), 0.0738720 secs] > > Just 30% of the total heap is used. > > After while it's going to do: > [GC pause (young) (initial-mark) 11603M->11391M(40960M), 0.1099990 secs] > [GC concurrent-root-region-scan-start] > [GC concurrent-root-region-scan-end, 0.0172380] > [GC concurrent-mark-start] > [GC concurrent-mark-end, 0.4824210 sec] > [GC remark, 0.0248680 secs] > [GC cleanup 11476M->11476M(40960M), 0.0116420 secs] > > Which looks pretty good. If I am not mistaken, concurrent-mark isn't > stop the world. remark is stop the world but is just 0.02 sec and GC > cleanup is also stop the world but is just 0.01 sec. > > By the look of it I could have a 20g heap rather than 40... Now I am > waiting to see what happens when it will clear the old gen but that will > take a while before it happens because it is growing slowly. > > Still mysterious to me but it looks like it's going to all work out. > > On 04/11/2013 03:06 PM, Jack Krupansky wrote: >> Same config? Do a compare with the new example config and see what settings >> are different/changed. There may have been some defaults that changed. Read >> the comments in the new config. >> >> If you had just taken or merged the new config, then I would suggest making >> sure that the update log is not enabled (or make sure you do hard commits >> relatively frequently rather than only soft commits.) >> >> -- Jack Krupansky >> >> -----Original Message----- >> From: Marc Des Garets >> Sent: Thursday, April 11, 2013 3:07 AM >> To: solr-user@lucene.apache.org >> Subject: Re: migration solr 3.5 to 4.1 - JVM GC problems >> >> Big heap because very large number of requests with more than 60 indexes >> and hundreds of million of documents (all indexes together). My problem >> is with solr 4.1. All is perfect with 3.5. I have 0.05 sec GCs every 1 >> or 2mn and 20Gb of the heap is used. >> >> With the 4.1 indexes it uses 30Gb-33Gb, the survivor space is all weird >> (it changed the size capacity to 6Mb at some point) and I have 2 sec GCs >> every minute. >> >> There must be something that has changed in 4.1 compared to 3.5 to cause >> this behavior. It's the same requests, same schemas (excepted 4 fields >> changed from sint to tint) and same config. >> >> On 04/10/2013 07:38 PM, Shawn Heisey wrote: >>> On 4/10/2013 9:48 AM, Marc Des Garets wrote: >>>> The JVM behavior is now radically different and doesn't seem to make >>>> sense. I was using ConcMarkSweepGC. I am now trying the G1 collector. >>>> >>>> The perm gen went from 410Mb to 600Mb. >>>> >>>> The eden space usage is a lot bigger and the survivor space usage is >>>> 100% all the time. >>>> >>>> I don't really understand what is happening. GC behavior really doesn't >>>> seem right. >>>> >>>> My jvm settings: >>>> -d64 -server -Xms40g -Xmx40g -XX:+UseG1GC -XX:NewRatio=1 >>>> -XX:SurvivorRatio=3 -XX:PermSize=728m -XX:MaxPermSize=728m >>> As Otis has already asked, why do you have a 40GB heap? The only way I >>> can imagine that you would actually NEED a heap that big is if your >>> index size is measured in hundreds of gigabytes. If you really do need >>> a heap that big, you will probably need to go with a JVM like Zing. I >>> don't know how much Zing costs, but they claim to be able to make any >>> heap size perform well under any load. It is Linux-only. >>> >>> I was running into extreme problems with GC pauses with my own setup, >>> and that was only with an 8GB heap. I was using the CMS collector and >>> NewRatio=1. Switching to G1 didn't help at all - it might have even >>> made the problem worse. I never did try the Zing JVM. >>> >>> After a lot of experimentation (which I will admit was not done very >>> methodically) I found JVM options that have reduced the GC pause problem >>> greatly. Below is what I am using now on Solr 4.2.1 with a total >>> per-server index size of about 45GB. This works properly on CentOS 6 >>> with Oracle Java 7u17, UseLargePages may require special kernel tuning >>> on other operating systems: >>> >>> -Xmx6144M -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 >>> -XX:NewRatio=3 -XX:MaxTenuringThreshold=8 -XX:+CMSParallelRemarkEnabled >>> -XX:+ParallelRefProcEnabled -XX:+UseLargePages -XX:+AggressiveOpts >>> >>> These options could probably use further tuning, but I haven't had time >>> for the kind of testing that will be required. >>> >>> If you decide to pay someone to make the problem going away instead: >>> >>> http://www.azulsystems.com/products/zing/whatisit >>> >>> Thanks, >>> Shawn >>> >>> >>> >> >> This transmission is strictly confidential, possibly legally privileged, and >> intended solely for the addressee. >> Any views or opinions expressed within it are those of the author and do not >> necessarily represent those of >> 192.com Ltd or any of its subsidiary companies. If you are not the intended >> recipient then you must >> not disclose, copy or take any action in reliance of this transmission. If >> you have received this >> transmission in error, please notify the sender as soon as possible. No >> employee or agent is authorised >> to conclude any binding agreement on behalf 192.com Ltd with another party >> by email without express written >> confirmation by an authorised employee of the company. http://www.192.com >> (Tel: 08000 192 192). >> 192.com Ltd is incorporated in England and Wales, company number 07180348, >> VAT No. GB 103226273. >> >> >> > > > This transmission is strictly confidential, possibly legally privileged, and > intended solely for the addressee. > Any views or opinions expressed within it are those of the author and do not > necessarily represent those of > 192.com Ltd or any of its subsidiary companies. If you are not the intended > recipient then you must > not disclose, copy or take any action in reliance of this transmission. If > you have received this > transmission in error, please notify the sender as soon as possible. No > employee or agent is authorised > to conclude any binding agreement on behalf 192.com Ltd with another party by > email without express written > confirmation by an authorised employee of the company. http://www.192.com > (Tel: 08000 192 192). > 192.com Ltd is incorporated in England and Wales, company number 07180348, > VAT No. GB 103226273.
