Re: edit gc parameters in solr.in.sh or solr?
On 3/28/2018 1:06 AM, Bernd Fehling wrote: Humongous Allocations are not genrally bad. Sure, the G1 part for humongous allocations is not that performant and takes time. But just try to limit humongous allocations and not to avoid it under all circumstances. I was told by Oracle engineers that humongous allocations can ONLY be collected by a full GC. A full GC on my servers, with an 8GB heap, can take 10-15 seconds. This isn't a guess -- I've examined the GC logs. And that's a full stop-the-world pause for the entire collection. If I increase the region size, then filterCache entries are not tagged as humongous, and full GCs almost never happen. Every single filterCache entry on my large shards was more than 2MB at the time I was experimenting. With an 8GB heap, the calculated G1 region size is 4MB. Anything that's half the region size or larger is considered a humongous allocation, so objects over 2MB get that tag. Each server has three of those large shards, so when a new query goes into the filterCache, three large filterCache entries are created in one JVM. These days, each of those shards has 31 million docs, which means that a filterCache entry has almost doubled in size since then. It's almost 4 MB. If one of those cores gets updated, multiple large objects become garbage, and because of cache autowarming (with autowarmCount set to 4), a few more get created. Each of our queries typically has several filter queries in it. Some of them don't really change, but a couple of them are highly variable from user to user. Solr burns through the heap pretty fast with those large filterCache entries. When it fills up with garbage objects tagged humongous, the only way to clear it out is a full GC, because the concurrent low-pause collectors are unable to collect humongous objects. Because the region size tops out at 32MB, that means that any object over 16MB is always humongous. So for indexes with a very large number of documents per core (more than about 134 million), G1 is not a good choice. Unless the filterCache is completely disabled, which isn't normally recommended, G1 can't be tuned to avoid full GC with an index that size. Thanks, Shawn
Re: edit gc parameters in solr.in.sh or solr?
Hi Shawn, the problem with heap regions is, you can't get one advantage without any disadvantage. According to your G1 example: 4GB heap with default 2MB region size = 2048 heap regions 4GB heap with G1HeapRegionSize set to 8MB = 512 heap regions You see, you only have 1/4th of heap regions left. This also means that objects which are only 1MB in size occupy 8MB on heap and therefore a whole region, which is already very low. Humongous Allocations are not genrally bad. Sure, the G1 part for humongous allocations is not that performant and takes time. But just try to limit humongous allocations and not to avoid it under all circumstances. Regards Bernd Am 27.03.2018 um 23:07 schrieb Shawn Heisey: > On 3/27/2018 12:13 AM, Bernd Fehling wrote: >> may I give you the advise to _NOT_ set XX:G1HeapRegionSize. >> That is computed during JAVA start by the engine according to heap and >> available memory. >> A wrong set size can even a huge machine with 31GB heap and 157GB RAM force >> into OOM. >> Guess how I figured that out, took me about one week to locate it. > > I have some notes on why I included that parameter on my wiki page. > > https://wiki.apache.org/solr/ShawnHeisey#G1_.28Garbage_First.29_Collector > > Basically, the filterCache entries were being marked as humongous > allocations, because each one for my indexes is over 2MB in size. > Apparently it takes a full collection to collect humongous allocations > that become garbage, at least in the versions of Java that I was > experimenting with. So without that parameter, full GCs were required, > and that will always make GC slow unless the heap size is very small. > > If Oracle has made it so that humongous allocations can be collected by > the generation-specific collectors, then that parameter may no longer be > required in newer Java versions. I do not know if this has happened. > > Thanks, > Shawn >
Re: edit gc parameters in solr.in.sh or solr?
On 3/27/2018 12:13 AM, Bernd Fehling wrote: > may I give you the advise to _NOT_ set XX:G1HeapRegionSize. > That is computed during JAVA start by the engine according to heap and > available memory. > A wrong set size can even a huge machine with 31GB heap and 157GB RAM force > into OOM. > Guess how I figured that out, took me about one week to locate it. I have some notes on why I included that parameter on my wiki page. https://wiki.apache.org/solr/ShawnHeisey#G1_.28Garbage_First.29_Collector Basically, the filterCache entries were being marked as humongous allocations, because each one for my indexes is over 2MB in size. Apparently it takes a full collection to collect humongous allocations that become garbage, at least in the versions of Java that I was experimenting with. So without that parameter, full GCs were required, and that will always make GC slow unless the heap size is very small. If Oracle has made it so that humongous allocations can be collected by the generation-specific collectors, then that parameter may no longer be required in newer Java versions. I do not know if this has happened. Thanks, Shawn
Re: edit gc parameters in solr.in.sh or solr?
Hi Walter, may I give you the advise to _NOT_ set XX:G1HeapRegionSize. That is computed during JAVA start by the engine according to heap and available memory. A wrong set size can even a huge machine with 31GB heap and 157GB RAM force into OOM. Guess how I figured that out, took me about one week to locate it. Regards Bernd Am 26.03.2018 um 17:08 schrieb Walter Underwood: > We use the G1 collector in Java 8u131 and it works well. We are running > 6.6.2. Our Solr instances do a LOT of allocation. We have long queries (25 > terms average) and many unique queries. > > SOLR_HEAP=8g > # Use G1 GC -- wunder 2017-01-23 > # Settings from https://wiki.apache.org/solr/ShawnHeisey > GC_TUNE=" \ > -XX:+UseG1GC \ > -XX:+ParallelRefProcEnabled \ > -XX:G1HeapRegionSize=8m \ > -XX:MaxGCPauseMillis=200 \ > -XX:+UseLargePages \ > -XX:+AggressiveOpts \ > " > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > >> On Mar 26, 2018, at 1:22 AM, Derek Pohwrote: >> >> Hi >> >> From your experience, would like to know if It is advisable to change the gc >> parameters in solr.in.sh or solrfile? >> It is mentioned in the documentation to edit solr.in.sh but would like >> toknow which file you actually edit. >> >> I am using Solr 6.6.2at the moment. >> >> Regards, >> Derek >> >>
Re: edit gc parameters in solr.in.sh or solr?
On 3/26/2018 6:41 PM, Derek Poh wrote: On my installation, "solr.in.sh" is in solr-6.6.2/bin directory. It is recommended to place the file in /etc/default? Regarding the "solrfile", I was referring to the file "solr". Sorry for the typo. The file "solr" is not edited normally? I've redirected my reply to this private message back to the list. http://people.apache.org/~hossman/#private_q If the active solr.in.sh file is not in /etc/default, that means that the service installer script was NOT used. I strongly recommend using the service installer script on systems that will support it. https://lucene.apache.org/solr/guide/6_6/taking-solr-to-production.html Moving the location of that file manually to /etc/default might not actually work. I have not reviewed the script recently enough to know for sure. It *might* work. My memory is fuzzy, but checking that location might be part of the bin/solr script already. Whether that memory is correct or not, I still recommend running the service installer script. The bin/solr script should not be edited unless you're fixing a bug in that script. A lot of Solr's startup settings can be changed with solr.in.sh. More settings are being added over time as Solr evolves. Thanks, Shawn
Re: edit gc parameters in solr.in.sh or solr?
We use the G1 collector in Java 8u131 and it works well. We are running 6.6.2. Our Solr instances do a LOT of allocation. We have long queries (25 terms average) and many unique queries. SOLR_HEAP=8g # Use G1 GC -- wunder 2017-01-23 # Settings from https://wiki.apache.org/solr/ShawnHeisey GC_TUNE=" \ -XX:+UseG1GC \ -XX:+ParallelRefProcEnabled \ -XX:G1HeapRegionSize=8m \ -XX:MaxGCPauseMillis=200 \ -XX:+UseLargePages \ -XX:+AggressiveOpts \ " wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Mar 26, 2018, at 1:22 AM, Derek Pohwrote: > > Hi > > From your experience, would like to know if It is advisable to change the gc > parameters in solr.in.sh or solrfile? > It is mentioned in the documentation to edit solr.in.sh but would like toknow > which file you actually edit. > > I am using Solr 6.6.2at the moment. > > Regards, > Derek > > > -- > CONFIDENTIALITY NOTICE > This e-mail (including any attachments) may contain confidential and/or > privileged information. If you are not the intended recipient or have > received this e-mail in error, please inform the sender immediately and > delete this e-mail (including any attachments) from your computer, and you > must not use, disclose to anyone else or copy this e-mail (including any > attachments), whether in whole or in part. > This e-mail and any reply to it may be monitored for security, legal, > regulatory compliance and/or other appropriate reasons.
Re: edit gc parameters in solr.in.sh or solr?
On 3/26/2018 2:22 AM, Derek Poh wrote: From your experience, would like to know if It is advisable to change the gc parameters in solr.in.sh or solrfile? It is mentioned in the documentation to edit solr.in.sh but would like toknow which file you actually edit. You need a GC_TUNE variable in solr.in.sh. The java commandline parameters specified there will replace the standard GC tuning parameters. If recommendations are followed, this file will be found in /etc/default, and could have "solr" in the filename replaced with something different, specifically the name given to the installed service. On my dev server, it is named "solr6.in.sh". What is the "solrfile" you have referenced? I've not heard of this. Thanks, Shawn
edit gc parameters in solr.in.sh or solr?
Hi From your experience, would like to know if It is advisable to change the gc parameters in solr.in.sh or solrfile? It is mentioned in the documentation to edit solr.in.sh but would like toknow which file you actually edit. I am using Solr 6.6.2at the moment. Regards, Derek -- CONFIDENTIALITY NOTICE This e-mail (including any attachments) may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error, please inform the sender immediately and delete this e-mail (including any attachments) from your computer, and you must not use, disclose to anyone else or copy this e-mail (including any attachments), whether in whole or in part. This e-mail and any reply to it may be monitored for security, legal, regulatory compliance and/or other appropriate reasons.