I am running a query that returns 10 MM docs in total and the number of rows per page is 100K.
On Wed, Apr 12, 2017 at 12:53 PM, Mikhail Khludnev <gge...@gmail.com> wrote: > And what is the rows parameter? > > 12 апр. 2017 г. 21:32 пользователь "Chetas Joshi" <chetas.jo...@gmail.com> > написал: > > > Thanks for your response Shawn and Wunder. > > > > Hi Shawn, > > > > Here is the system config: > > > > Total system memory = 512 GB > > each server handles two 500 MB cores > > Number of solr docs per 500 MB core = 200 MM > > > > The average heap usage is around 4-6 GB. When the read starts using the > > Cursor approach, the heap usage starts increasing with the base of the > > sawtooth at 8 GB and then shooting up to 17 GB. Even after the full GC, > the > > heap usage remains around 15 GB and then it comes down to 8 GB. > > > > With 100K docs, the requirement will be in MBs so it is strange it is > > jumping from 8 GB to 17 GB while preparing the sorted response. > > > > Thanks! > > > > > > > > On Tue, Apr 11, 2017 at 8:48 PM, Walter Underwood <wun...@wunderwood.org > > > > wrote: > > > > > JVM version? We’re running v8 update 121 with the G1 collector and it > is > > > working really well. We also have an 8GB heap. > > > > > > Graph your heap usage. You’ll see a sawtooth shape, where it grows, > then > > > there is a major GC. The maximum of the base of the sawtooth is the > > working > > > set of heap that your Solr installation needs. Set the heap to that > > value, > > > plus a gigabyte or so. We run with a 2GB eden (new space) because so > much > > > of Solr’s allocations have a lifetime of one request. So, the base of > the > > > sawtooth, plus a gigabyte breathing room, plus two more for eden. That > > > should work. > > > > > > I don’t set all the ratios and stuff. When were running CMS, I set a > size > > > for the heap and a size for the new space. Done. With G1, I don’t even > > get > > > that fussy. > > > > > > wunder > > > Walter Underwood > > > wun...@wunderwood.org > > > http://observer.wunderwood.org/ (my blog) > > > > > > > > > > On Apr 11, 2017, at 8:22 PM, Shawn Heisey <apa...@elyograg.org> > wrote: > > > > > > > > On 4/11/2017 2:56 PM, Chetas Joshi wrote: > > > >> I am using Solr (5.5.0) on HDFS. SolrCloud of 80 nodes. Sold > > collection > > > >> with number of shards = 80 and replication Factor=2 > > > >> > > > >> Sold JVM heap size = 20 GB > > > >> solr.hdfs.blockcache.enabled = true > > > >> solr.hdfs.blockcache.direct.memory.allocation = true > > > >> MaxDirectMemorySize = 25 GB > > > >> > > > >> I am querying a solr collection with index size = 500 MB per core. > > > > > > > > I see that you and I have traded messages before on the list. > > > > > > > > How much total system memory is there per server? How many of these > > > > 500MB cores are on each server? How many docs are in a 500MB core? > > The > > > > answers to these questions may affect the other advice that I give > you. > > > > > > > >> The off-heap (25 GB) is huge so that it can load the entire index. > > > > > > > > I still know very little about how HDFS handles caching and memory. > > You > > > > want to be sure that as much data as possible from your indexes is > > > > sitting in local memory on the server. > > > > > > > >> Using cursor approach (number of rows = 100K), I read 2 fields > (Total > > 40 > > > >> bytes per solr doc) from the Solr docs that satisfy the query. The > > docs > > > are sorted by "id" and then by those 2 fields. > > > >> > > > >> I am not able to understand why the heap memory is getting full and > > Full > > > >> GCs are consecutively running with long GC pauses (> 30 seconds). I > am > > > >> using CMS GC. > > > > > > > > A 20GB heap is quite large. Do you actually need it to be that > large? > > > > If you graph JVM heap usage over a long period of time, what are the > > low > > > > points in the graph? > > > > > > > > A result containing 100K docs is going to be pretty large, even with > a > > > > limited number of fields. It is likely to be several megabytes. It > > > > will need to be entirely built in the heap memory before it is sent > to > > > > the client -- both as Lucene data structures (which will probably be > > > > much larger than the actual response due to Java overhead) and as the > > > > actual response format. Then it will be garbage as soon as the > > response > > > > is done. Repeat this enough times, and you're going to go through > even > > > > a 20GB heap pretty fast, and need a full GC. Full GCs on a 20GB heap > > > > are slow. > > > > > > > > You could try switching to G1, as long as you realize that you're > going > > > > against advice from Lucene experts.... but honestly, I do not expect > > > > this to really help, because you would probably still need full GCs > due > > > > to the rate that garbage is being created. If you do try it, I would > > > > strongly recommend the latest Java 8, either Oracle or OpenJDK. > Here's > > > > my wiki page where I discuss this: > > > > > > > > https://wiki.apache.org/solr/ShawnHeisey#G1_.28Garbage_ > > > First.29_Collector > > > > > > > > Reducing the heap size (which may not be possible -- need to know the > > > > answer to the question about memory graphing) and reducing the number > > of > > > > rows per query are the only quick solutions I can think of. > > > > > > > > Thanks, > > > > Shawn > > > > > > > > > > > > >
