in case any one is interested, i made the memory changes as well as two changes to XX:ParallelGCThreads 8->20 XX:ConcGCThreads . 4->5
old: https://gceasy.io/diamondgc-report.jsp?p=c2hhcmVkLzIwMTkvMTIvNi8tLXNvbHJfZ2MubG9nLjAuY3VycmVudC0tMTQtMjEtMTA=&channel=WEB now: https://gceasy.io/diamondgc-report.jsp?p=c2hhcmVkLzIwMTkvMTIvOS8tLXNvbHJfZ2MubG9nLjAuY3VycmVudC0tMTQtMS02&channel=WEB however there hasnt been really anything noticeable as far as solr itself is concerned when it comes to qtimes, pre java changes: 43963 searches Complete SOLR average : 5.33 / 10th seconds for SOLR Raw SOLR over 10000/1000 secs : 208, 0.47% Raw SOLR over 1000/1000 secs : 5261, 11.97% post solr changes: 28369 searches Complete SOLR average : 4.77 / 10th seconds for SOLR Raw SOLR over 10000/1000 secs : 94, 0.33% Raw SOLR over 1000/1000 secs : 3583, 12.63% On Fri, Dec 6, 2019 at 9:39 AM David Hastings <hastings.recurs...@gmail.com> wrote: > Thanks you guys, this has been educational, i uploaded up to now, the > server was restarted after adding the extra memory, so > > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMTIvNi8tLXNvbHJfZ2MubG9nLjAuY3VycmVudC0tMTQtMjEtMTA=&channel=WEB > > is what im looking at. tuning the JVM is new to me, so im just going by > what ive researched and what this site is saying. > from what i can tell: > the peak looks like 31gb would be perfect, will implement that today > throughput is seems good, assuming gceasy recommendation of above 95% is > the target and im at 99.6 > latency looks like its as good as I really care to get, who really cares > about 200ms > as far as heap after a GC it looks like it recovered well? or am i > missing something? the red spikes of a full GC like 28gb, and right after > its down to 14gb > > I really appreciate this input, its educational/helpful > -Dave > > > > > > On Fri, Dec 6, 2019 at 7:48 AM Erick Erickson <erickerick...@gmail.com> > wrote: > >> A replication shouldn’t have consumed that much heap. It’s mostly I/O, >> just a write through. If replication really consumes huge amounts of heap >> we need to look at that more closely. Personally I suspect/hope it’s >> coincidental, but that’s only a guess. You can attach jconsole to the >> running process and monitor heap usage in real-time, jconsole is part of >> the JDK so should be relatively easy to install. It has a nifty “gc now” >> button that you can use to see if the heap you’re accumulating is just >> garbage or really accumulates… >> >> And if this really is related to replication and that much heap is >> actually used, we need to figure out why. Shawn’s observation that there is >> very little heap recovered is worrying. >> >> Best, >> Erick >> >> > On Dec 6, 2019, at 7:37 AM, Dave <hastings.recurs...@gmail.com> wrote: >> > >> > Actually at about that time the replication finished and added about >> 20-30gb to the index from the master. My current set up goes >> > Indexing master -> indexer slave/production master (only replicated on >> command)-> three search slaves (replicate each 15 minutes) >> > >> > We added about 2.3m docs, then I replicated it to the production master >> and since there was a change it replicated out to the slave node the gc >> came from >> > >> > I’ll set one of the slaves to 31/31 and force all load to that one and >> see how she does. Thanks! >> > >> > >> >> On Dec 6, 2019, at 1:02 AM, Shawn Heisey <apa...@elyograg.org> wrote: >> >> >> >> On 12/5/2019 12:57 PM, David Hastings wrote: >> >>> That probably isnt enough data, so if youre interested: >> >>> https://gofile.io/?c=rZQ2y4 >> >> >> >> The previous one was less than 4 minutes, so it doesn't reveal >> anything useful. >> >> >> >> This one is a little bit less than two hours. That's more useful, but >> still pretty short. >> >> >> >> Here's the "heap after GC" graph from the larger file: >> >> >> >> >> https://www.dropbox.com/s/q9hs8fl0gfkfqi1/david.hastings.gc.graph.2019.12.png?dl=0 >> >> >> >> At around 14:15, the heap usage was rather high. It got up over 25GB. >> There were some very long GCs right at that time, which probably means they >> were full GCs. And they didn't free up any significant amount of memory. >> So I'm betting that sometimes you actually *do* need a big chunk of that >> 60GB of heap. You might try reducing it to 31g instead of 60000m. Java's >> memory usage is a lot more efficient if the max heap size is less than 32 >> GB. >> >> >> >> I can't give you any information about what happened at that time >> which required so much heap. You could see if you have logfiles that cover >> that timeframe. >> >> >> >> Thanks, >> >> Shawn >> >>
