On Wed, Oct 26, 2016 at 4:53 AM, Shawn Heisey <apa...@elyograg.org> wrote: > On 10/25/2016 8:03 PM, Susheel Kumar wrote: >> Agree, Pushkar. I had docValues for sorting / faceting fields from >> begining (since I setup Solr 6.0). So good on that side. I am going to >> analyze the queries to find any potential issue. Two questions which I am >> puzzling with >> >> a) Should the below JVM parameter be included for Prod to get heap dump >> >> "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/the/dump" > > A heap dump can take a very long time to complete, and there may not be > enough memory in the machine to start another instance of Solr until the > first one has finished the heap dump. Also, I do not know whether Java > would release the listening port before the heap dump finishes. If not, > then a new instance would not be able to start immediately. > > If a different heap dump file is created each time, that might lead to > problems with disk space after repeated dumps. I don't know how the > option works. > >> b) Currently OOM script just kills the Solr instance. Shouldn't it be >> enhanced to wait and restart Solr instance > > As long as there is a problem causing OOMs, it seems rather pointless to > start Solr right back up, as another OOM is likely. The safest thing to > do is kill Solr (since its operation would be unpredictable after OOM) > and let the admin sort the problem out. >
Occasionally our cloud nodes can OOM, when particularly complex faceting is performed. The current OOM management can be exceedingly annoying; a user will make a too complex analysis request, bringing down one server, taking it out of the balancer. The user gets fed up at no response, so reloads the page, re-submitting the analysis and bringing down the next server in the cluster. Lather, rinse, repeat - and then you get to have a meeting to discuss why we invest so much in HA infrastructure that can be made non-HA by one user with a complex query. In those meetings it is much harder to justify not restarting. Cheers Tom
