The thing that’s unsettling about this is that assuming you were hitting OOMs, and were running the OOM-killer script, you _should_ have had very clear evidence that that was the cause.
If you were not running the killer script, the apologies for not asking about that in the first place. Java’s performance is unpredictable when OOMs happen, which is the point of the killer script: at least Solr stops rather than do something inexplicable. Best, Erick > On Jun 29, 2020, at 11:52 AM, David Hastings <hastings.recurs...@gmail.com> > wrote: > > sometimes just throwing money/ram/ssd at the problem is just the best > answer. > > On Mon, Jun 29, 2020 at 11:38 AM Ryan W <rya...@gmail.com> wrote: > >> Thanks everyone. Just to give an update on this issue, I bumped the RAM >> available to Solr up to 16GB a couple weeks ago, and haven’t had any >> problem since. >> >> >> On Tue, Jun 16, 2020 at 1:00 PM David Hastings < >> hastings.recurs...@gmail.com> >> wrote: >> >>> me personally, around 290gb. as much as we could shove into them >>> >>> On Tue, Jun 16, 2020 at 12:44 PM Erick Erickson <erickerick...@gmail.com >>> >>> wrote: >>> >>>> How much physical RAM? A rule of thumb is that you should allocate no >>> more >>>> than 25-50 percent of the total physical RAM to Solr. That's >> cumulative, >>>> i.e. the sum of the heap allocations across all your JVMs should be >> below >>>> that percentage. See Uwe Schindler's mmapdirectiry blog... >>>> >>>> Shot in the dark... >>>> >>>> On Tue, Jun 16, 2020, 11:51 David Hastings < >> hastings.recurs...@gmail.com >>>> >>>> wrote: >>>> >>>>> To add to this, i generally have solr start with this: >>>>> -Xms31000m-Xmx31000m >>>>> >>>>> and the only other thing that runs on them are maria db gallera >> cluster >>>>> nodes that are not in use (aside from replication) >>>>> >>>>> the 31gb is not an accident either, you dont want 32gb. >>>>> >>>>> >>>>> On Tue, Jun 16, 2020 at 11:26 AM Shawn Heisey <apa...@elyograg.org> >>>> wrote: >>>>> >>>>>> On 6/11/2020 11:52 AM, Ryan W wrote: >>>>>>>> I will check "dmesg" first, to find out any hardware error >>> message. >>>>>> >>>>>> <snip> >>>>>> >>>>>>> [1521232.781801] Out of memory: Kill process 117529 (httpd) >> score 9 >>>> or >>>>>>> sacrifice child >>>>>>> [1521232.782908] Killed process 117529 (httpd), UID 48, >>>>>> total-vm:675824kB, >>>>>>> anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB >>>>>>> >>>>>>> Is this a relevant "Out of memory" message? Does this suggest an >>> OOM >>>>>>> situation is the culprit? >>>>>> >>>>>> Because this was in the "dmesg" output, it indicates that it is the >>>>>> operating system killing programs because the *system* doesn't have >>> any >>>>>> memory left. It wasn't Java that did this, and it wasn't Solr that >>> was >>>>>> killed. It very well could have been Solr that was killed at >> another >>>>>> time, though. >>>>>> >>>>>> The process that it killed this time is named httpd ... which is >> most >>>>>> likely the Apache webserver. Because the UID is 48, this is >> probably >>>> an >>>>>> OS derived from Redhat, where the "apache" user has UID and GID 48 >> by >>>>>> default. Apache with its default config can be VERY memory hungry >>> when >>>>>> it gets busy. >>>>>> >>>>>>> -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912 >>>>>> >>>>>> This says that you started Solr with the default 512MB heap. Which >>> is >>>>>> VERY VERY small. The default is small so that Solr will start on >>>>>> virtually any hardware. Almost every user must increase the heap >>> size. >>>>>> And because the OS is killing processes, it is likely that the >> system >>>>>> does not have enough memory installed for what you have running on >>> it. >>>>>> >>>>>> It is generally not a good idea to share the server hardware >> between >>>>>> Solr and other software, unless the system has a lot of spare >>>> resources, >>>>>> memory in particular. >>>>>> >>>>>> Thanks, >>>>>> Shawn >>>>>> >>>>> >>>> >>> >>
