Thank you. I pasted those settings at the end of my /etc/default/solr.in.sh just now and restarted solr. I will see if that fixes it. Previously, I had no settings at all in solr.in.sh except for SOLR_PORT.
On Thu, Jun 11, 2020 at 1:59 PM Walter Underwood <wun...@wunderwood.org> wrote: > 1. You have a tiny heap. 536 Megabytes is not enough. > 2. I stopped using the CMS GC years ago. > > Here is the GC config we use on every one of our 150+ Solr hosts. We’re > still on Java 8, but will be upgrading soon. > > SOLR_HEAP=8g > # Use G1 GC -- wunder 2017-01-23 > # Settings from https://wiki.apache.org/solr/ShawnHeisey > GC_TUNE=" \ > -XX:+UseG1GC \ > -XX:+ParallelRefProcEnabled \ > -XX:G1HeapRegionSize=8m \ > -XX:MaxGCPauseMillis=200 \ > -XX:+UseLargePages \ > -XX:+AggressiveOpts \ > " > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > On Jun 11, 2020, at 10:52 AM, Ryan W <rya...@gmail.com> wrote: > > > > On Wed, Jun 10, 2020 at 8:35 PM Hup Chen <chai...@hotmail.com> wrote: > > > >> I will check "dmesg" first, to find out any hardware error message. > >> > > > > Here is what I see toward the end of the output from dmesg: > > > > [1521232.781785] [118857] 48 118857 108785 677 201 > > 901 0 httpd > > [1521232.781787] [118860] 48 118860 108785 710 201 > > 881 0 httpd > > [1521232.781788] [118862] 48 118862 113063 5256 210 > > 725 0 httpd > > [1521232.781790] [118864] 48 118864 114085 6634 212 > > 703 0 httpd > > [1521232.781791] [118871] 48 118871 139687 32323 262 > > 620 0 httpd > > [1521232.781793] [118873] 48 118873 108785 821 201 > > 792 0 httpd > > [1521232.781795] [118879] 48 118879 140263 32719 263 > > 621 0 httpd > > [1521232.781796] [118903] 48 118903 108785 812 201 > > 771 0 httpd > > [1521232.781798] [118905] 48 118905 113575 5606 211 > > 660 0 httpd > > [1521232.781800] [118906] 48 118906 113563 5694 211 > > 626 0 httpd > > [1521232.781801] Out of memory: Kill process 117529 (httpd) score 9 or > > sacrifice child > > [1521232.782908] Killed process 117529 (httpd), UID 48, > total-vm:675824kB, > > anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB > > > > Is this a relevant "Out of memory" message? Does this suggest an OOM > > situation is the culprit? > > > > When I grep in the solr logs for oom, I see some entries like this... > > > > ./solr_gc.log.4.current:CommandLine flags: -XX:CICompilerCount=4 > > -XX:CMSInitiatingOccupancyFraction=50 > -XX:CMSMaxAbortablePrecleanTime=6000 > > -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark > > -XX:ConcGCThreads=4 -XX:GCLogFileSize=20971520 > > -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912 > > -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=8 > > -XX:MinHeapDeltaBytes=196608 -XX:NewRatio=3 -XX:NewSize=134217728 > > -XX:NumberOfGCLogFiles=9 -XX:OldPLABSize=16 -XX:OldSize=402653184 > > -XX:-OmitStackTraceInFastThrow > > -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 > /opt/solr/server/logs > > -XX:ParallelGCThreads=4 -XX:+ParallelRefProcEnabled > > -XX:PretenureSizeThreshold=67108864 -XX:+PrintGC > > -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps > > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC > > -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4 > > -XX:TargetSurvivorRatio=90 -XX:ThreadStackSize=256 > > -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers > > -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation > > -XX:+UseParNewGC > > > > Buried in there I see "OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh". > But I > > think this is just a setting that indicates what to do in case of an OOM. > > And if I look in that oom_solr.sh file, I see it would write an entry to > a > > solr_oom_kill log. And there is no such log in the logs directory. > > > > Many thanks. > > > > > > > > > >> Then use some system admin tools to monitor that server, > >> for instance, top, vmstat, lsof, iostat ... or simply install some nice > >> free monitoring tool into this system, like monit, monitorix, nagios. > >> Good luck! > >> > >> ________________________________ > >> From: Ryan W <rya...@gmail.com> > >> Sent: Thursday, June 11, 2020 2:13 AM > >> To: solr-user@lucene.apache.org <solr-user@lucene.apache.org> > >> Subject: Re: How to determine why solr stops running? > >> > >> Hi all, > >> > >> People keep suggesting I check the logs for errors. What do those > errors > >> look like? Does anyone have examples of the text of a Solr oom error? > Or > >> the text of any other errors I should be looking for the next time solr > >> fails? Are there phrases I should grep for in the logs? Should I be > >> looking in the Solr logs for an OOM error, or in the Apache logs? > >> > >> There is nothing failing on the server except for solr -- at least not > that > >> I can see. There is no apparent problem with the hardware or anything > else > >> on the server. The OS is Red Hat Enterprise Linux. The server has 16 > GB of > >> RAM and hosts one website that does not get a huge amount of traffic. > >> > >> When the start command is given to solr, does it first check to see if > solr > >> is running, or does it always start solr whether it is already running > or > >> not? > >> > >> Many thanks! > >> Ryan > >> > >> > >> On Tue, Jun 9, 2020 at 7:58 AM Erick Erickson <erickerick...@gmail.com> > >> wrote: > >> > >>> To add to what Dave said, if you have a particular machine that’s prone > >> to > >>> suddenly stopping, that’s usually a red flag that you should seriously > >>> think about hardware issues. > >>> > >>> If the problem strikes different machines, then I agree with Shawn that > >>> the first thing I’d be suspicious of is OOM errors. > >>> > >>> FWIW, > >>> Erick > >>> > >>>> On Jun 9, 2020, at 6:05 AM, Dave <hastings.recurs...@gmail.com> > wrote: > >>>> > >>>> I’ll add that whenever I’ve had a solr instance shut down, for me it’s > >>> been a hardware failure. Either the ram or the disk got a “glitch” and > >> both > >>> of these are relatively fragile and wear and tear type parts of the > >>> machine, and should be expected to fail and be replaced from time to > >> time. > >>> Solr is pretty aggressive with its logging so there are a lot of writes > >>> always happening and of course reads, if the disk has any issues or the > >>> memory it can lock it up and bring her down, more so if you have any > >>> spellcheck dictionaries or suggesters being built on start up. > >>>> > >>>> Just my experience with this, could be wrong (most likely wrong) but > we > >>> always have extra drives and memory around the server room for this > >>> reason. At least once or twice a year we will have a disk failure in > the > >>> raid and need to swap in a new one. > >>>> > >>>> Good luck though, also solr should be logging it’s failures so it > would > >>> be good to look there too > >>>> > >>>>> On Jun 9, 2020, at 2:35 AM, Shawn Heisey <apa...@elyograg.org> > wrote: > >>>>> > >>>>> On 5/14/2020 7:22 AM, Ryan W wrote: > >>>>>> I manage a site where solr has stopped running a couple times in the > >>> past > >>>>>> week. The server hasn't been rebooted, so that's not the reason. > >> What > >>> else > >>>>>> causes solr to stop running? How can I investigate why this is > >>> happening? > >>>>> > >>>>> Any situation where Solr stops running and nobody requested the stop > >> is > >>> a result of a serious problem that must be thoroughly investigated. I > >>> think it's a bad idea for Solr to automatically restart when it stops > >>> unexpectedly. Chances are that whatever caused the crash is going to > >>> simply make the crash happen again until the problem is solved. > >>> Automatically restarting could hide problems from the system > >> administrator. > >>>>> > >>>>> The only way a Solr auto-restart would be acceptable to me is if it > >>> sends a high priority alert to the sysadmin EVERY time it executes an > >>> auto-restart. It really is that bad of a problem. > >>>>> > >>>>> The causes of Solr crashes (that I can think of) include the > >> following. > >>> I believe I have listed these four options from most likely to least > >> likely: > >>>>> > >>>>> * Java OutOfMemoryError exceptions. On non-windows systems, the > >>> "bin/solr" script starts Solr with an option that results in Solr's > death > >>> anytime one of these exceptions occurs. We do this because program > >>> operation is indeterminate and completely unpredictable when OOME > occurs, > >>> so it's far safer to stop running. That exception can be caused by > >> several > >>> things, some of which actually do not involve memory at all. If you're > >>> running on Windows via the bin\solr.cmd command, then this will not > >> happen > >>> ... but OOME could still cause a crash, because as I already mentioned, > >>> program operation is unpredictable when OOME occurs. > >>>>> > >>>>> * The OS kills Solr because system memory is completely exhausted and > >>> Solr is the process using the most memory. Linux calls this the > >>> "oom-killer" ... I am pretty sure something like it exists on most > >>> operating systems. > >>>>> > >>>>> * Corruption somewhere in the system. Could be in Java, the OS, > Solr, > >>> or data used by any of those. > >>>>> > >>>>> * A very serious bug in Solr's code that we haven't discovered yet. > >>>>> > >>>>> I included that last one simply for completeness. A bug that causes > a > >>> crash *COULD* exist, but as of right now, we have not seen any > supporting > >>> evidence. > >>>>> > >>>>> My guess is that Java OutOfMemoryError is the cause here, but I can't > >>> be certain. If that is happening, then some resource (which might not > be > >>> memory) is fully depleted. We would need to see the full > >> OutOfMemoryError > >>> exception in order to determine why it is happening. Sometimes the > >>> exception is logged in solr.log, sometimes it isn't. We cannot predict > >>> what part of the code will be running when OOME occurs, so it would be > >>> nearly impossible for us to guarantee logging. OOME can happen > ANYWHERE > >> - > >>> even in code that the compiler thinks is immune to exceptions. > >>>>> > >>>>> Side note to fellow committers: I wonder if we should implement an > >>> uncaught exception handler in Solr. I have found in my own programs > that > >>> it helps figure out thorny problems. And while I am on the subject of > >>> handlers that might not be general knowledge, I didn't find a shutdown > >> hook > >>> or a security manager outside of tests. > >>>>> > >>>>> Thanks, > >>>>> Shawn > >>> > >>> > >> > >
