Hi,
Maybe https://github.com/sematext/solr-diagnostics can be of use?
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
On Mon, Jun 29, 2020 at 3:46 PM Erick Erickson
wrote:
> Really look at your cache
Really look at your cache size settings.
This is to eliminate this scenario:
- your cache sizes are very large
- when you looked and the memory was 9G, you also had a lot of cache entries
- there was a commit, which threw out the old cache and reduced your cache size
This is frankly kind of
On Mon, Jun 29, 2020 at 3:13 PM Erick Erickson
wrote:
> ps aux | grep solr
>
[solr@faspbsy0002 database-backups]$ ps aux | grep solr
solr 72072 1.6 33.4 22847816 10966476 ? Sl 13:35 1:36 java
-server -Xms16g -Xmx16g -XX:+UseG1GC -XX:+ParallelRefProcEnabled
-XX:G1HeapRegionSize=8m
Maybe you can identify in the logfiles some critical queries?
What is the total size of the index?
What client are you using on the web app side? Are you reusing clients or
create one new for every query.
> Am 29.06.2020 um 21:14 schrieb Ryan W :
>
> On Mon, Jun 29, 2020 at 1:49 PM David
On Mon, Jun 29, 2020 at 1:49 PM David Hastings
wrote:
> little nit picky note here, use 31gb, never 32.
Good to know.
Just now I got this output from bin/solr status:
"solr_home":"/opt/solr/server/solr",
"version":"7.7.2 d4c30fc2856154f2c1fefc589eb7cd070a415b94 - janhoy -
2019-05-28
ps aux | grep solr
should show you all the parameters Solr is running with, as would the
admin screen. You should see something like:
-XX:OnOutOfMemoryError=your_solr_directory/bin/oom_solr.sh
And there should be some logs laying around if that was the case
similar to:
little nit picky note here, use 31gb, never 32.
On Mon, Jun 29, 2020 at 1:45 PM Ryan W wrote:
> It figures it would happen again a couple hours after I suggested the issue
> might be resolved. Just now, Solr stopped running. I cleared the cache in
> my app a couple times around the time that
It figures it would happen again a couple hours after I suggested the issue
might be resolved. Just now, Solr stopped running. I cleared the cache in
my app a couple times around the time that it happened, so perhaps that was
somehow too taxing for the server. However, I've never allocated so
The thing that’s unsettling about this is that assuming you were hitting OOMs,
and were running the OOM-killer script, you _should_ have had very clear
evidence that that was the cause.
If you were not running the killer script, the apologies for not asking about
that
in the first place. Java’s
sometimes just throwing money/ram/ssd at the problem is just the best
answer.
On Mon, Jun 29, 2020 at 11:38 AM Ryan W wrote:
> Thanks everyone. Just to give an update on this issue, I bumped the RAM
> available to Solr up to 16GB a couple weeks ago, and haven’t had any
> problem since.
>
>
> On
Thanks everyone. Just to give an update on this issue, I bumped the RAM
available to Solr up to 16GB a couple weeks ago, and haven’t had any
problem since.
On Tue, Jun 16, 2020 at 1:00 PM David Hastings
wrote:
> me personally, around 290gb. as much as we could shove into them
>
> On Tue, Jun
me personally, around 290gb. as much as we could shove into them
On Tue, Jun 16, 2020 at 12:44 PM Erick Erickson
wrote:
> How much physical RAM? A rule of thumb is that you should allocate no more
> than 25-50 percent of the total physical RAM to Solr. That's cumulative,
> i.e. the sum of the
How much physical RAM? A rule of thumb is that you should allocate no more
than 25-50 percent of the total physical RAM to Solr. That's cumulative,
i.e. the sum of the heap allocations across all your JVMs should be below
that percentage. See Uwe Schindler's mmapdirectiry blog...
Shot in the
To add to this, i generally have solr start with this:
-Xms31000m-Xmx31000m
and the only other thing that runs on them are maria db gallera cluster
nodes that are not in use (aside from replication)
the 31gb is not an accident either, you dont want 32gb.
On Tue, Jun 16, 2020 at 11:26 AM Shawn
On 6/11/2020 11:52 AM, Ryan W wrote:
I will check "dmesg" first, to find out any hardware error message.
[1521232.781801] Out of memory: Kill process 117529 (httpd) score 9 or
sacrifice child
[1521232.782908] Killed process 117529 (httpd), UID 48, total-vm:675824kB,
anon-rss:181844kB,
o=3 -XX:NewSize=134217728
>> >>>> -XX:NumberOfGCLogFiles=9 -XX:OldPLABSize=16 -XX:OldSize=402653184
>> >>>> -XX:-OmitStackTraceInFastThrow
>> >>>> -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983
>> >>> /opt/solr/server/logs
>&g
olr.sh 8983
> >>> /opt/solr/server/logs
> >>>> -XX:ParallelGCThreads=4 -XX:+ParallelRefProcEnabled
> >>>> -XX:PretenureSizeThreshold=67108864 -XX:+PrintGC
> >>>> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
> >>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtG
108864 -XX:+PrintGC
>>>> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
>>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
>>>> -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
>>>> -XX:TargetSurvivorRatio=90 -XX:ThreadStackSize=2
>> > -XX:+UseParNewGC
>> >
>> > Buried in there I see "OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh".
>> But I
>> > think this is just a setting that indicates what to do in case of an
>> OOM.
>> > And if I look in that oom_sol
MemoryError=/opt/solr/bin/oom_solr.sh".
> But I
> > think this is just a setting that indicates what to do in case of an OOM.
> > And if I look in that oom_solr.sh file, I see it would write an entry to
> a
> > solr_oom_kill log. And there is no such log in the logs direct
rver,
>> for instance, top, vmstat, lsof, iostat ... or simply install some nice
>> free monitoring tool into this system, like monit, monitorix, nagios.
>> Good luck!
>>
>>
>> From: Ryan W
>> Sent: Thursday, June 11, 202
y install some nice
> free monitoring tool into this system, like monit, monitorix, nagios.
> Good luck!
>
>
> From: Ryan W
> Sent: Thursday, June 11, 2020 2:13 AM
> To: solr-user@lucene.apache.org
> Subject: Re: How to determine why solr stop
On 6/10/2020 12:13 PM, Ryan W wrote:
People keep suggesting I check the logs for errors. What do those errors
look like? Does anyone have examples of the text of a Solr oom error? Or
the text of any other errors I should be looking for the next time solr
fails? Are there phrases I should
Good luck!
From: Ryan W
Sent: Thursday, June 11, 2020 2:13 AM
To: solr-user@lucene.apache.org
Subject: Re: How to determine why solr stops running?
Hi all,
People keep suggesting I check the logs for errors. What do those errors
look like? Does anyone have examples of the text of
Hi all,
People keep suggesting I check the logs for errors. What do those errors
look like? Does anyone have examples of the text of a Solr oom error? Or
the text of any other errors I should be looking for the next time solr
fails? Are there phrases I should grep for in the logs? Should I
To add to what Dave said, if you have a particular machine that’s prone to
suddenly stopping, that’s usually a red flag that you should seriously
think about hardware issues.
If the problem strikes different machines, then I agree with Shawn that
the first thing I’d be suspicious of is OOM
I’ll add that whenever I’ve had a solr instance shut down, for me it’s been a
hardware failure. Either the ram or the disk got a “glitch” and both of these
are relatively fragile and wear and tear type parts of the machine, and should
be expected to fail and be replaced from time to time. Solr
On 5/14/2020 7:22 AM, Ryan W wrote:
I manage a site where solr has stopped running a couple times in the past
week. The server hasn't been rebooted, so that's not the reason. What else
causes solr to stop running? How can I investigate why this is happening?
Any situation where Solr stops
I assumed it does, based on your description. If you installed it as a service
(systemd), then systemd can start the service again if it fails. (something
like Restart=always in your [Service] definition).
But if it doesn’t restart automatically now, I think it’s easier to
troubleshoot: just
"If Solr auto-restarts"
It doesn't auto-restart. Is there some auto-restart functionality? I'm
not aware of that.
On Mon, Jun 8, 2020 at 7:10 AM Radu Gheorghe
wrote:
> Hi Ryan,
>
> If Solr auto-restarts, I suppose it's systemd doing that. When it restarts
> the Solr service, systemd should
Hi Ryan,
If Solr auto-restarts, I suppose it's systemd doing that. When it restarts
the Solr service, systemd should log this (maybe somethibg like: journalctl
--no-pager | grep -i solr).
Then you can go in your Solr logs and check what happened right before that
time. Also, check system logs
Happened again today. Solr stopped running. Apache hasn't stopped in 10
days, so this is not due to a server reboot.
Solr is not being run with the oom-killer. And when I grep for ERROR in
the logs, there is nothing from today.
On Mon, May 18, 2020 at 3:15 PM James Greene
wrote:
> I usually
I usually do a combination of grepping for ERROR in solr logs and checking
journalctl to see if an external program may have killed the process.
Cheers,
/
* James Austin Greene
* www.jamesaustingreene.com
* 336-lol-nerd
ps aux | grep solr
on a *.nix system will show you all the runtime parameters.
> On May 18, 2020, at 12:46 PM, Ryan W wrote:
>
> Is there a config file containing the start params? I run solr like...
>
> bin/solr start
>
> I have not seen anything in the logs that seems informative. When I
Is there a config file containing the start params? I run solr like...
bin/solr start
I have not seen anything in the logs that seems informative. When I grep in
the logs directory for 'memory', I see nothing besides a couple entries
like...
2020-05-14 13:05:56.155 INFO (main) [ ]
Probably, but check that you are running with the oom-killer, it'll be in
your start params.
But absent that, something external will be the culprit, Solr doesn't stop
by itself. Do look at the Solr log once things stop, it should show if
someone or something stopped it.
On Mon, May 18, 2020,
I don't see any log file with "oom" in the file name. Does that mean there
hasn't been an out-of-memory issue? Thanks.
On Thu, May 14, 2020 at 10:05 AM James Greene
wrote:
> Check the log for for an OOM crash. Fatal exceptions will be in the main
> solr log and out of memory errors will be
Check the log for for an OOM crash. Fatal exceptions will be in the main
solr log and out of memory errors will be in their own -oom log.
I've encountered quite a few solr crashes and usually it's when there's a
threshold of concurrent users and/or indexing happening.
On Thu, May 14, 2020,
38 matches
Mail list logo
