RE: Script to check if solr is running

[Dunigan, Craig A.]

I agree with the systemd guys if you’re unfamiliar with scripting this sort of 
thing.  I’d wind up with piping through awk and grep and the like, which is as 
clear as mud if you don’t already know it.  Might as well learn and use the 
modern tools if you can.  We have an old-school hard division between sysadmins 
and app admins, so we don’t like to play with their toys.  /etc/rc.appstart and 
/etc/rc.appstop owned by the app account is the standard way here, and we build 
our own service monitoring.  And now that I write that all out, it really does 
look kinda clunky, doesn’t it?  Better to do it The Right Way©.

From: Walter Underwood 
Sent: Monday, June 8, 2020 11:30 AM
To: solr-user@lucene.apache.org
Subject: Re: Script to check if solr is running

WARNING: This email originated outside of Lands’ End. Please be on the lookout 
for phishing scams and do not open attachments or click links from people you 
do not know..

I could write a script, too, though I’d do it with straight shell code. But 
then I’d have to test it, check it in somewhere, document it for ops, install 
it, ...

Instead, when we switch from monit, I'll start with one of these systemd 
configs.

https://gist.github.com/hammady/3d7b5964c7b0f90997865ebef40bf5e1
 
>
https://netgen.io/blog/keeping-apache-solr-up-and-running-on-ez-platform-setup
 
>
https://issues.apache.org/jira/browse/SOLR-14410
 
>

Why have a cold backup and then switch? Every time I see that config, I wonder 
why people don’t have both servers live behind a load balancer. How do you know 
the cold server will work?

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/ (my blog)

> On Jun 8, 2020, at 9:20 AM, Dave 
> mailto:hastings.recurs...@gmail.com>> wrote:
>
> A simple Perl script would be able to cover this, I have a cron job Perl 
> script that does a search with an expected result, if the result isn’t there 
> it fails over to a backup search server, sends me an email, and I fix what’s 
> wrong. The backup search server is a direct clone of the live server and just 
> as strong, no interruption (aside from the five minute window)
>
> If you need a hand with this I’d gladly help, everything I run is Linux based 
> but it’s a simple curl command and server switch on failure.
>
>> On Jun 8, 2020, at 12:14 PM, Jörn Franke 
>> mailto:jornfra...@gmail.com>> wrote:
>>
>> Use the solution described by Walter. This allows you to automatically 
>> restart in case of failure and is also cleaner than defining a cronjob. 
>> Otherwise This would be another dependency one needs to keep in mind - means 
>> if there is an issue and someone does not know the system the person has to 
>> look at different places which never is good
>>
>>> Am 04.06.2020 um 18:36 schrieb Ryan W 
>>> mailto:rya...@gmail.com>>:
>>>
>>> Does anyone have a script that checks if solr is running and then starts it
>>> if it isn't running? Occasionally my solr stops running even if there has
>>> been no Apache restart. I haven't been able to determine the root cause,
>>> so the next best thing might be to check every 15 minutes or so if it's
>>> running and run it if it has stopped.
>>>
>>> Thanks.

Re: Script to check if solr is running

[David Hastings]

>
> Why have a cold backup and then switch?
>

my current set up is:
1. master indexer
2. master slave on a release/commit basis
3. 3 live slave searching nodes in two data different centers


the three live nodes are in front of nginx load balancing and they are
mostly hot but not all of them, i found that having all load into one made
the performance significantly better, but if one of them goes down theres a
likelihood that the other two went with it, they are also part of a
mysql gallera cluster and it has a possibility of going down (innodb can be
annoying), so the script will go through all three of the live slaves until
it has to fall back to the master slave, i know the cold master will work,
mostly out of faith, but if i lose four servers all at the same time, i
have larger problems to worry about than searching.

just adaptation over time, I cant say its the best set up but i can say it
operates pretty well, very well speed wise keeping one searcher super hot
with two clones ready to jump in if needed



On Mon, Jun 8, 2020 at 12:30 PM Walter Underwood 
wrote:

> I could write a script, too, though I’d do it with straight shell code.
> But then I’d have to test it, check it in somewhere, document it for ops,
> install it, ...
>
> Instead, when we switch from monit, I'll start with one of these systemd
> configs.
>
> https://gist.github.com/hammady/3d7b5964c7b0f90997865ebef40bf5e1 <
> https://gist.github.com/hammady/3d7b5964c7b0f90997865ebef40bf5e1>
>
> https://netgen.io/blog/keeping-apache-solr-up-and-running-on-ez-platform-setup
> <
> https://netgen.io/blog/keeping-apache-solr-up-and-running-on-ez-platform-setup
> >
> https://issues.apache.org/jira/browse/SOLR-14410 <
> https://issues.apache.org/jira/browse/SOLR-14410>
>
> Why have a cold backup and then switch? Every time I see that config, I
> wonder why people don’t have both servers live behind a load balancer. How
> do you know the cold server will work?
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Jun 8, 2020, at 9:20 AM, Dave  wrote:
> >
> > A simple Perl script would be able to cover this, I have a cron job Perl
> script that does a search with an expected result, if the result isn’t
> there it fails over to a backup search server, sends me an email, and I fix
> what’s wrong. The backup search server is a direct clone of the live server
> and just as strong, no interruption (aside from the five minute window)
> >
> > If you need a hand with this I’d gladly help, everything I run is Linux
> based but it’s a simple curl command and server switch on failure.
> >
> >> On Jun 8, 2020, at 12:14 PM, Jörn Franke  wrote:
> >>
> >> Use the solution described by Walter. This allows you to automatically
> restart in case of failure and is also cleaner than defining a cronjob.
> Otherwise This would be another dependency one needs to keep in mind -
> means if there is an issue and someone does not know the system the person
> has to look at different places which never is good
> >>
> >>> Am 04.06.2020 um 18:36 schrieb Ryan W :
> >>>
> >>> Does anyone have a script that checks if solr is running and then
> starts it
> >>> if it isn't running?  Occasionally my solr stops running even if there
> has
> >>> been no Apache restart.  I haven't been able to determine the root
> cause,
> >>> so the next best thing might be to check every 15 minutes or so if it's
> >>> running and run it if it has stopped.
> >>>
> >>> Thanks.
>
>

Re: Script to check if solr is running

[Walter Underwood]

I could write a script, too, though I’d do it with straight shell code. But 
then I’d have to test it, check it in somewhere, document it for ops, install 
it, ...

Instead, when we switch from monit, I'll start with one of these systemd 
configs.

https://gist.github.com/hammady/3d7b5964c7b0f90997865ebef40bf5e1 

https://netgen.io/blog/keeping-apache-solr-up-and-running-on-ez-platform-setup 

https://issues.apache.org/jira/browse/SOLR-14410 


Why have a cold backup and then switch? Every time I see that config, I wonder 
why people don’t have both servers live behind a load balancer. How do you know 
the cold server will work?

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Jun 8, 2020, at 9:20 AM, Dave  wrote:
> 
> A simple Perl script would be able to cover this, I have a cron job Perl 
> script that does a search with an expected result, if the result isn’t there 
> it fails over to a backup search server, sends me an email, and I fix what’s 
> wrong. The backup search server is a direct clone of the live server and just 
> as strong, no interruption (aside from the five minute window) 
> 
> If you need a hand with this I’d gladly help, everything I run is Linux based 
> but it’s a simple curl command and server switch on failure. 
> 
>> On Jun 8, 2020, at 12:14 PM, Jörn Franke  wrote:
>> 
>> Use the solution described by Walter. This allows you to automatically 
>> restart in case of failure and is also cleaner than defining a cronjob. 
>> Otherwise This would be another dependency one needs to keep in mind - means 
>> if there is an issue and someone does not know the system the person has to 
>> look at different places which never is good 
>> 
>>> Am 04.06.2020 um 18:36 schrieb Ryan W :
>>> 
>>> Does anyone have a script that checks if solr is running and then starts it
>>> if it isn't running?  Occasionally my solr stops running even if there has
>>> been no Apache restart.  I haven't been able to determine the root cause,
>>> so the next best thing might be to check every 15 minutes or so if it's
>>> running and run it if it has stopped.
>>> 
>>> Thanks.

Re: Script to check if solr is running

[Dave]

A simple Perl script would be able to cover this, I have a cron job Perl script 
that does a search with an expected result, if the result isn’t there it fails 
over to a backup search server, sends me an email, and I fix what’s wrong. The 
backup search server is a direct clone of the live server and just as strong, 
no interruption (aside from the five minute window) 

If you need a hand with this I’d gladly help, everything I run is Linux based 
but it’s a simple curl command and server switch on failure. 

> On Jun 8, 2020, at 12:14 PM, Jörn Franke  wrote:
> 
> Use the solution described by Walter. This allows you to automatically 
> restart in case of failure and is also cleaner than defining a cronjob. 
> Otherwise This would be another dependency one needs to keep in mind - means 
> if there is an issue and someone does not know the system the person has to 
> look at different places which never is good 
> 
>> Am 04.06.2020 um 18:36 schrieb Ryan W :
>> 
>> Does anyone have a script that checks if solr is running and then starts it
>> if it isn't running?  Occasionally my solr stops running even if there has
>> been no Apache restart.  I haven't been able to determine the root cause,
>> so the next best thing might be to check every 15 minutes or so if it's
>> running and run it if it has stopped.
>> 
>> Thanks.

Re: Script to check if solr is running

[Jörn Franke]

Use the solution described by Walter. This allows you to automatically restart 
in case of failure and is also cleaner than defining a cronjob. Otherwise This 
would be another dependency one needs to keep in mind - means if there is an 
issue and someone does not know the system the person has to look at different 
places which never is good 

> Am 04.06.2020 um 18:36 schrieb Ryan W :
> 
> Does anyone have a script that checks if solr is running and then starts it
> if it isn't running?  Occasionally my solr stops running even if there has
> been no Apache restart.  I haven't been able to determine the root cause,
> so the next best thing might be to check every 15 minutes or so if it's
> running and run it if it has stopped.
> 
> Thanks.

Re: Script to check if solr is running

[Ryan W]

"A simple cronjob with /bin/solr status and /bin/solr start should do the trick."

I don't know what that would look like.  Wouldn't the job have to check the
status and only give the start command if solr isn't running?  I don't
think it's possible to put logic in a cron job. I think it would have to be
in a script, with a cron job to run the script.  I've never had cause to
write such a script, though, so I don't know how it's done.


On Fri, Jun 5, 2020 at 11:19 AM Dunigan, Craig A. <
craig.duni...@landsend.com> wrote:

> A simple cronjob with /bin/solr status and  directory>/bin/solr start should do the trick.  There must be a Windows
> equivalent if that’s what you’re using.
>
> From: Ryan W 
> Sent: Thursday, June 4, 2020 11:39 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Script to check if solr is running
>
> WARNING: This email originated outside of Lands’ End. Please be on the
> lookout for phishing scams and do not open attachments or click links from
> people you do not know..
>
> Or is it not much overhead to give the command to start solr if it is
> already running? Maybe it's not necessary to check if it's running? Is
> there any downside to giving the start command every 15 minutes or so
> whether it is running or not?
>
> Thanks.
>
> On Thu, Jun 4, 2020 at 12:36 PM Ryan W  rya...@gmail.com>> wrote:
>
> > Does anyone have a script that checks if solr is running and then starts
> > it if it isn't running? Occasionally my solr stops running even if there
> > has been no Apache restart. I haven't been able to determine the root
> > cause, so the next best thing might be to check every 15 minutes or so if
> > it's running and run it if it has stopped.
> >
> > Thanks.
> >
>

RE: Script to check if solr is running

[Dunigan, Craig A.]

A simple cronjob with /bin/solr status and /bin/solr start should do the trick.  There must be a Windows 
equivalent if that’s what you’re using.

From: Ryan W 
Sent: Thursday, June 4, 2020 11:39 AM
To: solr-user@lucene.apache.org
Subject: Re: Script to check if solr is running

WARNING: This email originated outside of Lands’ End. Please be on the lookout 
for phishing scams and do not open attachments or click links from people you 
do not know..

Or is it not much overhead to give the command to start solr if it is
already running? Maybe it's not necessary to check if it's running? Is
there any downside to giving the start command every 15 minutes or so
whether it is running or not?

Thanks.

On Thu, Jun 4, 2020 at 12:36 PM Ryan W 
mailto:rya...@gmail.com>> wrote:

> Does anyone have a script that checks if solr is running and then starts
> it if it isn't running? Occasionally my solr stops running even if there
> has been no Apache restart. I haven't been able to determine the root
> cause, so the next best thing might be to check every 15 minutes or so if
> it's running and run it if it has stopped.
>
> Thanks.
>

Re: Script to check if solr is running

[Walter Underwood]

Most Linux distros are using systemd to manage server processes.

https://en.wikipedia.org/wiki/Systemd 

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Jun 5, 2020, at 8:08 AM, Mark H. Wood  wrote:
> 
> On Thu, Jun 04, 2020 at 12:36:30PM -0400, Ryan W wrote:
>> Does anyone have a script that checks if solr is running and then starts it
>> if it isn't running?  Occasionally my solr stops running even if there has
>> been no Apache restart.  I haven't been able to determine the root cause,
>> so the next best thing might be to check every 15 minutes or so if it's
>> running and run it if it has stopped.
> 
> I've used Monit for things that must be kept running:
> 
>  https://mmonit.com/monit/
> 
> -- 
> Mark H. Wood
> Lead Technology Analyst
> 
> University Library
> Indiana University - Purdue University Indianapolis
> 755 W. Michigan Street
> Indianapolis, IN 46202
> 317-274-0749
> www.ulib.iupui.edu

Re: Script to check if solr is running

[Mark H. Wood]

On Thu, Jun 04, 2020 at 12:36:30PM -0400, Ryan W wrote:
> Does anyone have a script that checks if solr is running and then starts it
> if it isn't running?  Occasionally my solr stops running even if there has
> been no Apache restart.  I haven't been able to determine the root cause,
> so the next best thing might be to check every 15 minutes or so if it's
> running and run it if it has stopped.

I've used Monit for things that must be kept running:

  https://mmonit.com/monit/

-- 
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu


signature.asc
Description: PGP signature

Re: Script to check if solr is running

[Vishal Vaibhav]

Fixing the root cause would certainly be the best thing.However if you
still wanna tread that path, you can do a healthchek on admin endpoint and
do the thing. A simple cron job would do trick

On Thu, 4 Jun 2020 at 10:09 PM, Ryan W  wrote:

> Or is it not much overhead to give the command to start solr if it is
> already running?  Maybe it's not necessary to check if it's running?  Is
> there any downside to giving the start command every 15 minutes or so
> whether it is running or not?
>
> Thanks.
>
> On Thu, Jun 4, 2020 at 12:36 PM Ryan W  wrote:
>
> > Does anyone have a script that checks if solr is running and then starts
> > it if it isn't running?  Occasionally my solr stops running even if there
> > has been no Apache restart.  I haven't been able to determine the root
> > cause, so the next best thing might be to check every 15 minutes or so if
> > it's running and run it if it has stopped.
> >
> > Thanks.
> >
>

Re: Script to check if solr is running

[Ryan W]

Or is it not much overhead to give the command to start solr if it is
already running?  Maybe it's not necessary to check if it's running?  Is
there any downside to giving the start command every 15 minutes or so
whether it is running or not?

Thanks.

On Thu, Jun 4, 2020 at 12:36 PM Ryan W  wrote:

> Does anyone have a script that checks if solr is running and then starts
> it if it isn't running?  Occasionally my solr stops running even if there
> has been no Apache restart.  I haven't been able to determine the root
> cause, so the next best thing might be to check every 15 minutes or so if
> it's running and run it if it has stopped.
>
> Thanks.
>