Re: Strange issue wherein cassandra not being started from cron

[Alain RODRIGUEZ]

Hi Ajay, honestly I would try to fix the main issue:

Sometimes, the cassandra-process gets killed (reason unknown as of now).


As focusing on how to restart Apache Cassandra every minute  sounds like a
wrong approach to me:

Adding this in cron would at least ensure that the maximum downtime is 59
> seconds (till the time root-cause of cassandra-crashing is known).


What happens if 2,3 or 10 nodes go down at once? Also, hints due to this
issue, read_repairs and other anti-entropy system will continuously be
triggered. It doesn't sound healthy, predictable or even a working solution.

When Cassandra stops it can be due to a heap OOM (see system.log or gc.log)
or a native OOM (see kernel / system logs) or some other stuff logged in
system.log. "nodetool tpstats" is also often useful. If you want to fix
this we would probably be able to help you with it.

About your cron issue, have you tried using sudo? As it is a quick-fix, it
probably can be dirty. Just make sure new sstables are being written with
the proper user ("cassandra"  and not "root"). But again, I would not go
that path, but rather fix the issue in Cassandra without loosing 1 minute
with crontab.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2017-01-12 7:55 GMT+01:00 Benjamin Roth :

> Yes, but it is legitimate to supervise and monitor nodes. I only doubt
> that cron is the best tool for it.
>
> 2017-01-12 7:42 GMT+01:00 Martin Schröder :
>
>> 2017-01-12 6:12 GMT+01:00 Ajay Garg :
>> > Sometimes, the cassandra-process gets killed (reason unknown as of now).
>>
>> That's why you have a cluster of them.
>>
>> Best
>>Martin
>>
>
>
>
> --
> Benjamin Roth
> Prokurist
>
> Jaumo GmbH · www.jaumo.com
> Wehrstraße 46 · 73035 Göppingen · Germany
> Phone +49 7161 304880-6 <+49%207161%203048806> · Fax +49 7161 304880-1
> <+49%207161%203048801>
> AG Ulm · HRB 731058 · Managing Director: Jens Kammerer
>

Re: Strange issue wherein cassandra not being started from cron

[Benjamin Roth]

Yes, but it is legitimate to supervise and monitor nodes. I only doubt that
cron is the best tool for it.

2017-01-12 7:42 GMT+01:00 Martin Schröder :

> 2017-01-12 6:12 GMT+01:00 Ajay Garg :
> > Sometimes, the cassandra-process gets killed (reason unknown as of now).
>
> That's why you have a cluster of them.
>
> Best
>Martin
>



-- 
Benjamin Roth
Prokurist

Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer

Re: Strange issue wherein cassandra not being started from cron

[Martin Schröder]

2017-01-12 6:12 GMT+01:00 Ajay Garg :
> Sometimes, the cassandra-process gets killed (reason unknown as of now).

That's why you have a cluster of them.

Best
   Martin

Re: Strange issue wherein cassandra not being started from cron

[Benjamin Roth]

I think you should take a look at supervisord or sth similar. This is a
much more reliable solution than using crons.

Am 12.01.2017 06:12 schrieb "Ajay Garg" :



On Wed, Jan 11, 2017 at 8:29 PM, Martin Schröder  wrote:

> 2017-01-11 15:42 GMT+01:00 Ajay Garg :
> > Tried everything.
>
> Then try
>service cassandra start
> or
>systemctl start cassandra
>
> You still haven't explained to us why you want to start cassandra every
> minute.
>

Hi Martin.

Sometimes, the cassandra-process gets killed (reason unknown as of now).
Doing a manual "service cassandra start" works then.

Adding this in cron would at least ensure that the maximum downtime is 59
seconds (till the time root-cause of cassandra-crashing is known).



>
> Best
>Martin
>



-- 
Regards,
Ajay

Re: Strange issue wherein cassandra not being started from cron

[Ajay Garg]

Hi Hannu.

On Wed, Jan 11, 2017 at 8:31 PM, Hannu Kröger  wrote:

> One possible reason is that cassandra process gets different user when run
> differently. Check who owns the data files and check also what gets written
> into the /var/log/cassandra/system.log (or whatever that was).
>

Absolutely nothing gets written to /var/log/cassandra/system.log (when
trying to invoke cassandra via cron).


>
> Hannu
>
>
> On 11 Jan 2017, at 16.42, Ajay Garg  wrote:
>
> Tried everything.
> Every other cron job/script I try works, just the cassandra-service does
> not.
>
> On Wed, Jan 11, 2017 at 8:51 AM, Edward Capriolo 
> wrote:
>
>>
>>
>> On Tuesday, January 10, 2017, Jonathan Haddad  wrote:
>>
>>> Last I checked, cron doesn't load the same, full environment you see
>>> when you log in. Also, why put Cassandra on a cron?
>>> On Mon, Jan 9, 2017 at 9:47 PM Bhuvan Rawal  wrote:
>>>
 Hi Ajay,

 Have you had a look at cron logs? - mine is in path /var/log/cron

 Thanks & Regards,

 On Tue, Jan 10, 2017 at 9:45 AM, Ajay Garg 
 wrote:

> Hi All.
>
> Facing a very weird issue, wherein the command
>
> */etc/init.d/cassandra start*
>
> causes cassandra to start when the command is run from command-line.
>
>
> However, if I put the above as a cron job
>
>
>
> ** * * * * /etc/init.d/cassandra start*
> cassandra never starts.
>
>
> I have checked, and "cron" service is running.
>
>
> Any ideas what might be wrong?
> I am pasting the cassandra script for brevity.
>
>
> Thanks and Regards,
> Ajay
>
>
> 
> 
> #! /bin/sh
> ### BEGIN INIT INFO
> # Provides:  cassandra
> # Required-Start:$remote_fs $network $named $time
> # Required-Stop: $remote_fs $network $named $time
> # Should-Start:  ntp mdadm
> # Should-Stop:   ntp mdadm
> # Default-Start: 2 3 4 5
> # Default-Stop:  0 1 6
> # Short-Description: distributed storage system for structured data
> # Description:   Cassandra is a distributed (peer-to-peer) system
> for
> #the management and storage of structured data.
> ### END INIT INFO
>
> # Author: Eric Evans 
>
> DESC="Cassandra"
> NAME=cassandra
> PIDFILE=/var/run/$NAME/$NAME.pid
> SCRIPTNAME=/etc/init.d/$NAME
> CONFDIR=/etc/cassandra
> WAIT_FOR_START=10
> CASSANDRA_HOME=/usr/share/cassandra
> FD_LIMIT=10
>
> [ -e /usr/share/cassandra/apache-cassandra.jar ] || exit 0
> [ -e /etc/cassandra/cassandra.yaml ] || exit 0
> [ -e /etc/cassandra/cassandra-env.sh ] || exit 0
>
> # Read configuration variable file if it is present
> [ -r /etc/default/$NAME ] && . /etc/default/$NAME
>
> # Read Cassandra environment file.
> . /etc/cassandra/cassandra-env.sh
>
> if [ -z "$JVM_OPTS" ]; then
> echo "Initialization failed; \$JVM_OPTS not set!" >&2
> exit 3
> fi
>
> export JVM_OPTS
>
> # Export JAVA_HOME, if set.
> [ -n "$JAVA_HOME" ] && export JAVA_HOME
>
> # Load the VERBOSE setting and other rcS variables
> . /lib/init/vars.sh
>
> # Define LSB log_* functions.
> # Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
> . /lib/lsb/init-functions
>
> #
> # Function that returns 0 if process is running, or nonzero if not.
> #
> # The nonzero value is 3 if the process is simply not running, and 1
> if the
> # process is not running but the pidfile exists (to match the exit
> codes for
> # the "status" command; see LSB core spec 3.1, section 20.2)
> #
> CMD_PATT="cassandra.+CassandraDaemon"
> is_running()
> {
> if [ -f $PIDFILE ]; then
> pid=`cat $PIDFILE`
> grep -Eq "$CMD_PATT" "/proc/$pid/cmdline" 2>/dev/null &&
> return 0
> return 1
> fi
> return 3
> }
> #
> # Function that starts the daemon/service
> #
> do_start()
> {
> # Return
> #   0 if daemon has been started
> #   1 if daemon was already running
> #   2 if daemon could not be started
>
> ulimit -l unlimited
> ulimit -n "$FD_LIMIT"
>
> cassandra_home=`getent passwd cassandra | awk -F ':' '{ print $6;
> }'`
> heap_dump_f="$cassandra_home/java_`date +%s`.hprof"
> error_log_f="$cassandra_home/hs_err_`date +%s`.log"
>
> [ -e `dirname "$PIDFILE"` ] || \
> install -d -ocassandra -gcassandra -m755 `dirname $PIDFILE`
>
>
>
> start-stop-daemon -S -c 

Re: Strange issue wherein cassandra not being started from cron

[Ajay Garg]

On Wed, Jan 11, 2017 at 8:29 PM, Martin Schröder  wrote:

> 2017-01-11 15:42 GMT+01:00 Ajay Garg :
> > Tried everything.
>
> Then try
>service cassandra start
> or
>systemctl start cassandra
>
> You still haven't explained to us why you want to start cassandra every
> minute.
>

Hi Martin.

Sometimes, the cassandra-process gets killed (reason unknown as of now).
Doing a manual "service cassandra start" works then.

Adding this in cron would at least ensure that the maximum downtime is 59
seconds (till the time root-cause of cassandra-crashing is known).



>
> Best
>Martin
>



-- 
Regards,
Ajay

Re: Strange issue wherein cassandra not being started from cron

[Hannu Kröger]

One possible reason is that cassandra process gets different user when run 
differently. Check who owns the data files and check also what gets written 
into the /var/log/cassandra/system.log (or whatever that was).

Hannu

> On 11 Jan 2017, at 16.42, Ajay Garg  wrote:
> 
> Tried everything.
> Every other cron job/script I try works, just the cassandra-service does not.
> 
> On Wed, Jan 11, 2017 at 8:51 AM, Edward Capriolo  > wrote:
> 
> 
> On Tuesday, January 10, 2017, Jonathan Haddad  > wrote:
> Last I checked, cron doesn't load the same, full environment you see when you 
> log in. Also, why put Cassandra on a cron?
> On Mon, Jan 9, 2017 at 9:47 PM Bhuvan Rawal > wrote:
> Hi Ajay,
> 
> Have you had a look at cron logs? - mine is in path /var/log/cron
> 
> Thanks & Regards,
> 
> On Tue, Jan 10, 2017 at 9:45 AM, Ajay Garg > wrote:
> Hi All.
> 
> Facing a very weird issue, wherein the command
> 
> /etc/init.d/cassandra start
> 
> causes cassandra to start when the command is run from command-line.
> 
> 
> However, if I put the above as a cron job
> 
> * * * * * /etc/init.d/cassandra start
> 
> cassandra never starts.
> 
> 
> I have checked, and "cron" service is running.
> 
> 
> Any ideas what might be wrong?
> I am pasting the cassandra script for brevity.
> 
> 
> Thanks and Regards,
> Ajay
> 
> 
> 
> #! /bin/sh
> ### BEGIN INIT INFO
> # Provides:  cassandra
> # Required-Start:$remote_fs $network $named $time
> # Required-Stop: $remote_fs $network $named $time
> # Should-Start:  ntp mdadm
> # Should-Stop:   ntp mdadm
> # Default-Start: 2 3 4 5
> # Default-Stop:  0 1 6
> # Short-Description: distributed storage system for structured data
> # Description:   Cassandra is a distributed (peer-to-peer) system for
> #the management and storage of structured data.
> ### END INIT INFO
> 
> # Author: Eric Evans >
> 
> DESC="Cassandra"
> NAME=cassandra
> PIDFILE=/var/run/$NAME/$NAME.pid
> SCRIPTNAME=/etc/init.d/$NAME
> CONFDIR=/etc/cassandra
> WAIT_FOR_START=10
> CASSANDRA_HOME=/usr/share/cassandra
> FD_LIMIT=10
> 
> [ -e /usr/share/cassandra/apache-cassandra.jar ] || exit 0
> [ -e /etc/cassandra/cassandra.yaml ] || exit 0
> [ -e /etc/cassandra/cassandra-env.sh ] || exit 0
> 
> # Read configuration variable file if it is present
> [ -r /etc/default/$NAME ] && . /etc/default/$NAME
> 
> # Read Cassandra environment file.
> . /etc/cassandra/cassandra-env.sh
> 
> if [ -z "$JVM_OPTS" ]; then
> echo "Initialization failed; \$JVM_OPTS not set!" >&2
> exit 3
> fi
> 
> export JVM_OPTS
> 
> # Export JAVA_HOME, if set.
> [ -n "$JAVA_HOME" ] && export JAVA_HOME
> 
> # Load the VERBOSE setting and other rcS variables
> . /lib/init/vars.sh
> 
> # Define LSB log_* functions.
> # Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
> . /lib/lsb/init-functions
> 
> #
> # Function that returns 0 if process is running, or nonzero if not.
> #
> # The nonzero value is 3 if the process is simply not running, and 1 if the
> # process is not running but the pidfile exists (to match the exit codes for
> # the "status" command; see LSB core spec 3.1, section 20.2)
> #
> CMD_PATT="cassandra.+CassandraDaemon"
> is_running()
> {
> if [ -f $PIDFILE ]; then
> pid=`cat $PIDFILE`
> grep -Eq "$CMD_PATT" "/proc/$pid/cmdline" 2>/dev/null && return 0
> return 1
> fi
> return 3
> }
> #
> # Function that starts the daemon/service
> #
> do_start()
> {
> # Return
> #   0 if daemon has been started
> #   1 if daemon was already running
> #   2 if daemon could not be started
> 
> ulimit -l unlimited
> ulimit -n "$FD_LIMIT"
> 
> cassandra_home=`getent passwd cassandra | awk -F ':' '{ print $6; }'`
> heap_dump_f="$cassandra_home/java_`date +%s`.hprof"
> error_log_f="$cassandra_home/hs_err_`date +%s`.log"
> 
> [ -e `dirname "$PIDFILE"` ] || \
> install -d -ocassandra -gcassandra -m755 `dirname $PIDFILE`
> 
> 
> 
> start-stop-daemon -S -c cassandra -a /usr/sbin/cassandra -q -p "$PIDFILE" 
> -t >/dev/null || return 1
> 
> start-stop-daemon -S -c cassandra -a /usr/sbin/cassandra -b -p "$PIDFILE" 
> -- \
> -p "$PIDFILE" -H "$heap_dump_f" -E "$error_log_f" >/dev/null || 
> return 2
> 
> }
> 
> #
> # Function that stops the daemon/service
> #
> do_stop()
> {
> # Return
> #   0 if daemon has been stopped
> #   1 if daemon was already stopped
> #   2 if daemon could not be stopped
> #   other if a failure occurred
> start-stop-daemon -K -p "$PIDFILE" -R TERM/30/KILL/5 >/dev/null
> RET=$?
> rm -f "$PIDFILE"
> return $RET
> }
> 
> case "$1" in
>   start)
>   

Re: Strange issue wherein cassandra not being started from cron

[Martin Schröder]

2017-01-11 15:42 GMT+01:00 Ajay Garg :
> Tried everything.

Then try
   service cassandra start
or
   systemctl start cassandra

You still haven't explained to us why you want to start cassandra every minute.

Best
   Martin

Re: Strange issue wherein cassandra not being started from cron

[Ajay Garg]

Tried everything.
Every other cron job/script I try works, just the cassandra-service does
not.

On Wed, Jan 11, 2017 at 8:51 AM, Edward Capriolo 
wrote:

>
>
> On Tuesday, January 10, 2017, Jonathan Haddad  wrote:
>
>> Last I checked, cron doesn't load the same, full environment you see when
>> you log in. Also, why put Cassandra on a cron?
>> On Mon, Jan 9, 2017 at 9:47 PM Bhuvan Rawal  wrote:
>>
>>> Hi Ajay,
>>>
>>> Have you had a look at cron logs? - mine is in path /var/log/cron
>>>
>>> Thanks & Regards,
>>>
>>> On Tue, Jan 10, 2017 at 9:45 AM, Ajay Garg 
>>> wrote:
>>>
 Hi All.

 Facing a very weird issue, wherein the command

 */etc/init.d/cassandra start*

 causes cassandra to start when the command is run from command-line.


 However, if I put the above as a cron job



 ** * * * * /etc/init.d/cassandra start*
 cassandra never starts.


 I have checked, and "cron" service is running.


 Any ideas what might be wrong?
 I am pasting the cassandra script for brevity.


 Thanks and Regards,
 Ajay


 
 
 #! /bin/sh
 ### BEGIN INIT INFO
 # Provides:  cassandra
 # Required-Start:$remote_fs $network $named $time
 # Required-Stop: $remote_fs $network $named $time
 # Should-Start:  ntp mdadm
 # Should-Stop:   ntp mdadm
 # Default-Start: 2 3 4 5
 # Default-Stop:  0 1 6
 # Short-Description: distributed storage system for structured data
 # Description:   Cassandra is a distributed (peer-to-peer) system
 for
 #the management and storage of structured data.
 ### END INIT INFO

 # Author: Eric Evans 

 DESC="Cassandra"
 NAME=cassandra
 PIDFILE=/var/run/$NAME/$NAME.pid
 SCRIPTNAME=/etc/init.d/$NAME
 CONFDIR=/etc/cassandra
 WAIT_FOR_START=10
 CASSANDRA_HOME=/usr/share/cassandra
 FD_LIMIT=10

 [ -e /usr/share/cassandra/apache-cassandra.jar ] || exit 0
 [ -e /etc/cassandra/cassandra.yaml ] || exit 0
 [ -e /etc/cassandra/cassandra-env.sh ] || exit 0

 # Read configuration variable file if it is present
 [ -r /etc/default/$NAME ] && . /etc/default/$NAME

 # Read Cassandra environment file.
 . /etc/cassandra/cassandra-env.sh

 if [ -z "$JVM_OPTS" ]; then
 echo "Initialization failed; \$JVM_OPTS not set!" >&2
 exit 3
 fi

 export JVM_OPTS

 # Export JAVA_HOME, if set.
 [ -n "$JAVA_HOME" ] && export JAVA_HOME

 # Load the VERBOSE setting and other rcS variables
 . /lib/init/vars.sh

 # Define LSB log_* functions.
 # Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
 . /lib/lsb/init-functions

 #
 # Function that returns 0 if process is running, or nonzero if not.
 #
 # The nonzero value is 3 if the process is simply not running, and 1 if
 the
 # process is not running but the pidfile exists (to match the exit
 codes for
 # the "status" command; see LSB core spec 3.1, section 20.2)
 #
 CMD_PATT="cassandra.+CassandraDaemon"
 is_running()
 {
 if [ -f $PIDFILE ]; then
 pid=`cat $PIDFILE`
 grep -Eq "$CMD_PATT" "/proc/$pid/cmdline" 2>/dev/null && return
 0
 return 1
 fi
 return 3
 }
 #
 # Function that starts the daemon/service
 #
 do_start()
 {
 # Return
 #   0 if daemon has been started
 #   1 if daemon was already running
 #   2 if daemon could not be started

 ulimit -l unlimited
 ulimit -n "$FD_LIMIT"

 cassandra_home=`getent passwd cassandra | awk -F ':' '{ print $6;
 }'`
 heap_dump_f="$cassandra_home/java_`date +%s`.hprof"
 error_log_f="$cassandra_home/hs_err_`date +%s`.log"

 [ -e `dirname "$PIDFILE"` ] || \
 install -d -ocassandra -gcassandra -m755 `dirname $PIDFILE`



 start-stop-daemon -S -c cassandra -a /usr/sbin/cassandra -q -p
 "$PIDFILE" -t >/dev/null || return 1

 start-stop-daemon -S -c cassandra -a /usr/sbin/cassandra -b -p
 "$PIDFILE" -- \
 -p "$PIDFILE" -H "$heap_dump_f" -E "$error_log_f" >/dev/null ||
 return 2

 }

 #
 # Function that stops the daemon/service
 #
 do_stop()
 {
 # Return
 #   0 if daemon has been stopped
 #   1 if daemon was already stopped
 #   2 if daemon could not be stopped
 #   other if a failure occurred
 start-stop-daemon -K -p "$PIDFILE" -R TERM/30/KILL/5 >/dev/null
 RET=$?
 

Re: Strange issue wherein cassandra not being started from cron

[Edward Capriolo]

On Tuesday, January 10, 2017, Jonathan Haddad  wrote:

> Last I checked, cron doesn't load the same, full environment you see when
> you log in. Also, why put Cassandra on a cron?
> On Mon, Jan 9, 2017 at 9:47 PM Bhuvan Rawal  > wrote:
>
>> Hi Ajay,
>>
>> Have you had a look at cron logs? - mine is in path /var/log/cron
>>
>> Thanks & Regards,
>>
>> On Tue, Jan 10, 2017 at 9:45 AM, Ajay Garg > > wrote:
>>
>>> Hi All.
>>>
>>> Facing a very weird issue, wherein the command
>>>
>>> */etc/init.d/cassandra start*
>>>
>>> causes cassandra to start when the command is run from command-line.
>>>
>>>
>>> However, if I put the above as a cron job
>>>
>>>
>>>
>>> ** * * * * /etc/init.d/cassandra start*
>>> cassandra never starts.
>>>
>>>
>>> I have checked, and "cron" service is running.
>>>
>>>
>>> Any ideas what might be wrong?
>>> I am pasting the cassandra script for brevity.
>>>
>>>
>>> Thanks and Regards,
>>> Ajay
>>>
>>>
>>> 
>>> 
>>> #! /bin/sh
>>> ### BEGIN INIT INFO
>>> # Provides:  cassandra
>>> # Required-Start:$remote_fs $network $named $time
>>> # Required-Stop: $remote_fs $network $named $time
>>> # Should-Start:  ntp mdadm
>>> # Should-Stop:   ntp mdadm
>>> # Default-Start: 2 3 4 5
>>> # Default-Stop:  0 1 6
>>> # Short-Description: distributed storage system for structured data
>>> # Description:   Cassandra is a distributed (peer-to-peer) system for
>>> #the management and storage of structured data.
>>> ### END INIT INFO
>>>
>>> # Author: Eric Evans >> >
>>>
>>> DESC="Cassandra"
>>> NAME=cassandra
>>> PIDFILE=/var/run/$NAME/$NAME.pid
>>> SCRIPTNAME=/etc/init.d/$NAME
>>> CONFDIR=/etc/cassandra
>>> WAIT_FOR_START=10
>>> CASSANDRA_HOME=/usr/share/cassandra
>>> FD_LIMIT=10
>>>
>>> [ -e /usr/share/cassandra/apache-cassandra.jar ] || exit 0
>>> [ -e /etc/cassandra/cassandra.yaml ] || exit 0
>>> [ -e /etc/cassandra/cassandra-env.sh ] || exit 0
>>>
>>> # Read configuration variable file if it is present
>>> [ -r /etc/default/$NAME ] && . /etc/default/$NAME
>>>
>>> # Read Cassandra environment file.
>>> . /etc/cassandra/cassandra-env.sh
>>>
>>> if [ -z "$JVM_OPTS" ]; then
>>> echo "Initialization failed; \$JVM_OPTS not set!" >&2
>>> exit 3
>>> fi
>>>
>>> export JVM_OPTS
>>>
>>> # Export JAVA_HOME, if set.
>>> [ -n "$JAVA_HOME" ] && export JAVA_HOME
>>>
>>> # Load the VERBOSE setting and other rcS variables
>>> . /lib/init/vars.sh
>>>
>>> # Define LSB log_* functions.
>>> # Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
>>> . /lib/lsb/init-functions
>>>
>>> #
>>> # Function that returns 0 if process is running, or nonzero if not.
>>> #
>>> # The nonzero value is 3 if the process is simply not running, and 1 if
>>> the
>>> # process is not running but the pidfile exists (to match the exit codes
>>> for
>>> # the "status" command; see LSB core spec 3.1, section 20.2)
>>> #
>>> CMD_PATT="cassandra.+CassandraDaemon"
>>> is_running()
>>> {
>>> if [ -f $PIDFILE ]; then
>>> pid=`cat $PIDFILE`
>>> grep -Eq "$CMD_PATT" "/proc/$pid/cmdline" 2>/dev/null && return 0
>>> return 1
>>> fi
>>> return 3
>>> }
>>> #
>>> # Function that starts the daemon/service
>>> #
>>> do_start()
>>> {
>>> # Return
>>> #   0 if daemon has been started
>>> #   1 if daemon was already running
>>> #   2 if daemon could not be started
>>>
>>> ulimit -l unlimited
>>> ulimit -n "$FD_LIMIT"
>>>
>>> cassandra_home=`getent passwd cassandra | awk -F ':' '{ print $6; }'`
>>> heap_dump_f="$cassandra_home/java_`date +%s`.hprof"
>>> error_log_f="$cassandra_home/hs_err_`date +%s`.log"
>>>
>>> [ -e `dirname "$PIDFILE"` ] || \
>>> install -d -ocassandra -gcassandra -m755 `dirname $PIDFILE`
>>>
>>>
>>>
>>> start-stop-daemon -S -c cassandra -a /usr/sbin/cassandra -q -p
>>> "$PIDFILE" -t >/dev/null || return 1
>>>
>>> start-stop-daemon -S -c cassandra -a /usr/sbin/cassandra -b -p
>>> "$PIDFILE" -- \
>>> -p "$PIDFILE" -H "$heap_dump_f" -E "$error_log_f" >/dev/null ||
>>> return 2
>>>
>>> }
>>>
>>> #
>>> # Function that stops the daemon/service
>>> #
>>> do_stop()
>>> {
>>> # Return
>>> #   0 if daemon has been stopped
>>> #   1 if daemon was already stopped
>>> #   2 if daemon could not be stopped
>>> #   other if a failure occurred
>>> start-stop-daemon -K -p "$PIDFILE" -R TERM/30/KILL/5 >/dev/null
>>> RET=$?
>>> rm -f "$PIDFILE"
>>> return $RET
>>> }
>>>
>>> case "$1" in
>>>   start)
>>> [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
>>> do_start
>>> 

Re: Strange issue wherein cassandra not being started from cron

[Jonathan Haddad]

Last I checked, cron doesn't load the same, full environment you see when
you log in. Also, why put Cassandra on a cron?
On Mon, Jan 9, 2017 at 9:47 PM Bhuvan Rawal  wrote:

> Hi Ajay,
>
> Have you had a look at cron logs? - mine is in path /var/log/cron
>
> Thanks & Regards,
>
> On Tue, Jan 10, 2017 at 9:45 AM, Ajay Garg  wrote:
>
> Hi All.
>
> Facing a very weird issue, wherein the command
>
> */etc/init.d/cassandra start*
>
> causes cassandra to start when the command is run from command-line.
>
>
> However, if I put the above as a cron job
>
>
>
> ** * * * * /etc/init.d/cassandra start*
> cassandra never starts.
>
>
> I have checked, and "cron" service is running.
>
>
> Any ideas what might be wrong?
> I am pasting the cassandra script for brevity.
>
>
> Thanks and Regards,
> Ajay
>
>
>
> 
> #! /bin/sh
> ### BEGIN INIT INFO
> # Provides:  cassandra
> # Required-Start:$remote_fs $network $named $time
> # Required-Stop: $remote_fs $network $named $time
> # Should-Start:  ntp mdadm
> # Should-Stop:   ntp mdadm
> # Default-Start: 2 3 4 5
> # Default-Stop:  0 1 6
> # Short-Description: distributed storage system for structured data
> # Description:   Cassandra is a distributed (peer-to-peer) system for
> #the management and storage of structured data.
> ### END INIT INFO
>
> # Author: Eric Evans 
>
> DESC="Cassandra"
> NAME=cassandra
> PIDFILE=/var/run/$NAME/$NAME.pid
> SCRIPTNAME=/etc/init.d/$NAME
> CONFDIR=/etc/cassandra
> WAIT_FOR_START=10
> CASSANDRA_HOME=/usr/share/cassandra
> FD_LIMIT=10
>
> [ -e /usr/share/cassandra/apache-cassandra.jar ] || exit 0
> [ -e /etc/cassandra/cassandra.yaml ] || exit 0
> [ -e /etc/cassandra/cassandra-env.sh ] || exit 0
>
> # Read configuration variable file if it is present
> [ -r /etc/default/$NAME ] && . /etc/default/$NAME
>
> # Read Cassandra environment file.
> . /etc/cassandra/cassandra-env.sh
>
> if [ -z "$JVM_OPTS" ]; then
> echo "Initialization failed; \$JVM_OPTS not set!" >&2
> exit 3
> fi
>
> export JVM_OPTS
>
> # Export JAVA_HOME, if set.
> [ -n "$JAVA_HOME" ] && export JAVA_HOME
>
> # Load the VERBOSE setting and other rcS variables
> . /lib/init/vars.sh
>
> # Define LSB log_* functions.
> # Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
> . /lib/lsb/init-functions
>
> #
> # Function that returns 0 if process is running, or nonzero if not.
> #
> # The nonzero value is 3 if the process is simply not running, and 1 if the
> # process is not running but the pidfile exists (to match the exit codes
> for
> # the "status" command; see LSB core spec 3.1, section 20.2)
> #
> CMD_PATT="cassandra.+CassandraDaemon"
> is_running()
> {
> if [ -f $PIDFILE ]; then
> pid=`cat $PIDFILE`
> grep -Eq "$CMD_PATT" "/proc/$pid/cmdline" 2>/dev/null && return 0
> return 1
> fi
> return 3
> }
> #
> # Function that starts the daemon/service
> #
> do_start()
> {
> # Return
> #   0 if daemon has been started
> #   1 if daemon was already running
> #   2 if daemon could not be started
>
> ulimit -l unlimited
> ulimit -n "$FD_LIMIT"
>
> cassandra_home=`getent passwd cassandra | awk -F ':' '{ print $6; }'`
> heap_dump_f="$cassandra_home/java_`date +%s`.hprof"
> error_log_f="$cassandra_home/hs_err_`date +%s`.log"
>
> [ -e `dirname "$PIDFILE"` ] || \
> install -d -ocassandra -gcassandra -m755 `dirname $PIDFILE`
>
>
>
> start-stop-daemon -S -c cassandra -a /usr/sbin/cassandra -q -p
> "$PIDFILE" -t >/dev/null || return 1
>
> start-stop-daemon -S -c cassandra -a /usr/sbin/cassandra -b -p
> "$PIDFILE" -- \
> -p "$PIDFILE" -H "$heap_dump_f" -E "$error_log_f" >/dev/null ||
> return 2
>
> }
>
> #
> # Function that stops the daemon/service
> #
> do_stop()
> {
> # Return
> #   0 if daemon has been stopped
> #   1 if daemon was already stopped
> #   2 if daemon could not be stopped
> #   other if a failure occurred
> start-stop-daemon -K -p "$PIDFILE" -R TERM/30/KILL/5 >/dev/null
> RET=$?
> rm -f "$PIDFILE"
> return $RET
> }
>
> case "$1" in
>   start)
> [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
> do_start
> case "$?" in
> 0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
> 2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
> esac
> ;;
>   stop)
> [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
> do_stop
> case "$?" in
> 0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
> 2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
> esac
> ;;
>   restart|force-reload)
> log_daemon_msg "Restarting $DESC" "$NAME"
> do_stop
> case "$?" in
>   

Re: Strange issue wherein cassandra not being started from cron

[Bhuvan Rawal]

Hi Ajay,

Have you had a look at cron logs? - mine is in path /var/log/cron

Thanks & Regards,

On Tue, Jan 10, 2017 at 9:45 AM, Ajay Garg  wrote:

> Hi All.
>
> Facing a very weird issue, wherein the command
>
> */etc/init.d/cassandra start*
>
> causes cassandra to start when the command is run from command-line.
>
>
> However, if I put the above as a cron job
>
>
>
> ** * * * * /etc/init.d/cassandra start*
> cassandra never starts.
>
>
> I have checked, and "cron" service is running.
>
>
> Any ideas what might be wrong?
> I am pasting the cassandra script for brevity.
>
>
> Thanks and Regards,
> Ajay
>
>
> 
> 
> #! /bin/sh
> ### BEGIN INIT INFO
> # Provides:  cassandra
> # Required-Start:$remote_fs $network $named $time
> # Required-Stop: $remote_fs $network $named $time
> # Should-Start:  ntp mdadm
> # Should-Stop:   ntp mdadm
> # Default-Start: 2 3 4 5
> # Default-Stop:  0 1 6
> # Short-Description: distributed storage system for structured data
> # Description:   Cassandra is a distributed (peer-to-peer) system for
> #the management and storage of structured data.
> ### END INIT INFO
>
> # Author: Eric Evans 
>
> DESC="Cassandra"
> NAME=cassandra
> PIDFILE=/var/run/$NAME/$NAME.pid
> SCRIPTNAME=/etc/init.d/$NAME
> CONFDIR=/etc/cassandra
> WAIT_FOR_START=10
> CASSANDRA_HOME=/usr/share/cassandra
> FD_LIMIT=10
>
> [ -e /usr/share/cassandra/apache-cassandra.jar ] || exit 0
> [ -e /etc/cassandra/cassandra.yaml ] || exit 0
> [ -e /etc/cassandra/cassandra-env.sh ] || exit 0
>
> # Read configuration variable file if it is present
> [ -r /etc/default/$NAME ] && . /etc/default/$NAME
>
> # Read Cassandra environment file.
> . /etc/cassandra/cassandra-env.sh
>
> if [ -z "$JVM_OPTS" ]; then
> echo "Initialization failed; \$JVM_OPTS not set!" >&2
> exit 3
> fi
>
> export JVM_OPTS
>
> # Export JAVA_HOME, if set.
> [ -n "$JAVA_HOME" ] && export JAVA_HOME
>
> # Load the VERBOSE setting and other rcS variables
> . /lib/init/vars.sh
>
> # Define LSB log_* functions.
> # Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
> . /lib/lsb/init-functions
>
> #
> # Function that returns 0 if process is running, or nonzero if not.
> #
> # The nonzero value is 3 if the process is simply not running, and 1 if the
> # process is not running but the pidfile exists (to match the exit codes
> for
> # the "status" command; see LSB core spec 3.1, section 20.2)
> #
> CMD_PATT="cassandra.+CassandraDaemon"
> is_running()
> {
> if [ -f $PIDFILE ]; then
> pid=`cat $PIDFILE`
> grep -Eq "$CMD_PATT" "/proc/$pid/cmdline" 2>/dev/null && return 0
> return 1
> fi
> return 3
> }
> #
> # Function that starts the daemon/service
> #
> do_start()
> {
> # Return
> #   0 if daemon has been started
> #   1 if daemon was already running
> #   2 if daemon could not be started
>
> ulimit -l unlimited
> ulimit -n "$FD_LIMIT"
>
> cassandra_home=`getent passwd cassandra | awk -F ':' '{ print $6; }'`
> heap_dump_f="$cassandra_home/java_`date +%s`.hprof"
> error_log_f="$cassandra_home/hs_err_`date +%s`.log"
>
> [ -e `dirname "$PIDFILE"` ] || \
> install -d -ocassandra -gcassandra -m755 `dirname $PIDFILE`
>
>
>
> start-stop-daemon -S -c cassandra -a /usr/sbin/cassandra -q -p
> "$PIDFILE" -t >/dev/null || return 1
>
> start-stop-daemon -S -c cassandra -a /usr/sbin/cassandra -b -p
> "$PIDFILE" -- \
> -p "$PIDFILE" -H "$heap_dump_f" -E "$error_log_f" >/dev/null ||
> return 2
>
> }
>
> #
> # Function that stops the daemon/service
> #
> do_stop()
> {
> # Return
> #   0 if daemon has been stopped
> #   1 if daemon was already stopped
> #   2 if daemon could not be stopped
> #   other if a failure occurred
> start-stop-daemon -K -p "$PIDFILE" -R TERM/30/KILL/5 >/dev/null
> RET=$?
> rm -f "$PIDFILE"
> return $RET
> }
>
> case "$1" in
>   start)
> [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
> do_start
> case "$?" in
> 0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
> 2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
> esac
> ;;
>   stop)
> [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
> do_stop
> case "$?" in
> 0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
> 2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
> esac
> ;;
>   restart|force-reload)
> log_daemon_msg "Restarting $DESC" "$NAME"
> do_stop
> case "$?" in
>   0|1)
> do_start
> case "$?" in
>   0|1)
> do_start
> case "$?" in
> 0) log_end_msg 0 ;;
>