cassandra just gone..no heap dump, no log info
Anyone know how to debug cassandra processes just exiting? There is no info in the cassandra logs and there is no heap dump file(which in the past has shown up in /opt/cassandra/bin directory for me). This occurs when running a map/reduce job that put severe load on the system. The logs look completely fine. I find it odd 1. No logs of why it exited at all 2. No heap dump which would imply there would be no logs as it crashed Is there any other way a process can die and linux would log it somehow? (like running out of memory) Thanks, Dean
Re: cassandra just gone..no heap dump, no log info
A random guess - possibly an OOM (Out of Memory) where Linux will kill a process to recover memory when it is desperately low on memory. Have a look in either your syslog output of the output of dmesg cheers On Wed, Sep 18, 2013 at 10:21 PM, Hiller, Dean dean.hil...@nrel.gov wrote: Anyone know how to debug cassandra processes just exiting? There is no info in the cassandra logs and there is no heap dump file(which in the past has shown up in /opt/cassandra/bin directory for me). This occurs when running a map/reduce job that put severe load on the system. The logs look completely fine. I find it odd 1. No logs of why it exited at all 2. No heap dump which would imply there would be no logs as it crashed Is there any other way a process can die and linux would log it somehow? (like running out of memory) Thanks, Dean -- *Franc Carter* | Systems architect | Sirca Ltd marc.zianideferra...@sirca.org.au franc.car...@sirca.org.au | www.sirca.org.au Tel: +61 2 8355 2514 Level 4, 55 Harrington St, The Rocks NSW 2000 PO Box H58, Australia Square, Sydney NSW 1215
Re: cassandra just gone..no heap dump, no log info
Check if java process got crashed. You can find hs_err*.log file in root directory or cassandra working directory or temporary files directory. Information in this log file can give an idea about failure. On Wed, Sep 18, 2013 at 5:51 PM, Hiller, Dean dean.hil...@nrel.gov wrote: Anyone know how to debug cassandra processes just exiting? There is no info in the cassandra logs and there is no heap dump file(which in the past has shown up in /opt/cassandra/bin directory for me). This occurs when running a map/reduce job that put severe load on the system. The logs look completely fine. I find it odd 1. No logs of why it exited at all 2. No heap dump which would imply there would be no logs as it crashed Is there any other way a process can die and linux would log it somehow? (like running out of memory) Thanks, Dean
Re: cassandra just gone..no heap dump, no log info
This shouldn't happen if you have swap active in the server
On Wednesday, September 18, 2013, Franc Carter wrote:
A random guess - possibly an OOM (Out of Memory) where Linux will kill a
process to recover memory when it is desperately low on memory. Have a look
in either your syslog output of the output of dmesg
cheers
On Wed, Sep 18, 2013 at 10:21 PM, Hiller, Dean
dean.hil...@nrel.govjavascript:_e({}, 'cvml', 'dean.hil...@nrel.gov');
wrote:
Anyone know how to debug cassandra processes just exiting? There is no
info in the cassandra logs and there is no heap dump file(which in the past
has shown up in /opt/cassandra/bin directory for me).
This occurs when running a map/reduce job that put severe load on the
system. The logs look completely fine. I find it odd
1. No logs of why it exited at all
2. No heap dump which would imply there would be no logs as it crashed
Is there any other way a process can die and linux would log it somehow?
(like running out of memory)
Thanks,
Dean
--
*Franc Carter* | Systems architect | Sirca Ltd
javascript:_e({}, 'cvml', 'marc.zianideferra...@sirca.org.au');
franc.car...@sirca.org.au javascript:_e({}, 'cvml',
'franc.car...@sirca.org.au'); | www.sirca.org.au
Tel: +61 2 8355 2514
Level 4, 55 Harrington St, The Rocks NSW 2000
PO Box H58, Australia Square, Sydney NSW 1215
--
*Juan Manuel Formoso
*Senior Geek
http://twitter.com/juanformoso
http://seniorgeek.com.ar
LLAP
Re: cassandra just gone..no heap dump, no log info
Ah neat, I didn't know the dmesg command…that works great. Dean From: Franc Carter franc.car...@sirca.org.aumailto:franc.car...@sirca.org.au Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Wednesday, September 18, 2013 6:45 AM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: cassandra just gone..no heap dump, no log info A random guess - possibly an OOM (Out of Memory) where Linux will kill a process to recover memory when it is desperately low on memory. Have a look in either your syslog output of the output of dmesg cheers On Wed, Sep 18, 2013 at 10:21 PM, Hiller, Dean dean.hil...@nrel.govmailto:dean.hil...@nrel.gov wrote: Anyone know how to debug cassandra processes just exiting? There is no info in the cassandra logs and there is no heap dump file(which in the past has shown up in /opt/cassandra/bin directory for me). This occurs when running a map/reduce job that put severe load on the system. The logs look completely fine. I find it odd 1. No logs of why it exited at all 2. No heap dump which would imply there would be no logs as it crashed Is there any other way a process can die and linux would log it somehow? (like running out of memory) Thanks, Dean -- Franc Carter| Systems architect|Sirca Ltd mailto:marc.zianideferra...@sirca.org.au franc.car...@sirca.org.aumailto:franc.car...@sirca.org.au | www.sirca.org.auhttp://www.sirca.org.au/ Tel: +61 2 8355 2514 Level 4, 55 Harrington St, The Rocks NSW 2000 PO Box H58, Australia Square, Sydney NSW 1215
Re: cassandra just gone..no heap dump, no log info
Swappiness is set to 60 though cassandra recommendations are to turn swap
completely off(we still have not done that as far as I know though) and sure
enough linux killed it.
Dean
From: Juan Manuel Formoso jform...@gmail.commailto:jform...@gmail.com
Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Date: Wednesday, September 18, 2013 6:57 AM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: cassandra just gone..no heap dump, no log info
This shouldn't happen if you have swap active in the server
On Wednesday, September 18, 2013, Franc Carter wrote:
A random guess - possibly an OOM (Out of Memory) where Linux will kill a
process to recover memory when it is desperately low on memory. Have a look in
either your syslog output of the output of dmesg
cheers
On Wed, Sep 18, 2013 at 10:21 PM, Hiller, Dean
dean.hil...@nrel.govjavascript:_e({},%20'cvml',%20'dean.hil...@nrel.gov');
wrote:
Anyone know how to debug cassandra processes just exiting? There is no info in
the cassandra logs and there is no heap dump file(which in the past has shown
up in /opt/cassandra/bin directory for me).
This occurs when running a map/reduce job that put severe load on the system.
The logs look completely fine. I find it odd
1. No logs of why it exited at all
2. No heap dump which would imply there would be no logs as it crashed
Is there any other way a process can die and linux would log it somehow? (like
running out of memory)
Thanks,
Dean
--
Franc Carter| Systems architect|Sirca Ltd
javascript:_e({},%20'cvml',%20'marc.zianideferra...@sirca.org.au');
franc.car...@sirca.org.aujavascript:_e({},%20'cvml',%20'franc.car...@sirca.org.au');
| www.sirca.org.auhttp://www.sirca.org.au/
Tel: +61 2 8355 2514
Level 4, 55 Harrington St, The Rocks NSW 2000
PO Box H58, Australia Square, Sydney NSW 1215
--
Juan Manuel Formoso
Senior Geek
http://twitter.com/juanformoso
http://seniorgeek.com.arhttp://seniorgeek.com.ar/
LLAP
Re: cassandra just gone..no heap dump, no log info
We ran into this while tuning heap sizes. With Cassandra 1.2 making use of off-heap memory, if we made our JVM too large relative to the server memory, the system would just bail. We found for our app that the limit of the JVM size relative to server memory was about 50%. On Wed, Sep 18, 2013 at 8:57 AM, Juan Manuel Formoso jform...@gmail.comwrote: This shouldn't happen if you have swap active in the server On Wednesday, September 18, 2013, Franc Carter wrote: A random guess - possibly an OOM (Out of Memory) where Linux will kill a process to recover memory when it is desperately low on memory. Have a look in either your syslog output of the output of dmesg cheers On Wed, Sep 18, 2013 at 10:21 PM, Hiller, Dean dean.hil...@nrel.govwrote: Anyone know how to debug cassandra processes just exiting? There is no info in the cassandra logs and there is no heap dump file(which in the past has shown up in /opt/cassandra/bin directory for me). This occurs when running a map/reduce job that put severe load on the system. The logs look completely fine. I find it odd 1. No logs of why it exited at all 2. No heap dump which would imply there would be no logs as it crashed Is there any other way a process can die and linux would log it somehow? (like running out of memory) Thanks, Dean -- *Franc Carter* | Systems architect | Sirca Ltd franc.car...@sirca.org.au | www.sirca.org.au Tel: +61 2 8355 2514 Level 4, 55 Harrington St, The Rocks NSW 2000 PO Box H58, Australia Square, Sydney NSW 1215 -- *Juan Manuel Formoso *Senior Geek http://twitter.com/juanformoso http://seniorgeek.com.ar LLAP -- *Ken Hancock *| System Architect, Advanced Advertising SeaChange International 50 Nagog Park Acton, Massachusetts 01720 ken.hanc...@schange.com | www.schange.com | NASDAQ:SEAChttp://www.schange.com/en-US/Company/InvestorRelations.aspx Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hanc...@schange.com | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks [image: LinkedIn]http://www.linkedin.com/in/kenhancock [image: SeaChange International] http://www.schange.com/This e-mail and any attachments may contain information which is SeaChange International confidential. The information enclosed is intended only for the addressees herein and may not be copied or forwarded without permission from SeaChange International.
RE: cassandra just gone..no heap dump, no log info
We faced same issue sometimes too.
1) Linux OOM killer kill your Cassandra process. You should find this event log
in /var/log/message.2) The JVM crashed. You should be able to find the
hs_err_pid file under /tmp folder, if you didn't specify the location when you
started your JVM.
We still cannot find out the root reason why the JVM crashed. The core dump
file indicates the JVM crashed in compile Java code, several times though. It
didn't help even we upgraded from latest JDK 1.6 to latest JDK 1.7.
Yong
From: dean.hil...@nrel.gov
To: user@cassandra.apache.org
Date: Wed, 18 Sep 2013 06:59:07 -0600
Subject: Re: cassandra just gone..no heap dump, no log info
Swappiness is set to 60 though cassandra recommendations are to turn swap
completely off(we still have not done that as far as I know though) and sure
enough linux killed it.
Dean
From: Juan Manuel Formoso jform...@gmail.commailto:jform...@gmail.com
Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Date: Wednesday, September 18, 2013 6:57 AM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: cassandra just gone..no heap dump, no log info
This shouldn't happen if you have swap active in the server
On Wednesday, September 18, 2013, Franc Carter wrote:
A random guess - possibly an OOM (Out of Memory) where Linux will kill a
process to recover memory when it is desperately low on memory. Have a look
in either your syslog output of the output of dmesg
cheers
On Wed, Sep 18, 2013 at 10:21 PM, Hiller, Dean
dean.hil...@nrel.govjavascript:_e({},%20'cvml',%20'dean.hil...@nrel.gov');
wrote:
Anyone know how to debug cassandra processes just exiting? There is no info
in the cassandra logs and there is no heap dump file(which in the past has
shown up in /opt/cassandra/bin directory for me).
This occurs when running a map/reduce job that put severe load on the system.
The logs look completely fine. I find it odd
1. No logs of why it exited at all
2. No heap dump which would imply there would be no logs as it crashed
Is there any other way a process can die and linux would log it somehow?
(like running out of memory)
Thanks,
Dean
--
Franc Carter| Systems architect|Sirca Ltd
javascript:_e({},%20'cvml',%20'marc.zianideferra...@sirca.org.au');
franc.car...@sirca.org.aujavascript:_e({},%20'cvml',%20'franc.car...@sirca.org.au');
| www.sirca.org.auhttp://www.sirca.org.au/
Tel: +61 2 8355 2514
Level 4, 55 Harrington St, The Rocks NSW 2000
PO Box H58, Australia Square, Sydney NSW 1215
--
Juan Manuel Formoso
Senior Geek
http://twitter.com/juanformoso
http://seniorgeek.com.arhttp://seniorgeek.com.ar/
LLAP
Re: cassandra just gone..no heap dump, no log info
We had hs_err_pid files moths ago and it was happening every 6 days or so and
we switched to this JVM and we have not seen one since(including today)…that
worked for us at least.
java version 1.6.0_41
Java(TM) SE Runtime Environment (build 1.6.0_41-b02)
Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
Dean
From: java8964 java8964 java8...@hotmail.commailto:java8...@hotmail.com
Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Date: Wednesday, September 18, 2013 7:08 AM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: RE: cassandra just gone..no heap dump, no log info
We faced same issue sometimes too.
1) Linux OOM killer kill your Cassandra process. You should find this event log
in /var/log/message.
2) The JVM crashed. You should be able to find the hs_err_pid file under /tmp
folder, if you didn't specify the location when you started your JVM.
We still cannot find out the root reason why the JVM crashed. The core dump
file indicates the JVM crashed in compile Java code, several times though. It
didn't help even we upgraded from latest JDK 1.6 to latest JDK 1.7.
Yong
From: dean.hil...@nrel.govmailto:dean.hil...@nrel.gov
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
Date: Wed, 18 Sep 2013 06:59:07 -0600
Subject: Re: cassandra just gone..no heap dump, no log info
Swappiness is set to 60 though cassandra recommendations are to turn swap
completely off(we still have not done that as far as I know though) and sure
enough linux killed it.
Dean
From: Juan Manuel Formoso
jform...@gmail.commailto:jform...@gmail.commailto:jform...@gmail.com
Reply-To:
user@cassandra.apache.orgmailto:user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.orgmailto:user@cassandra.apache.org
Date: Wednesday, September 18, 2013 6:57 AM
To:
user@cassandra.apache.orgmailto:user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: cassandra just gone..no heap dump, no log info
This shouldn't happen if you have swap active in the server
On Wednesday, September 18, 2013, Franc Carter wrote:
A random guess - possibly an OOM (Out of Memory) where Linux will kill a
process to recover memory when it is desperately low on memory. Have a look
in either your syslog output of the output of dmesg
cheers
On Wed, Sep 18, 2013 at 10:21 PM, Hiller, Dean
dean.hil...@nrel.govmailto:dean.hil...@nrel.govjavascript:_e({},%20'cvml',%20'dean.hil...@nrel.govmailto:%20'dean.hil...@nrel.gov');
wrote:
Anyone know how to debug cassandra processes just exiting? There is no info
in the cassandra logs and there is no heap dump file(which in the past has
shown up in /opt/cassandra/bin directory for me).
This occurs when running a map/reduce job that put severe load on the system.
The logs look completely fine. I find it odd
1. No logs of why it exited at all
2. No heap dump which would imply there would be no logs as it crashed
Is there any other way a process can die and linux would log it somehow?
(like running out of memory)
Thanks,
Dean
--
Franc Carter| Systems architect|Sirca Ltd
javascript:_e({},%20'cvml',%20'marc.zianideferra...@sirca.org.aumailto:%20'marc.zianideferra...@sirca.org.au');
franc.car...@sirca.org.aumailto:franc.car...@sirca.org.aujavascript:_e({},%20'cvml',%20'franc.car...@sirca.org.aumailto:%20'franc.car...@sirca.org.au');
| www.sirca.org.auhttp://www.sirca.org.au/
Tel: +61 2 8355 2514
Level 4, 55 Harrington St, The Rocks NSW 2000
PO Box H58, Australia Square, Sydney NSW 1215
--
Juan Manuel Formoso
Senior Geek
http://twitter.com/juanformoso
http://seniorgeek.com.arhttp://seniorgeek.com.ar/
LLAP
