[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287277#comment-16287277 ] Benjamin Lerer commented on CASSANDRA-13006: [~jjirsa] Thanks for the reminder. I fixed the review comments and rerun CI on our internal instances. Tests looks good including the DTests ones (no new failing tests). > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Assignee: Benjamin Lerer >Priority: Minor > Attachments: 13006-3.0.9.txt > > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286970#comment-16286970 ] Jeff Jirsa commented on CASSANDRA-13006: Friendly ping [~blerer] that this is sitting in "Ready to Commit". > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Assignee: Benjamin Lerer >Priority: Minor > Attachments: 13006-3.0.9.txt > > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260929#comment-16260929
]
Benjamin Lerer commented on CASSANDRA-13006:
Thanks for the review.
{quote}
* In our launch scripts, why have a separate JVM_ON_OUT_OF_MEMORY_ERROR_OPT
instead of nesting it inside JVM_OPTS? The latter seems more consistent w/other
configuration approaches. This would also clean up launching branch logic a
touch.
{quote}
As bash commands split words on white spaces without taking quotes into
account, specifying the {{OnOutOfMemoryError}} as part of the {{JVM_OPTS}}
variable cannot work for an option value such as: {{kill -9 %p}}. I wasted a
couple of days on that and could not come up with a better solution that the
proposed one.
I will take care of the nits.
> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by
> default if the process encountered an OOM error. These heap dumps are stored
> in the Apache Cassandra home directory unless configured otherwise (see
> [Cassandra Support
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
> for this feature).
>
> The creation and storage of heap dumps aides debugging and investigative
> workflows, but is not be desirable for a production environment where these
> heap dumps may occupy a large amount of disk space and require manual
> intervention for cleanups.
>
> Managing heap dumps on out of memory errors and configuring the paths for
> these heap dumps are available as JVM options in JVM. The current behavior
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.
>
> A patch can be proposed here that would make the heap dump on OOM error honor
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM
> option.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260876#comment-16260876
]
Joshua McKenzie commented on CASSANDRA-13006:
-
Sorry for the delay - didn't realize I was reviewer on this.
Feedback follows:
general feedback:
- In our launch scripts, why have a separate JVM_ON_OUT_OF_MEMORY_ERROR_OPT
instead of nesting it inside JVM_OPTS? The latter seems more consistent w/other
configuration approaches. This would also clean up launching branch logic a
touch.
nits:
- Not sure why StartupChecks.checkOutOfMemoryHandling() is scoped to protected.
Should be able to be private.
- remove added whitespace at top of bin/cassandra.sh
- misspelling: {code} # uncomment the prefered option {code} should be
{{preferred}}
- Same holds for conf/cassandra-env.sh
- Having parseJavaVersion and parseUpdate both take no arguments and
System.getProperty("java.version") would be a cleaner interface. This isn't
critical path in any way so the duplicate calls to getProperty would be no real
issue.
- "jmcd" should read "jcmd" in comments in HeapUtils.java, line 55
- Since HeapUtils is the only call-site for {{logProcessOutput}}, that can be
inlined if you're in the mood for it.
Basically a bag of trivial things - looks good on the whole. +1 with the above
nits dealt with as you see fit.
> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by
> default if the process encountered an OOM error. These heap dumps are stored
> in the Apache Cassandra home directory unless configured otherwise (see
> [Cassandra Support
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
> for this feature).
>
> The creation and storage of heap dumps aides debugging and investigative
> workflows, but is not be desirable for a production environment where these
> heap dumps may occupy a large amount of disk space and require manual
> intervention for cleanups.
>
> Managing heap dumps on out of memory errors and configuring the paths for
> these heap dumps are available as JVM options in JVM. The current behavior
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.
>
> A patch can be proposed here that would make the heap dump on OOM error honor
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM
> option.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247177#comment-16247177
]
Benjamin Lerer commented on CASSANDRA-13006:
I updated the patches for {{3.0}}, {{3.11}} and {{trunk}}. I replaced the use
of {{jmap}} by {{jcmd}}.
CI results look good for the unit tests.
> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by
> default if the process encountered an OOM error. These heap dumps are stored
> in the Apache Cassandra home directory unless configured otherwise (see
> [Cassandra Support
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
> for this feature).
>
> The creation and storage of heap dumps aides debugging and investigative
> workflows, but is not be desirable for a production environment where these
> heap dumps may occupy a large amount of disk space and require manual
> intervention for cleanups.
>
> Managing heap dumps on out of memory errors and configuring the paths for
> these heap dumps are available as JVM options in JVM. The current behavior
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.
>
> A patch can be proposed here that would make the heap dump on OOM error honor
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM
> option.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195167#comment-16195167
]
Benjamin Lerer commented on CASSANDRA-13006:
Just realized that I forgot to switch from {{jmap}} to {{jcmd}}, for generating
heap histograms, as suggested by [~nibin.gv]. I will update my patches in the
coming days.
> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.15
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by
> default if the process encountered an OOM error. These heap dumps are stored
> in the Apache Cassandra home directory unless configured otherwise (see
> [Cassandra Support
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
> for this feature).
>
> The creation and storage of heap dumps aides debugging and investigative
> workflows, but is not be desirable for a production environment where these
> heap dumps may occupy a large amount of disk space and require manual
> intervention for cleanups.
>
> Managing heap dumps on out of memory errors and configuring the paths for
> these heap dumps are available as JVM options in JVM. The current behavior
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.
>
> A patch can be proposed here that would make the heap dump on OOM error honor
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM
> option.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195149#comment-16195149
]
Benjamin Lerer commented on CASSANDRA-13006:
[~urandom], [~brandon.williams], [~tjake]
Sorry, for the delay.
I pushed some patches for
[2.2|https://github.com/apache/cassandra/compare/cassandra-2.2...blerer:13006-2.2],
[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...blerer:13006-3.0],
[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...blerer:13006-3.11]
and
[trunk|https://github.com/apache/cassandra/compare/trunk...blerer:13006-trunk].
The branches differs only at the level of the configuration files
({{cassandra-env.sh}} and {{cassandra-env.ps1}}).
The patches let the JVM handle the {{OutOfMemoryErrors}} throught the JVM
{{OnOutOfMemoryError}}, {{ExitOnOutOfMemoryError}} or
{{CrashOnOutOfMemoryError}} options.
As the {{ExitOnOutOfMemoryError}} and {{CrashOnOutOfMemoryError}} options are
only supported since Oracle JDK 7 update 101 and since JDK 8 update 92,
Cassandra uses by default the {{OnOutOfMemoryError}} option.
A startup check will emit a warning if none of the options is used. This check
is there to ensure that {{OOM}} errors are properly handled and that C* cannot
continue to run in an unstable state that could cause data corruption.
The patch add no check for the {{HeapDumpOnOutOfMemoryError}} option as in some
cases administrators prefer to disable them.
The {{cassandra-env.sh}} has a new variable {{JVM_ON_OUT_OF_MEMORY_ERROR_OPT}}
which should be use to specify the {{OnOutOfMemoryError}} option. As bash
commands split words on white spaces without taking quotes into account,
specifying the {{OnOutOfMemoryError}} as part of the {{JVM_OPTS}} variable
cannot work for an option value such as: {{kill -9 %p}}.
Before generating an heap dump, C* use to log an Heap histogram using {{jmap}}.
If the heap size was large, reading the heap dump could take a few hours and
the heap histogram can help to debug the problem much faster. The patches keep
the posibility to print an heap histogram on OOM error but disables it by
default. To enable it the {{cassandra.printHeapHistogramOnOutOfMemoryError}}
system property must be set to {{true}}.
As generating the histogram for only the live objects (using {{jmap
histo:live}}) would trigger a garbage collection before generating the
histogram, I prefered to stick with {{jmap histo}} to minimize the risks.
The previous implementation was suffering of 2 problems:
* If several OOM errors were thrown in a short time span, each of them would
trigger an heap histogram and an heap dump (see this
[comment|https://issues.apache.org/jira/browse/CASSANDRA-13006?focusedCommentId=16118421=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16118421])
* If an exception was thrown while C* was trying to generate the heap dump, C*
would not be shutdown and would continue running in an unstable state (see
CASSANDRA-13886)
The patches fix those problems for the case were an heap histogram need to be
logged. In the case were the histogram is not requested those problems do not
exist anymore.
CI looks good for the unit tests.
The changes to the {{cassandra}} startup script break the DTests but before
changing the {{DTests}} framework I would prefer having a first review of the
patches.
[~JoshuaMcKenzie] could you review the patches?
> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.15
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by
> default if the process encountered an OOM error. These heap dumps are stored
> in the Apache Cassandra home directory unless configured otherwise (see
> [Cassandra Support
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
> for this feature).
>
> The creation and storage of heap dumps aides debugging and investigative
> workflows, but is not be desirable for a production environment where these
> heap dumps may occupy a large amount of disk space and require manual
> intervention for cleanups.
>
> Managing heap dumps on out of memory errors and configuring the paths for
> these heap dumps are available as JVM options in JVM. The current behavior
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.
>
> A patch can be proposed here that would make the heap dump on OOM error honor
> the HeapDumpOnOutOfMemoryError flag.
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167937#comment-16167937 ] Andrew Jorgensen commented on CASSANDRA-13006: -- Just to ask a clarifying question for my own understanding. This change would change the default from always producing a heap dump to having to explicitly turn it on which I think is good. One place this might trip some people up is cassandra has a command line [argument|https://github.com/apache/cassandra/blob/fe3cfe3d7df296f022c50c9c0d22f91a0fc0a217/bin/cassandra#L245-L247] that allows setting the specific location of the heap dump. One place that I know this is used is in the [debian init|https://github.com/apache/cassandra/blob/fe3cfe3d7df296f022c50c9c0d22f91a0fc0a217/debian/init#L86-L87] script. Presumably if that flag is set the user wants to get heap dumps, is it work also checking to see if -XX:HeapDumpPath or -XX:+HeapDumpOnOutOfMemoryError are present when creating a heap dump to make sure the behavior stays consistent? I also have another change [CASSANDRA-13843|https://issues.apache.org/jira/browse/CASSANDRA-13843] that I am proposing to fix the debian init script since it shadows the HeapDumpPath environment variable and renders it unchangeable from the default in its current form. > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Assignee: Benjamin Lerer >Priority: Minor > Fix For: 3.0.9 > > Attachments: 13006-3.0.9.txt > > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137242#comment-16137242
]
Joshua McKenzie commented on CASSANDRA-13006:
-
bq. Joshua McKenzie you worked on CASSANDRA-7507. Do you have any concern with
the approach I am suggesting?
No major concerns; maybe we should have a commented out option in the config to
add {{CrashOnOutOfMemoryError}} for operators that would prefer to use that?
bq. in the News.txt upgrade procedure: request people to use a java version >=
8u92
We can also enforce that or warn when we version check in the startup scripts.
> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.9
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by
> default if the process encountered an OOM error. These heap dumps are stored
> in the Apache Cassandra home directory unless configured otherwise (see
> [Cassandra Support
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
> for this feature).
>
> The creation and storage of heap dumps aides debugging and investigative
> workflows, but is not be desirable for a production environment where these
> heap dumps may occupy a large amount of disk space and require manual
> intervention for cleanups.
>
> Managing heap dumps on out of memory errors and configuring the paths for
> these heap dumps are available as JVM options in JVM. The current behavior
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.
>
> A patch can be proposed here that would make the heap dump on OOM error honor
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM
> option.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135080#comment-16135080
]
Benjamin Lerer commented on CASSANDRA-13006:
[~urandom] Sorry, this ticket felt out of my radar.
bq. IMO, the sanest strategy here would be to leave the creation of heap dumps
to the JVM.
I fully agree with you.
Initially, we started to catch {{OOM}} errors to prevent C* to run in an
unknown state that could cause data corruption (see CASSANDRA-7507). The
problem with that approach was that Heap dumps were not created anymore on
{{OOM}} errors. I tried to fix that in CASSANDRA-9861 but that approach seems
to have some serious issues and limitations.
In April 2016, roughly at the same time that I was fixing CASSANDRA-9861,
Oracle released the [JDK
8u92|http://www.oracle.com/technetwork/java/javase/8u92-relnotes-2949471.html]
which added 2 new JVM Options: {{ExitOnOutOfMemoryError}} and
{{CrashOnOutOfMemoryError}}.
With that in mind, my idea would be to:
# stop handling the {{OOM}} errors on the C* side (and producing Heap dump)
# add {{ExitOnOutOfMemoryError}} to the default JVM options in the startup
scripts
# in the {{News.txt}} upgrade procedure: request people to use a java version
>= {{8u92}}
[~JoshuaMcKenzie] you worked on CASSANDRA-7507. Do you have any concern with
the approach I am suggesting?
My only concern right now is that I know that some C* users are using Zing and
I do not if it support the {{ExitOnOutOfMemoryError}} option.
I asked the Zing support and will update the ticket as soon as I know more.
> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.9
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by
> default if the process encountered an OOM error. These heap dumps are stored
> in the Apache Cassandra home directory unless configured otherwise (see
> [Cassandra Support
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
> for this feature).
>
> The creation and storage of heap dumps aides debugging and investigative
> workflows, but is not be desirable for a production environment where these
> heap dumps may occupy a large amount of disk space and require manual
> intervention for cleanups.
>
> Managing heap dumps on out of memory errors and configuring the paths for
> these heap dumps are available as JVM options in JVM. The current behavior
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.
>
> A patch can be proposed here that would make the heap dump on OOM error honor
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM
> option.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118421#comment-16118421
]
Eric Evans commented on CASSANDRA-13006:
I think this behavior (invoking {{jmap}} on OOM) is a pretty serious violation
to the Element of Least-surprise. We already provide mechanisms for passing
arguments to the JVM, and TTBMK, all of them provide some means for dropping a
heap dump on out-of-memory.
It definitely caught me be surprise. We carried over
{{-XX:+HeapDumpOnOutOfMemoryError}} from our 2.2.x environment, only to have
Cassandra and the JVM racing to create a dump of the same name.
Additionally, something about all of this is buggy, because on more than one
occasion we've had Cassandra fork-bombing {{jmap}} processes
{noformat}
● cassandra-b.service - distributed storage system for structured data
Loaded: loaded (/lib/systemd/system/cassandra-b.service; static)
Active: active (running) since Sat 2017-08-05 22:32:07 UTC; 23h ago
Main PID: 25025 (java)
CGroup: /system.slice/cassandra-b.service
├─ 9213 jmap -histo 25025
├─ 9214 jmap -histo 25025
├─ 9284 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9285 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9388 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9453 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9519 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9520 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9733 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9735 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9736 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─14835 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─14836 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─14837 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─14839 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─14841 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─14844 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18932 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18933 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18934 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18935 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18936 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18937 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18938 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18939 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18940 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18942 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18943 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18944 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18945 jmap
-dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
[ ... ]
{noformat}
IMO, the sanest strategy here would be to leave the creation of heap dumps to
the JVM.
> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.9
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by
> default if the process encountered an OOM error. These heap dumps are stored
> in the Apache Cassandra home directory unless configured otherwise (see
> [Cassandra Support
>
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975265#comment-15975265 ] Nibin G commented on CASSANDRA-13006: - Why can't we delegate the heap dump generation to JVM if jmap is not available in class path ? JRE can generate heap dump even if jmap is not there in the path. > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Assignee: Benjamin Lerer >Priority: Minor > Fix For: 3.0.9 > > Attachments: 13006-3.0.9.txt > > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975192#comment-15975192 ] Jeremiah Jordan commented on CASSANDRA-13006: - We could fall back to trying to use the "com.sun.management:type=HotSpotDiagnostic" bean directly if we can't find jmap. Some links for doing this: https://blogs.oracle.com/sundararajan/entry/programmatically_dumping_heap_from_java http://stackoverflow.com/a/12297339/138693 > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Assignee: Benjamin Lerer >Priority: Minor > Fix For: 3.0.9 > > Attachments: 13006-3.0.9.txt > > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975153#comment-15975153 ] Nibin G commented on CASSANDRA-13006: - Oracle Java's JRE 8 and Server JRE 8 for linux environments are not shipping jmap anymore. That means, we have to use Oracle Java's JDK for the heap dumps to be generated. And some of the security compliance won't permit the use of JDK in production. It would be great if an option is provided to disable heap dump from the application code[1]. So that JVM can generate the heap dump. [1] https://github.com/apache/cassandra/blob/81f6c784ce967fadb6ed7f58de1328e713eaf53c/src/java/org/apache/cassandra/utils/JVMStabilityInspector.java#L56 > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Assignee: Benjamin Lerer >Priority: Minor > Fix For: 3.0.9 > > Attachments: 13006-3.0.9.txt > > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880141#comment-15880141
]
Benjamin Lerer commented on CASSANDRA-13006:
Sorry, for the misunderstanding.
Your patch fix only one part of the problem and I would rather fix all the
problems at once than solving them in multiple tickets.
Basically, the fix need to take into account the JVM being used and handle the
processing of {{OOM}} errors accordingly. It also need to support options like
{{-XX:OnOutOfMemoryError}}.
Apparently, {{Oracle}} and {{Zing}} JVMs use the same options names. The
{{IBM}} one seems to use different options.
The patch should also log a clear error message if the JVM is not supported.
If you are still interested in working on this issue, feel free to reasign it
to yourself.
> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.9
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by
> default if the process encountered an OOM error. These heap dumps are stored
> in the Apache Cassandra home directory unless configured otherwise (see
> [Cassandra Support
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
> for this feature).
>
> The creation and storage of heap dumps aides debugging and investigative
> workflows, but is not be desirable for a production environment where these
> heap dumps may occupy a large amount of disk space and require manual
> intervention for cleanups.
>
> Managing heap dumps on out of memory errors and configuring the paths for
> these heap dumps are available as JVM options in JVM. The current behavior
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.
>
> A patch can be proposed here that would make the heap dump on OOM error honor
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM
> option.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880056#comment-15880056 ] anmols commented on CASSANDRA-13006: I have suggested a patch for this issue, can you please let me know if this is acceptable for this issue? > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Assignee: Benjamin Lerer >Priority: Minor > Fix For: 3.0.9 > > Attachments: 13006-3.0.9.txt > > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750847#comment-15750847 ] Nitsan Wakart commented on CASSANDRA-13006: --- OnOutOfMemoryError and HeapDumpOnOutOfMemoryError are both available on Zing. The default for HeapDumpOnOutOfMemoryError is false. Let me know if you need more details. > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Assignee: Benjamin Lerer >Priority: Minor > Fix For: 3.0.9 > > Attachments: 13006-3.0.9.txt > > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15741620#comment-15741620
]
Benjamin Lerer commented on CASSANDRA-13006:
One thing that I did not consider why working on CASSANDRA-9861 was that not
everybody use the Oracle JVM. It is clear that some people use {{Zing}} and I
guess that some people might also be using the IBM JVM. All of those JVM seems
to provide {{jmap}} but they do not seems to use the same configuration
arguments for handling {{OOM}} errors. I could not find the {{Zing}} options
([~nitsanw] do you have any idea of where I could get them?) but the IBM JVM
does not use the same options as the Oracle one.
Moreover, the Oracle JVM support also options like {{-XX:OnOutOfMemoryError}}
which we currently ignore.
To be honest, I am not fully sure how we should handle all that. We could:
# try to determine which JVM is being used and check for the relevants options
(knowing that it will never be fully bulletproof)
# ignore the JVM options and create our own configuration for it (knowing that
it might be confusing for the administrators)
Any opinions or suggestions?
> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.9
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by
> default if the process encountered an OOM error. These heap dumps are stored
> in the Apache Cassandra home directory unless configured otherwise (see
> [Cassandra Support
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
> for this feature).
>
> The creation and storage of heap dumps aides debugging and investigative
> workflows, but is not be desirable for a production environment where these
> heap dumps may occupy a large amount of disk space and require manual
> intervention for cleanups.
>
> Managing heap dumps on out of memory errors and configuring the paths for
> these heap dumps are available as JVM options in JVM. The current behavior
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.
>
> A patch can be proposed here that would make the heap dump on OOM error honor
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM
> option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730400#comment-15730400 ] anmols commented on CASSANDRA-13006: [~blerer] We encountered a condition in one of our clusters that resulted in repeated OOMs across many of the nodes. The reason for the OOMs is an issue (CASSANDRA-12796) that was causing heap exhaustion when rebuilding secondary indexes over wide partitions. > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Assignee: Benjamin Lerer >Priority: Minor > Fix For: 3.0.9 > > Attachments: 13006-3.0.9.txt > > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15728130#comment-15728130 ] Benjamin Lerer commented on CASSANDRA-13006: I also believe that the issue is legitimate as the code is already relying on the JVM argument for the location where the dumps must be generated. After I am not convinced that turning it off is a wise thing to do. > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Priority: Minor > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727672#comment-15727672 ] Jeremiah Jordan commented on CASSANDRA-13006: - [~cnlwsu] This isn't proposing we remove heap dumps, just that we honor the JVM flags for making and storing them. I think that is probably a good idea. We should still default our flags to make them in env files, but users should have control over if they are to be written out, besides hacks like setting the output to a read only location. > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Priority: Minor > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726095#comment-15726095 ] Benjamin Lerer commented on CASSANDRA-13006: [~anmolsharma.141]] Out of curiosity, how often do your nodes dies of an OOM error? > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Priority: Minor > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15725927#comment-15725927
]
Chris Lohfink commented on CASSANDRA-13006:
---
I am {{-1}} on this. With current compaction strategies production nodes should
have 50-30% free disk space and a heap dump of 8gb is difference between being
able to debug bad issues and not. Its like removing log files to save disk
space. If anything making it a "rotating" heap dumpa to only have at most 5 or
something sounds like a better idea (like log files).
> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
>Reporter: anmols
>Priority: Minor
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by
> default if the process encountered an OOM error. These heap dumps are stored
> in the Apache Cassandra home directory unless configured otherwise (see
> [Cassandra Support
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
> for this feature).
>
> The creation and storage of heap dumps aides debugging and investigative
> workflows, but is not be desirable for a production environment where these
> heap dumps may occupy a large amount of disk space and require manual
> intervention for cleanups.
>
> Managing heap dumps on out of memory errors and configuring the paths for
> these heap dumps are available as JVM options in JVM. The current behavior
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.
>
> A patch can be proposed here that would make the heap dump on OOM error honor
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM
> option.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
