[
https://issues.apache.org/jira/browse/CASSANDRA-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886475#comment-16886475
]
Joseph Lynch commented on CASSANDRA-15214:
------------------------------------------
We've (Netlfix) found handling OOMs to be generally hard to do correctly in all
the various Java codebases we have so we built an agent solution which attaches
to the JVM in [https://github.com/Netflix-Skunkworks/jvmquake]. I think the
only reason that we couldn't just directly include that in C* is because it's a
C JVMTI agent instead of a Java one, but perhaps we could just solve this with
some documentation and making it really easy to include agents (which is useful
regardless)?
The following is the patch for supporting easy pluggable agents for C*:
{noformat}
diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh
index d6c48be0a3..92061db3ab 100644
--- a/conf/cassandra-env.sh
+++ b/conf/cassandra-env.sh
@@ -134,6 +134,29 @@ do
JVM_OPTS="$JVM_OPTS $opt"
done
+# Pull in any agents present in CASSANDRA_HOME
+for agent_file in ${CASSANDRA_HOME}/agents/*.jar; do
+ if [ -e "${agent_file}" ]; then
+ base_file="${agent_file%.jar}"
+ if [ -s "${base_file}.options" ]; then
+ options=`cat ${base_file}.options`
+ agent_file="${agent_file}=${options}"
+ fi
+ JVM_OPTS="$JVM_OPTS -javaagent:${agent_file}"
+ fi
+done
+
+for agent_file in ${CASSANDRA_HOME}/agents/*.so; do
+ if [ -e "${agent_file}" ]; then
+ base_file="${agent_file%.so}"
+ if [ -s "${base_file}.options" ]; then
+ options=`cat ${base_file}.options`
+ agent_file="${agent_file}=${options}"
+ fi
+ JVM_OPTS="$JVM_OPTS -agentpath:${agent_file}"
+ fi
+done
{noformat}
Then we can just drop agents into the {{CASSANDRA_HOME/agents}} folder and they
are loaded automatically by Cassandra. From a security perspective this is
identical to "drop a jar".
> OOMs caught and not rethrown
> ----------------------------
>
> Key: CASSANDRA-15214
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15214
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Client, Messaging/Internode
> Reporter: Benedict
> Priority: Normal
> Fix For: 4.0
>
>
> Netty (at least, and perhaps elsewhere in Executors) catches all exceptions,
> so presently there is no way to ensure that an OOM reaches the JVM handler to
> trigger a crash/heapdump.
> It may be that the simplest most consistent way to do this would be to have a
> single thread spawned at startup that waits for any exceptions we must
> propagate to the Runtime.
> We could probably submit a patch upstream to Netty, but for a guaranteed
> future proof approach, it may be worth paying the cost of a single thread.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
