[ 
https://issues.apache.org/jira/browse/CASSANDRA-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886475#comment-16886475
 ] 

Joseph Lynch edited comment on CASSANDRA-15214 at 7/16/19 9:36 PM:
-------------------------------------------------------------------

We've (Netlfix) found handling OOMs to be generally hard to do correctly in all 
the various Java codebases we have so we built an agent solution which attaches 
to the JVM in [https://github.com/Netflix-Skunkworks/jvmquake]. I think the 
only reason that we couldn't just directly include that in C* is because it's a 
C JVMTI agent instead of a Java one, but perhaps we could just solve this with 
some documentation and making it really easy to include agents (which is useful 
regardless)?  I can also spend some time and see if I can make it a java agent 
instead of a c one.

The following is the patch for supporting easy pluggable agents for C*:
{noformat}
diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh
index d6c48be0a3..92061db3ab 100644
--- a/conf/cassandra-env.sh
+++ b/conf/cassandra-env.sh
@@ -134,6 +134,29 @@ do
   JVM_OPTS="$JVM_OPTS $opt"
 done
 
+# Pull in any agents present in CASSANDRA_HOME
+for agent_file in ${CASSANDRA_HOME}/agents/*.jar; do
+  if [ -e "${agent_file}" ]; then
+    base_file="${agent_file%.jar}"
+    if [ -s "${base_file}.options" ]; then
+      options=`cat ${base_file}.options`
+      agent_file="${agent_file}=${options}"
+    fi
+    JVM_OPTS="$JVM_OPTS -javaagent:${agent_file}"
+  fi
+done
+
+for agent_file in ${CASSANDRA_HOME}/agents/*.so; do
+  if [ -e "${agent_file}" ]; then
+    base_file="${agent_file%.so}"
+    if [ -s "${base_file}.options" ]; then
+      options=`cat ${base_file}.options`
+      agent_file="${agent_file}=${options}"
+    fi
+    JVM_OPTS="$JVM_OPTS -agentpath:${agent_file}"
+  fi
+done
{noformat}
Then we can just drop agents into the {{CASSANDRA_HOME/agents}} folder and they 
are loaded automatically by Cassandra. From a security perspective this is 
identical to "drop a jar".


was (Author: jolynch):
We've (Netlfix) found handling OOMs to be generally hard to do correctly in all 
the various Java codebases we have so we built an agent solution which attaches 
to the JVM in [https://github.com/Netflix-Skunkworks/jvmquake]. I think the 
only reason that we couldn't just directly include that in C* is because it's a 
C JVMTI agent instead of a Java one, but perhaps we could just solve this with 
some documentation and making it really easy to include agents (which is useful 
regardless)?

The following is the patch for supporting easy pluggable agents for C*:
{noformat}
diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh
index d6c48be0a3..92061db3ab 100644
--- a/conf/cassandra-env.sh
+++ b/conf/cassandra-env.sh
@@ -134,6 +134,29 @@ do
   JVM_OPTS="$JVM_OPTS $opt"
 done
 
+# Pull in any agents present in CASSANDRA_HOME
+for agent_file in ${CASSANDRA_HOME}/agents/*.jar; do
+  if [ -e "${agent_file}" ]; then
+    base_file="${agent_file%.jar}"
+    if [ -s "${base_file}.options" ]; then
+      options=`cat ${base_file}.options`
+      agent_file="${agent_file}=${options}"
+    fi
+    JVM_OPTS="$JVM_OPTS -javaagent:${agent_file}"
+  fi
+done
+
+for agent_file in ${CASSANDRA_HOME}/agents/*.so; do
+  if [ -e "${agent_file}" ]; then
+    base_file="${agent_file%.so}"
+    if [ -s "${base_file}.options" ]; then
+      options=`cat ${base_file}.options`
+      agent_file="${agent_file}=${options}"
+    fi
+    JVM_OPTS="$JVM_OPTS -agentpath:${agent_file}"
+  fi
+done
{noformat}
Then we can just drop agents into the {{CASSANDRA_HOME/agents}} folder and they 
are loaded automatically by Cassandra. From a security perspective this is 
identical to "drop a jar".

> OOMs caught and not rethrown
> ----------------------------
>
>                 Key: CASSANDRA-15214
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-15214
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Messaging/Client, Messaging/Internode
>            Reporter: Benedict
>            Priority: Normal
>             Fix For: 4.0
>
>
> Netty (at least, and perhaps elsewhere in Executors) catches all exceptions, 
> so presently there is no way to ensure that an OOM reaches the JVM handler to 
> trigger a crash/heapdump.
> It may be that the simplest most consistent way to do this would be to have a 
> single thread spawned at startup that waits for any exceptions we must 
> propagate to the Runtime.
> We could probably submit a patch upstream to Netty, but for a guaranteed 
> future proof approach, it may be worth paying the cost of a single thread.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to