Hi guys,
Anyone knows how to add the "-javaagent" argument in the Jetty bootstrap?
The main motivation is to get some metrics from PredictionIO using the APM
NewRelic. Just to add some context about our architecture, we've created a
container that runs the "pio deploy" inside a Kubernetes cluster, NewRelic
is the official monitoring tool for our company that's why I'm trying to
configure NewRelic.
I've already tried some workarounds but no success yet.
First, I've tried to override the JVM_OPTS env, so I realized that
PredicitonIO executes two java processes when the "pio deploy" is executed,
see the example below:
ps -auxww
*root 12 2.2 2.6 7343400 434516 ? Sl 21:35 0:32
/usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
/PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/plugins/*:/PredictionIO-0.11.0-incubating/lib/spark/*:/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
-javaagent:/PredictionIO-0.11.0-incubating/lib/newrelic/newrelic.jar
-Dpio.log.dir=/root org.apache.predictionio.tools.console.Console deploy
--port 8080 --engine-instance-id 994cf863-3fc7-48e9-8fce-c206f9ac10e1
--pio-home /PredictionIO-0.11.0-incubating*
*root 94 2.2 3.1 4686384 519036 ? Sl 21:35 0:32
/usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
/PredictionIO-0.11.0-incubating/conf/:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/conf/:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/jars/*
-Xmx1g -Dpio.log.dir=/root org.apache.spark.deploy.SparkSubmit --conf
spark.driver.extraClassPath=/PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
--conf spark.driver.extraJavaOptions=-Dpio.log.dir=/root --class
org.apa....*
However, the jetty was started by the second java process 94, and any
metric arrives in NewRelic.
Second, I've tried to override the arguments from "pio deploy" as the
example below:
*pio deploy --port 8080 --engine-instance-id ${ENGINE_INSTANCE_ID} **-- --
-javaagent:/PredictionIO-0.11.0-incubating/lib/newrelic/newrelic.jar*
Looking at the "pio help" the first "--" is the arguments for spark and the
second "--" , but the '-javaagent' argument is still executed in the first
java process:
*root 12 4.9 1.3 6825028 223544 ? Sl 17:08 0:05
/usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
/PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/plugins/*:/PredictionIO-0.11.0-incubating/lib/spark/*:/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
-Dpio.log.dir=/root org.apache.predictionio.tools.console.Console deploy
--port 8080 --engine-instance-id 994cf863-3fc7-48e9-8fce-c206f9ac10e1
--pio-home /PredictionIO-0.11.0-incubating -- --
-javaagent:/PredictionIO-0.11.0-incubating/lib/newrelic/newrelic.jar*
*root 72 15.7 2.6 4684976 426376 ? Sl 17:08 0:17
/usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
/PredictionIO-0.11.0-incubating/conf/:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/conf/:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/jars/*
-Xmx1g -Dpio.log.dir=/root org.apache.spark.deploy.SparkSubmit --conf
spark.driver.extraClassPath=/PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
--conf spark.driver.extraJavaOptions=-Dpio.log.dir=/root --class
org.apache.predictionio.workflow.CreateServer ...*
We know that's Newrelic is not a common use case for PredictionIO, so we
are just trying to find a way to make the Jetty process receive the
"-javaagent" as an argument in the JVM,
If you need any other information please let me know.
Thanks !
