Re: PredictionIO NewRelic Monitoring

2017-09-29 Thread Rodrigo Vieira 
Hi Guys!

Thanks for the support! It worked for us.

Example command:

pio deploy --port 8080 --engine-instance-id ${ENGINE_INSTANCE_ID} --
--driver-java-options
"-javaagent:/PredictionIO-0.11.0-incubating/lib/newrelic/newrelic.jar"

Thanks

On Thu, Sep 28, 2017 at 8:30 PM Mars Hall  wrote:

> A quick addition to Donald's answer, because this config style bit us
> (please ignore the actual config value, this is just an example):
>
> # Options are silently dropped when set through `--conf`.# Bad example; don't 
> use this:--conf 
> 'spark.driver.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4'
> # Instead, pass them using `--driver-java-options`.# Good example; do 
> this:--driver-java-options '-Dcom.amazonaws.services.s3.enableV4'
>
>
>
> On Fri, Sep 29, 2017 at 06:47 Donald Szeto  wrote:
>
>> Hey,
>>
>> I think you are probably looking for a passthrough argument like this.
>>
>> pio deploy ... -- --conf "spark.driver.extraJavaOptions=-javaagent:..."
>>
>> Please take a look at
>> http://spark.apache.org/docs/latest/configuration.html for more details.
>>
>> Regards,
>> Donald
>>
>> On Thu, Sep 28, 2017 at 11:31 AM, Rodrigo Vieira <
>> rodrigo.vie...@guiabolso.com.br> wrote:
>>
>>> Hi guys,
>>>
>>> Anyone knows how to add the "-javaagent" argument in the Jetty bootstrap?
>>>
>>> The main motivation is to get some metrics from PredictionIO using the
>>> APM NewRelic. Just to add some context about our architecture, we've
>>> created a container that runs the "pio deploy" inside a Kubernetes cluster,
>>> NewRelic is the official monitoring tool for our company that's why I'm
>>> trying to configure NewRelic.
>>>
>>> I've already tried some workarounds but no success yet.
>>>
>>> First, I've tried to override the JVM_OPTS env, so I realized that
>>> PredicitonIO executes two java processes when the "pio deploy" is executed,
>>> see the example below:
>>>
>>> ps -auxww
>>> *root12  2.2  2.6 7343400 434516 ?  Sl   21:35   0:32
>>> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
>>> /PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/plugins/*:/PredictionIO-0.11.0-incubating/lib/spark/*:/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
>>> -javaagent:/PredictionIO-0.11.0-incubating/lib/newrelic/newrelic.jar
>>> -Dpio.log.dir=/root org.apache.predictionio.tools.console.Console deploy
>>> --port 8080 --engine-instance-id 994cf863-3fc7-48e9-8fce-c206f9ac10e1
>>> --pio-home /PredictionIO-0.11.0-incubating*
>>>
>>> *root94  2.2  3.1 4686384 519036 ?  Sl   21:35   0:32
>>> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
>>> /PredictionIO-0.11.0-incubating/conf/:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/conf/:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/jars/*
>>> -Xmx1g -Dpio.log.dir=/root org.apache.spark.deploy.SparkSubmit --conf
>>> spark.driver.extraClassPath=/PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/root --class
>>> org.apa*
>>>
>>> However, the jetty was started by the second java process 94, and any
>>> metric arrives in NewRelic.
>>>
>>> Second, I've tried to override the arguments from "pio deploy" as the
>>> example below:
>>>
>>> *pio deploy --port 8080 --engine-instance-id ${ENGINE_INSTANCE_ID} **--
>>> -- -javaagent:/PredictionIO-0.11.0-incubating/lib/newrelic/newrelic.jar*
>>>
>>> Looking at the "pio help" the first "--" is the arguments for spark and
>>> the second "--" , but the '-javaagent' argument is still executed in the
>>> first java process:
>>>
>>>
>>> *root12  4.9  1.3 6825028 223544 ?  Sl   17:08   0:05
>>> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
>>> /PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/plugins/*:/PredictionIO-0.11.0-incubating/lib/spark/*:/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
>>> -Dpio.log.dir=/root org.apache.predictionio.tools.console.Console deploy
>>> --port 8080 --engine-instance-id 994cf863-3fc7-48e9-8fce-c206f9ac10e1
>>> --pio-home /PredictionIO-0.11.0-incubating -- --
>>> -javaagent:/PredictionIO-0.11.0-incubating/lib/newrelic/newrelic.jar*
>>>
>>> *root72 15.7  2.6 4684976 426376 ?  Sl   17:08   0:17
>>> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
>>>

Re: PredictionIO NewRelic Monitoring

2017-09-28 Thread Mars Hall 
A quick addition to Donald's answer, because this config style bit us
(please ignore the actual config value, this is just an example):

# Options are silently dropped when set through `--conf`.# Bad
example; don't use this:--conf
'spark.driver.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4'
# Instead, pass them using `--driver-java-options`.# Good example; do
this:--driver-java-options '-Dcom.amazonaws.services.s3.enableV4'



On Fri, Sep 29, 2017 at 06:47 Donald Szeto  wrote:

> Hey,
>
> I think you are probably looking for a passthrough argument like this.
>
> pio deploy ... -- --conf "spark.driver.extraJavaOptions=-javaagent:..."
>
> Please take a look at
> http://spark.apache.org/docs/latest/configuration.html for more details.
>
> Regards,
> Donald
>
> On Thu, Sep 28, 2017 at 11:31 AM, Rodrigo Vieira <
> rodrigo.vie...@guiabolso.com.br> wrote:
>
>> Hi guys,
>>
>> Anyone knows how to add the "-javaagent" argument in the Jetty bootstrap?
>>
>> The main motivation is to get some metrics from PredictionIO using the
>> APM NewRelic. Just to add some context about our architecture, we've
>> created a container that runs the "pio deploy" inside a Kubernetes cluster,
>> NewRelic is the official monitoring tool for our company that's why I'm
>> trying to configure NewRelic.
>>
>> I've already tried some workarounds but no success yet.
>>
>> First, I've tried to override the JVM_OPTS env, so I realized that
>> PredicitonIO executes two java processes when the "pio deploy" is executed,
>> see the example below:
>>
>> ps -auxww
>> *root12  2.2  2.6 7343400 434516 ?  Sl   21:35   0:32
>> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
>> /PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/plugins/*:/PredictionIO-0.11.0-incubating/lib/spark/*:/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
>> -javaagent:/PredictionIO-0.11.0-incubating/lib/newrelic/newrelic.jar
>> -Dpio.log.dir=/root org.apache.predictionio.tools.console.Console deploy
>> --port 8080 --engine-instance-id 994cf863-3fc7-48e9-8fce-c206f9ac10e1
>> --pio-home /PredictionIO-0.11.0-incubating*
>>
>> *root94  2.2  3.1 4686384 519036 ?  Sl   21:35   0:32
>> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
>> /PredictionIO-0.11.0-incubating/conf/:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/conf/:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/jars/*
>> -Xmx1g -Dpio.log.dir=/root org.apache.spark.deploy.SparkSubmit --conf
>> spark.driver.extraClassPath=/PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/root --class
>> org.apa*
>>
>> However, the jetty was started by the second java process 94, and any
>> metric arrives in NewRelic.
>>
>> Second, I've tried to override the arguments from "pio deploy" as the
>> example below:
>>
>> *pio deploy --port 8080 --engine-instance-id ${ENGINE_INSTANCE_ID} **--
>> -- -javaagent:/PredictionIO-0.11.0-incubating/lib/newrelic/newrelic.jar*
>>
>> Looking at the "pio help" the first "--" is the arguments for spark and
>> the second "--" , but the '-javaagent' argument is still executed in the
>> first java process:
>>
>>
>> *root12  4.9  1.3 6825028 223544 ?  Sl   17:08   0:05
>> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
>> /PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/plugins/*:/PredictionIO-0.11.0-incubating/lib/spark/*:/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
>> -Dpio.log.dir=/root org.apache.predictionio.tools.console.Console deploy
>> --port 8080 --engine-instance-id 994cf863-3fc7-48e9-8fce-c206f9ac10e1
>> --pio-home /PredictionIO-0.11.0-incubating -- --
>> -javaagent:/PredictionIO-0.11.0-incubating/lib/newrelic/newrelic.jar*
>>
>> *root72 15.7  2.6 4684976 426376 ?  Sl   17:08   0:17
>> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
>> /PredictionIO-0.11.0-incubating/conf/:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/conf/:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/jars/*
>> -Xmx1g -Dpio.log.dir=/root org.apache.spark.deploy.SparkSubmit --conf
>>

Re: PredictionIO NewRelic Monitoring

2017-09-28 Thread Donald Szeto 
Hey,

I think you are probably looking for a passthrough argument like this.

pio deploy ... -- --conf "spark.driver.extraJavaOptions=-javaagent:..."

Please take a look at http://spark.apache.org/docs/latest/configuration.html
for more details.

Regards,
Donald

On Thu, Sep 28, 2017 at 11:31 AM, Rodrigo Vieira <
rodrigo.vie...@guiabolso.com.br> wrote:

> Hi guys,
>
> Anyone knows how to add the "-javaagent" argument in the Jetty bootstrap?
>
> The main motivation is to get some metrics from PredictionIO using the APM
> NewRelic. Just to add some context about our architecture, we've created a
> container that runs the "pio deploy" inside a Kubernetes cluster, NewRelic
> is the official monitoring tool for our company that's why I'm trying to
> configure NewRelic.
>
> I've already tried some workarounds but no success yet.
>
> First, I've tried to override the JVM_OPTS env, so I realized that
> PredicitonIO executes two java processes when the "pio deploy" is executed,
> see the example below:
>
> ps -auxww
> *root12  2.2  2.6 7343400 434516 ?  Sl   21:35   0:32
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
> /PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/plugins/*:/PredictionIO-0.11.0-incubating/lib/spark/*:/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
> -javaagent:/PredictionIO-0.11.0-incubating/lib/newrelic/newrelic.jar
> -Dpio.log.dir=/root org.apache.predictionio.tools.console.Console deploy
> --port 8080 --engine-instance-id 994cf863-3fc7-48e9-8fce-c206f9ac10e1
> --pio-home /PredictionIO-0.11.0-incubating*
>
> *root94  2.2  3.1 4686384 519036 ?  Sl   21:35   0:32
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
> /PredictionIO-0.11.0-incubating/conf/:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/conf/:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/jars/*
> -Xmx1g -Dpio.log.dir=/root org.apache.spark.deploy.SparkSubmit --conf
> spark.driver.extraClassPath=/PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/root --class
> org.apa*
>
> However, the jetty was started by the second java process 94, and any
> metric arrives in NewRelic.
>
> Second, I've tried to override the arguments from "pio deploy" as the
> example below:
>
> *pio deploy --port 8080 --engine-instance-id ${ENGINE_INSTANCE_ID} **--
> -- -javaagent:/PredictionIO-0.11.0-incubating/lib/newrelic/newrelic.jar*
>
> Looking at the "pio help" the first "--" is the arguments for spark and
> the second "--" , but the '-javaagent' argument is still executed in the
> first java process:
>
>
> *root12  4.9  1.3 6825028 223544 ?  Sl   17:08   0:05
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
> /PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/plugins/*:/PredictionIO-0.11.0-incubating/lib/spark/*:/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
> -Dpio.log.dir=/root org.apache.predictionio.tools.console.Console deploy
> --port 8080 --engine-instance-id 994cf863-3fc7-48e9-8fce-c206f9ac10e1
> --pio-home /PredictionIO-0.11.0-incubating -- --
> -javaagent:/PredictionIO-0.11.0-incubating/lib/newrelic/newrelic.jar*
>
> *root72 15.7  2.6 4684976 426376 ?  Sl   17:08   0:17
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
> /PredictionIO-0.11.0-incubating/conf/:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/conf/:/PredictionIO-0.11.0-incubating/vendors/spark-2.1.0-bin-hadoop2.7/jars/*
> -Xmx1g -Dpio.log.dir=/root org.apache.spark.deploy.SparkSubmit --conf
> spark.driver.extraClassPath=/PredictionIO-0.11.0-incubating/conf:/PredictionIO-0.11.0-incubating/lib/postgresql-42.0.0.jar:/PredictionIO-0.11.0-incubating/lib/mysql-connector-java-5.1.41.jar
> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/root --class
> org.apache.predictionio.workflow.CreateServer ...*
>
> We know that's Newrelic is not a common use case for PredictionIO, so we
> are just trying to find a way to make the Jetty process receive the
> "-javaagent" as an argument in the JVM,
>
> If you need any other information please let me know.
>
> Thanks !
>
>
>