Hi everyone,
I faced a strange situation recently.
One of our engines has frequently old processes after daily redeployment.
We used to redeploy by crontab rule that looks like:
nohup pio deploy --port 8004 >> ${DEPLOY_LOG_FILE} 2>&1 &
This is output of ps aux command:
[da_mcom_milan@mdc2pr113 ~]$ ps aux | grep 8004
da_mcom+ 7854 0.0 0.0 112660 996 pts/1 S+ 21:34 0:00 grep
--color=auto *8004*
1. da_mcom+ 30932 0.6 0.9 26473632 595932 ? Sl 01:57 7:51 java
-cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/
pio-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.
0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
-Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
2. da_mcom+ 31097 1.2 1.2 17065124 822596 ? Sl 01:57 14:48
/usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
/data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
hdp/current/spark-client/conf/:/data/PredictionIO/vendors/
spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.
3.0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-
api-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/
datanucleus-core-3.2.10.jar:/data/PredictionIO/vendors/
spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/
hadoop/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-
sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-
java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
-Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d
ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/
hadoop/conf:/data/PredictionIO/vendors/hbase/conf --conf
spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class
org.apache.predictionio.workflow.CreateServer --jars
file:/data/PredictionIO/some-engine/target/scala-2.10/
universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/
data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elast
icsearch1-assembly-0.11.0-incubating.jar,file:/data/Pred
ictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-
assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.
11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incu
bating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/
spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/
data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-local
fs-assembly-0.11.0-incubating.jar --files file:/data/PredictionIO-0.11.0
-incubating/conf/log4j.properties,file:/usr/hdp/2.5.3.0-37/
hadoop/conf/core-site.xml,file:/data/PredictionIO/vendors/hbase/conf/hbase-site.xml
file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
--engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port *8004*
--event-server-ip 0.0.0.0 --event-server-port 7070 --json-extractor Both
--env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_
STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/
home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEA
RCH_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/
data/PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0.11.0-
incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/
engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/da_mcom_
milan/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_
TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_
SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_
STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_
STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros-dev,PIO_FS_
TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_
REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_
REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-
incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=
9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
3. da_mcom+ 32505 0.6 0.8 26473632 581164 ? Sl 02:00 7:54 java
-cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/
pio-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.
0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
-Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
4. da_mcom+ 32671 1.4 1.3 19757488 881448 ? Sl 02:00 16:27
/usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
/data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
hdp/current/spark-client/conf/:/data/PredictionIO/vendors/
spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.
3.0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-
api-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/
datanucleus-core-3.2.10.jar:/data/PredictionIO/vendors/
spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/
hadoop/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-
sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-
java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
-Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d
ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/
hadoop/conf:/data/PredictionIO/vendors/hbase/conf --conf
spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class
org.apache.predictionio.workflow.CreateServer --jars
file:/data/PredictionIO/some-engine/target/scala-2.10/
universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/
data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elast
icsearch1-assembly-0.11.0-incubating.jar,file:/data/Pred
ictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-
assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.
11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incu
bating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/
spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/
data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-local
fs-assembly-0.11.0-incubating.jar --files file:/data/PredictionIO-0.11.0
-incubating/conf/log4j.properties,file:/usr/hdp/2.5.3.0-37/
hadoop/conf/core-site.xml,file:/data/PredictionIO/vendors/hbase/conf/hbase-site.xml
file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
--engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port *8004*
--event-server-ip 0.0.0.0 --event-server-port 7070 --json-extractor Both
--env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_
STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/
home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEA
RCH_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/
data/PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0.11.0-
incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/
engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/da_mcom_
milan/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_
TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_
SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_
STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_
STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros-dev,PIO_FS_
TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_
REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_
REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-
incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=
9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
For some reasons, processes 30932 and 31097 still alive and process
listened by 32671
Any ideas?
Thank you,
George
