Hi George, Mind checking the content of pio.log at where you run `pio deploy`? It should contain some messages regarding undeploying existing engine servers on the same port.
Regards, Donald On Thu, Apr 5, 2018 at 6:43 PM, George Yarish <[email protected]> wrote: > Hi everyone, > > I faced a strange situation recently. > One of our engines has frequently old processes after daily redeployment. > > We used to redeploy by crontab rule that looks like: > nohup pio deploy --port 8004 >> ${DEPLOY_LOG_FILE} 2>&1 & > > > This is output of ps aux command: > > [da_mcom_milan@mdc2pr113 ~]$ ps aux | grep 8004 > > da_mcom+ 7854 0.0 0.0 112660 996 pts/1 S+ 21:34 0:00 grep > --color=auto *8004* > > 1. da_mcom+ 30932 0.6 0.9 26473632 595932 ? Sl 01:57 7:51 java > -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO > -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub > ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p > io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/ > hadoop/conf:/data/PredictionIO/vendors/hbase/conf > -Dpio.log.dir=/home/da_mcom_milan > org.apache.predictionio.tools.console.Console > deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating > > 2. da_mcom+ 31097 1.2 1.2 17065124 822596 ? Sl 01:57 14:48 > /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp > /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0- > 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/ > hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp > ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3. > 0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-ap > i-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanuc > leus-core-3.2.10.jar:/data/PredictionIO/vendors/spark/ > lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop > /conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1. > 10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java- > sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar > -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan > org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d > ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/ > hadoop/conf:/data/PredictionIO/vendors/hbase/conf --conf > spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class > org.apache.predictionio.workflow.CreateServer --jars > file:/data/PredictionIO/some-engine/target/scala-2.10/univer > sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine > /target/scala-2.10/universal-recommender-assembly > -0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating > /lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubatin > g.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/ > pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/ > PredictionIO-0.11.0-incubating/lib/spark/pio-data- > hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO- > 0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0- > incubating.jar,file:/data/PredictionIO-0.11.0-incubating > /lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar --files > file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert > ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file: > /data/PredictionIO/vendors/hbase/conf/hbase-site.xml > file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar > --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant > file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port *8004* > --event-server-ip 0.0.0.0 --event-server-port 7070 --json-extractor Both > --env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_ST > ORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/ho > me/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARC > H_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/data/ > PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0. > 11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_ > milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/ > home/da_mcom_milan/.pio_store/models,PIO_STORAGE_SOURCES_ > ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE= > ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE= > LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME= > pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME= > pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp, > PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_ > STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_ > CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_ > STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_ > SOURCES_LOCALFS_TYPE=localfs > > 3. da_mcom+ 32505 0.6 0.8 26473632 581164 ? Sl 02:00 7:54 java > -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO > -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub > ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p > io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/ > hadoop/conf:/data/PredictionIO/vendors/hbase/conf > -Dpio.log.dir=/home/da_mcom_milan > org.apache.predictionio.tools.console.Console > deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating > > 4. da_mcom+ 32671 1.4 1.3 19757488 881448 ? Sl 02:00 16:27 > /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp > /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0- > 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/ > hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp > ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3. > 0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-ap > i-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanuc > leus-core-3.2.10.jar:/data/PredictionIO/vendors/spark/ > lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop > /conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1. > 10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java- > sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar > -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan > org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d > ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/ > hadoop/conf:/data/PredictionIO/vendors/hbase/conf --conf > spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class > org.apache.predictionio.workflow.CreateServer --jars > file:/data/PredictionIO/some-engine/target/scala-2.10/univer > sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine > /target/scala-2.10/universal-recommender-assembly > -0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating > /lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubatin > g.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/ > pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/ > PredictionIO-0.11.0-incubating/lib/spark/pio-data- > hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO- > 0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0- > incubating.jar,file:/data/PredictionIO-0.11.0-incubating > /lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar --files > file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert > ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file: > /data/PredictionIO/vendors/hbase/conf/hbase-site.xml > file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar > --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant > file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port *8004* > --event-server-ip 0.0.0.0 --event-server-port 7070 --json-extractor Both > --env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_ST > ORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/ho > me/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARC > H_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/data/ > PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0. > 11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_ > milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/ > home/da_mcom_milan/.pio_store/models,PIO_STORAGE_SOURCES_ > ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE= > ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE= > LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME= > pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME= > pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp, > PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_ > STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_ > CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_ > STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_ > SOURCES_LOCALFS_TYPE=localfs > > For some reasons, processes 30932 and 31097 still alive and process > listened by 32671 > Any ideas? > > Thank you, > George >
