Have you try examining the output of lsof on those processes? It may tell you which open I/O is holding up the JVM.
On Wed, Apr 11, 2018 at 4:42 PM George Yarish <[email protected]> wrote: > That totally makes sense. We use our custom algorithm based on Universal > Recommender (UR), so we use elasticsearch and hbase in serving code. But we > use it via PIO api. > For hbase we use > org.apache.predictionio.data.storage.hbase.HBLEvents.futureFind > For elasticsearch we use EsClient from UR code. > > On Wed, Apr 11, 2018 at 4:14 PM, Donald Szeto <[email protected]> wrote: > >> What engine template are you running? I think the server inside the JVM >> process has already shut down (otherwise the new one won't be able to bind >> to the same port), but it may have some I/O (e.g. database connections) >> that is still active which would have hold the JVM process from >> terminating. Is there a customization like that in your serving code? >> >> On Wed, Apr 11, 2018 at 4:09 PM, George Yarish <[email protected]> >> wrote: >> >>> I'm sorry for that, I have a few environments in which that problem >>> reproduced. That was the one with *8000* port. >>> It's pretty the same: >>> >>> $ ps aux | grep *8000* | wc -l >>> 13 >>> >>> and processes looks like: >>> >>> da_mcom+ 17028 0.5 0.9 26473704 603220 ? Sl апр06 42:05 java >>> -cp >>> /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incubating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/usr/hdp/ >>> 2.6.1.0-129/hbase/conf -Dpio.log.dir=/home/da_mcom_milan >>> org.apache.predictionio.tools.console.Console deploy --port *8000* >>> --pio-home /data/PredictionIO-0.11.0-incubating >>> da_mcom+ 17269 1.1 1.5 19765888 996524 ? Sl апр06 87:45 >>> /usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp >>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.1.0-129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar >>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan >>> org.apache.spark.deploy.SparkSubmit --conf >>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf >>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan >>> --class org.apache.predictionio.workflow.CreateServer --jars >>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar >>> --files >>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.properties,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml >>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar >>> --engineInstanceId AWKbr8hAdWI0xyRGjxPY --engine-variant >>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json >>> --ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0 >>> --event-server-port 7070 --json-extractor Both --env >>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,mdc2pr069,mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/usr/hdp/2.6.1.0-129/hbase,PIO_HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_SOURCES_HDFS_PATH=/tmp/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros_es_prod,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300 >>> da_mcom+ 17305 11.9 1.0 17214656 662920 pts/1 Sl 19:00 0:46 >>> /usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp >>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.1.0-129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar >>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan >>> org.apache.spark.deploy.SparkSubmit --conf >>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf >>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan >>> --class org.apache.predictionio.workflow.CreateServer --jars >>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar >>> --files >>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.properties,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml >>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar >>> --engineInstanceId AWK1b5NndWI0xyRGBxWM --engine-variant >>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json >>> --ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0 >>> --event-server-port 7070 --json-extractor Both --env >>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,mdc2pr069,mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/usr/hdp/2.6.1.0-129/hbase,PIO_HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_SOURCES_HDFS_PATH=/tmp/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros_es_prod,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300 >>> da_mcom+ 18988 0.0 0.0 112652 984 pts/1 R+ 19:06 0:00 grep >>> --color=auto *8000* >>> da_mcom+ 48849 0.5 0.9 26473704 607720 ? Sl апр05 50:01 java >>> -cp >>> /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incubating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/usr/hdp/ >>> 2.6.1.0-129/hbase/conf -Dpio.log.dir=/home/da_mcom_milan >>> org.apache.predictionio.tools.console.Console deploy --port *8000* >>> --pio-home /data/PredictionIO-0.11.0-incubating >>> da_mcom+ 49016 1.1 1.4 19766916 967792 ? Sl апр05 102:52 >>> /usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp >>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.1.0-129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar >>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan >>> org.apache.spark.deploy.SparkSubmit --conf >>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf >>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan >>> --class org.apache.predictionio.workflow.CreateServer --jars >>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar >>> --files >>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.properties,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml >>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar >>> --engineInstanceId AWKWiWsxdWI0xyRGjxPJ --engine-variant >>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json >>> --ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0 >>> --event-server-port 7070 --json-extractor Both --env >>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,mdc2pr069,mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/usr/hdp/2.6.1.0-129/hbase,PIO_HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_SOURCES_HDFS_PATH=/tmp/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros_es_prod,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300 >>> >>> >>> >>> On Wed, Apr 11, 2018 at 3:39 PM, Donald Szeto <[email protected]> wrote: >>> >>>> Are you able to find any occurrence of port 8004? The excerpt you >>>> showed was about port 8000. >>>> >>>> On Wed, Apr 11, 2018 at 3:18 PM, George Yarish < >>>> [email protected]> wrote: >>>> >>>>> Hi Donald >>>>> >>>>> Thank you for the answer. >>>>> it looks fine to me, but processes still alive: >>>>> >>>>> 2018-04-11 13:00:25,331 INFO >>>>> org.apache.predictionio.workflow.MasterActor >>>>> [pio-server-akka.actor.default-dispatcher-5] - Undeploying any existing >>>>> engine instance at http://0.0.0.0:8000 >>>>> 2018-04-11 13:00:26,467 INFO >>>>> org.apache.predictionio.workflow.MasterActor >>>>> [pio-server-akka.actor.default-dispatcher-9] - Stop server command >>>>> received. >>>>> 2018-04-11 13:00:26,468 INFO >>>>> org.apache.predictionio.workflow.MasterActor >>>>> [pio-server-akka.actor.default-dispatcher-9] - Server is shutting down. >>>>> 2018-04-11 13:00:26,617 INFO spray.can.server.HttpListener >>>>> [pio-server-akka.actor.default-dispatcher-2] - Bound to /0.0.0.0:8000 >>>>> 2018-04-11 13:00:26,620 INFO >>>>> org.apache.predictionio.workflow.MasterActor >>>>> [pio-server-akka.actor.default-dispatcher-4] - Engine is deployed and >>>>> running. Engine API is live at http://0.0.0.0:8000 >>>>> >>>>> Thank you, >>>>> George >>>>> >>>>> >>>>> On Wed, Apr 11, 2018 at 11:48 AM, Donald Szeto <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi George, >>>>>> >>>>>> Mind checking the content of pio.log at where you run `pio deploy`? >>>>>> It should contain some messages regarding undeploying existing engine >>>>>> servers on the same port. >>>>>> >>>>>> Regards, >>>>>> Donald >>>>>> >>>>>> On Thu, Apr 5, 2018 at 6:43 PM, George Yarish < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi everyone, >>>>>>> >>>>>>> I faced a strange situation recently. >>>>>>> One of our engines has frequently old processes after daily >>>>>>> redeployment. >>>>>>> >>>>>>> We used to redeploy by crontab rule that looks like: >>>>>>> nohup pio deploy --port 8004 >> ${DEPLOY_LOG_FILE} 2>&1 & >>>>>>> >>>>>>> >>>>>>> This is output of ps aux command: >>>>>>> >>>>>>> [da_mcom_milan@mdc2pr113 ~]$ ps aux | grep 8004 >>>>>>> >>>>>>> da_mcom+ 7854 0.0 0.0 112660 996 pts/1 S+ 21:34 0:00 >>>>>>> grep --color=auto *8004* >>>>>>> >>>>>>> 1. da_mcom+ 30932 0.6 0.9 26473632 595932 ? Sl 01:57 7:51 >>>>>>> java -cp >>>>>>> /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incubating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/usr/hdp/ >>>>>>> 2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf >>>>>>> -Dpio.log.dir=/home/da_mcom_milan >>>>>>> org.apache.predictionio.tools.console.Console deploy --port *8004* >>>>>>> --pio-home /data/PredictionIO-0.11.0-incubating >>>>>>> >>>>>>> 2. da_mcom+ 31097 1.2 1.2 17065124 822596 ? Sl 01:57 14:48 >>>>>>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp >>>>>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/PredictionIO/vendors/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar >>>>>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan >>>>>>> org.apache.spark.deploy.SparkSubmit --conf >>>>>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf >>>>>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan >>>>>>> --class org.apache.predictionio.workflow.CreateServer --jars >>>>>>> file:/data/PredictionIO/some-engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/ >>>>>>> some-engine/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar >>>>>>> --files >>>>>>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.properties,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:/data/PredictionIO/vendors/hbase/conf/hbase-site.xml >>>>>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar >>>>>>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant >>>>>>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port >>>>>>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070 >>>>>>> --json-extractor Both --env >>>>>>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/da_mcom_milan/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs >>>>>>> >>>>>>> 3. da_mcom+ 32505 0.6 0.8 26473632 581164 ? Sl 02:00 7:54 >>>>>>> java -cp >>>>>>> /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incubating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/usr/hdp/ >>>>>>> 2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf >>>>>>> -Dpio.log.dir=/home/da_mcom_milan >>>>>>> org.apache.predictionio.tools.console.Console deploy --port *8004* >>>>>>> --pio-home /data/PredictionIO-0.11.0-incubating >>>>>>> >>>>>>> 4. da_mcom+ 32671 1.4 1.3 19757488 881448 ? Sl 02:00 16:27 >>>>>>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp >>>>>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/PredictionIO/vendors/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar >>>>>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan >>>>>>> org.apache.spark.deploy.SparkSubmit --conf >>>>>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf >>>>>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan >>>>>>> --class org.apache.predictionio.workflow.CreateServer --jars >>>>>>> file:/data/PredictionIO/some-engine >>>>>>> /target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/ >>>>>>> some-engine/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar >>>>>>> --files >>>>>>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.properties,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:/data/PredictionIO/vendors/hbase/conf/hbase-site.xml >>>>>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar >>>>>>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant >>>>>>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port >>>>>>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070 >>>>>>> --json-extractor Both --env >>>>>>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/da_mcom_milan/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs >>>>>>> >>>>>>> For some reasons, processes 30932 and 31097 still alive and process >>>>>>> listened by 32671 >>>>>>> Any ideas? >>>>>>> >>>>>>> Thank you, >>>>>>> George >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> George Yarish, Java Developer >>>>> >>>>> Grid Dynamics >>>>> >>>>> 197101, Rentgena Str., 5A, St.Petersburg, Russia >>>>> >>>>> Cell: +7 950 030-1941 >>>>> >>>>> Read Grid Dynamics' Tech Blog >>>>> <http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email> >>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> >>> George Yarish, Java Developer >>> >>> Grid Dynamics >>> >>> 197101, Rentgena Str., 5A, St.Petersburg, Russia >>> >>> Cell: +7 950 030-1941 >>> >>> Read Grid Dynamics' Tech Blog >>> <http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email> >>> >>> >>> >> > > > -- > > George Yarish, Java Developer > > Grid Dynamics > > 197101, Rentgena Str., 5A, St.Petersburg, Russia > > Cell: +7 950 030-1941 > > Read Grid Dynamics' Tech Blog > <http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email> > > >
