wuchang created SPARK-20578:
-------------------------------
Summary: oozie submit spark job, the driver ui port is 0, instead
of 4040
Key: SPARK-20578
URL: https://issues.apache.org/jira/browse/SPARK-20578
Project: Spark
Issue Type: Question
Components: Web UI
Affects Versions: 2.1.0
Reporter: wuchang
My spark Job is submmited by oozie in hue. The spark is running in yarn-cluster
mode. I am trying to monitor the running application status by so-called
driver's 4040 port, but I cannot find the 4040 port , I check the process:
{code}
appuser 137872 137870 0 18:55 ? 00:00:00 /bin/bash -c /home/jdk/bin/java
-server -Xmx4096m
-Djava.io.tmpdir=/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0547/container_1493800575189_0547_01_000004/tmp
'-Dspark.driver.port=36503' '-Dspark.ui.port=0'
-Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493800575189_0547/container_1493800575189_0547_01_000004
-XX:OnOutOfMemoryError='kill %p'
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://[email protected]:36503 --executor-id 3 --hostname
10.120.117.100 --cores 1 --app-id application_1493800575189_0547
--user-class-path
file:/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0547/container_1493800575189_0547_01_000004/__app__.jar
1>
/home/log/hadoop/logs/userlogs/application_1493800575189_0547/container_1493800575189_0547_01_000004/stdout
2>
/home/log/hadoop/logs/userlogs/application_1493800575189_0547/container_1493800575189_0547_01_000004/stderr
appuser 138337 137872 99 18:55 ? 00:05:11 /home/jdk/bin/java -server -Xmx4096m
-Djava.io.tmpdir=/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0547/container_1493800575189_0547_01_000004/tmp
-Dspark.driver.port=36503 -Dspark.ui.port=0
-Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493800575189_0547/container_1493800575189_0547_01_000004
-XX:OnOutOfMemoryError=kill %p
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://[email protected]:36503 --executor-id 3 --hostname
10.120.117.100 --cores 1 --app-id application_1493800575189_0547
--user-class-path
file:/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0547/container_1493800575189_0547_01_000004/__app__.jar
{code}
I don't know why the spark.ui.port is 0 instead of 4040. Of course , the port 0
is not allowed by my linux system.So , I cannot monitor the application status
from REST api.
Anybody can give me some suggestions?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
