Are you running Spark on YARN? If so, you can get to the Spark UI via the YARN ResourceManager. Each running Spark application will have a link on the YARN ResourceManager labeled "ApplicationMaster". If you click that, it will take you to the Spark UI, even if it is running on a slave node in the case of yarn-cluster mode. It does this by proxying the Spark UI through the YARN Proxy Server on the master node.
For completed applications, the link will be labeled "History" and will take you to the Spark History Server (provided you have set spark.yarn.historyServer.address in spark-defaults.conf). As for getting the URL programmatically, the URL using the YARN ProxyServer is easy to determine. It's just http://<master node address>:<YARN ProxyServer port>/proxy/<YARN application ID>. (e.g., http://ip-10-150-65-11.ec2.internal:20888/proxy/application_1450128858020_0001/) Then again, I'm not sure how easy it is to get the YARN application ID for a Spark application without parsing the spark-submit logs. Or at least I think I remember some other thread where that was mentioned. ~ Jonathan On Mon, Dec 14, 2015 at 1:57 PM, Ashish Nigam <ashnigamt...@gmail.com> wrote: > Hi, > I run spark streaming job in cluster mode. This means that driver can run > in any data node. And Spark UI can run in any dynamic port. > At present, I know about the port by looking at container logs that look > something like this - > > server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:50571 > INFO util.Utils: Successfully started service 'SparkUI' on port 50571. > INFO ui.SparkUI: Started SparkUI at http://xxx:50571 > > > Is there any way to know about the UI port automatically using some API? > > Thanks > Ashish >
