Oh, nice, I did not know about that property. Thanks! On Mon, Dec 14, 2015 at 4:28 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> w.r.t. getting application Id, please take a look at the following > in SparkContext : > > /** > * A unique identifier for the Spark application. > * Its format depends on the scheduler implementation. > * (i.e. > * in case of local spark app something like 'local-1433865536131' > * in case of YARN something like 'application_1433865536131_34483' > * ) > */ > def applicationId: String = _applicationId > > On Mon, Dec 14, 2015 at 2:33 PM, Jonathan Kelly <jonathaka...@gmail.com> > wrote: > >> Are you running Spark on YARN? If so, you can get to the Spark UI via the >> YARN ResourceManager. Each running Spark application will have a link on >> the YARN ResourceManager labeled "ApplicationMaster". If you click that, it >> will take you to the Spark UI, even if it is running on a slave node in the >> case of yarn-cluster mode. It does this by proxying the Spark UI through >> the YARN Proxy Server on the master node. >> >> For completed applications, the link will be labeled "History" and will >> take you to the Spark History Server (provided you have >> set spark.yarn.historyServer.address in spark-defaults.conf). >> >> As for getting the URL programmatically, the URL using the YARN >> ProxyServer is easy to determine. It's just http://<master node >> address>:<YARN ProxyServer port>/proxy/<YARN application ID>. (e.g., >> http://ip-10-150-65-11.ec2.internal:20888/proxy/application_1450128858020_0001/) >> Then again, I'm not sure how easy it is to get the YARN application ID for >> a Spark application without parsing the spark-submit logs. Or at least I >> think I remember some other thread where that was mentioned. >> >> ~ Jonathan >> >> On Mon, Dec 14, 2015 at 1:57 PM, Ashish Nigam <ashnigamt...@gmail.com> >> wrote: >> >>> Hi, >>> I run spark streaming job in cluster mode. This means that driver can >>> run in any data node. And Spark UI can run in any dynamic port. >>> At present, I know about the port by looking at container logs that look >>> something like this - >>> >>> server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:50571 >>> INFO util.Utils: Successfully started service 'SparkUI' on port 50571. >>> INFO ui.SparkUI: Started SparkUI at http://xxx:50571 >>> >>> >>> Is there any way to know about the UI port automatically using some API? >>> >>> Thanks >>> Ashish >>> >> >> >
