On YARN, spark does not manage the cluster, but YARN does. Usually the
cluster manager UI is under http://:9026/cluster. I believe
that it chooses the port for the spark driver UI randomly, but an easy way
of accessing it is by clicking on the "Application Master" link under the
"Tracking UI" colum
Hello folks,
I'm trying to deploy a Spark driver on Amazon EMR in yarn-cluster mode
expecting to be able to access the Spark UI from the :4040
address (default port). The problem here is that the Spark UI port is
always defined randomly at runtime, although I also tried to specify it in
the spark-
