rajatahujaatinmobi commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r443860777
##########
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict
with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same
box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
Correct me if I am wrong
As per the discussions that UI port is always get assigned in the range
based on spark.port.maxRetries. So we can always mention the starting port and
afterward, it will try to start service based in between (starting port,
starting port +spark.port.maxRetries). So we are including the range
implicitly.
Copying the comment
``other port types were spawned in the range (port_mentioned to
port_mentioned+spark.port.maxRetries) except the UI port which started on
random port.``
We have a Knox proxy that accesses UI via RM link. And Knox proxy has a
certain range of ports that it can access on the cluster which is
(18000-18159). Unless we start web UI on that range, we can not access the UI.
and we can achieve above the range if we set the following properties
spark.ui.port=18000
spark.port.maxRetries=159
Hence setting up a spark.ui.port as starting port is enough instead of
setting up a random port number in the range.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
