rajatahujaatinmobi commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444552932
##########
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict
with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same
box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
I blocked a set of Port range from 18000 to 18050 and configure web or
to start at 18008 but it eventually Started at 18051
**Properties Set: spark.ui.port=18018 spark.port.maxRetries=60**
Successful Case:
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18035. Attempting port 18036.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18036. Attempting port 18037.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18037. Attempting port 18038.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18038. Attempting port 18039.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18039. Attempting port 18040.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18040. Attempting port 18041.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18041. Attempting port 18042.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18042. Attempting port 18043.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18043. Attempting port 18044.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18044. Attempting port 18045.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18045. Attempting port 18046.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18046. Attempting port 18047.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18047. Attempting port 18048.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18048. Attempting port 18049.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18049. Attempting port 18050.
20/06/23 21:55:30 WARN util.Utils: Service 'SparkUI' could not bind on port
18050. Attempting port 18051.
20/06/23 21:55:30 INFO server.AbstractConnector: Started
ServerConnector@7ba63fe5{HTTP/1.1,[http/1.1]}{0.0.0.0:18051}
**20/06/23 21:55:30 INFO util.Utils: Successfully started service 'SparkUI'
on port 18051.
And Job succeeded**
Failure Case
**Properties Set: spark.ui.port=18018 spark.port.maxRetries=6**
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port
18018. Attempting port 18019.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port
18019. Attempting port 18020.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port
18020. Attempting port 18021.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port
18021. Attempting port 18022.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port
18022. Attempting port 18023.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port
18023. Attempting port 18024.
**20/06/23 22:16:15 ERROR ui.SparkUI: Failed to bind SparkUI
java.net.BindException: Address already in use: Service 'SparkUI' failed
after 6 retries (starting from 18018)! Consider explicitly setting the
appropriate port for the service 'SparkUI' (for example spark.ui.port for
SparkUI) to an available port or increasing spark.port.maxRetries.
at sun.nio.ch.Net.bind0(Native Method)
Job Failed ...**
@tgravescs @Ngone51
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
