rajatahujaatinmobi commented on a change in pull request #28880:
URL: https://github.com/apache/spark/pull/28880#discussion_r444552932
##########
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
##########
@@ -211,9 +211,11 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
- // Set the web ui port to be ephemeral for yarn so we don't conflict
with
- // other spark processes running on the same box
- System.setProperty(UI_PORT.key, "0")
+ // Set the web ui port to be ephemeral for yarn if not set explicitly
+ // so we don't conflict with other spark processes running on the same
box
+ if (System.getProperty(UI_PORT.key) != null) {
+ System.setProperty(UI_PORT.key, "0")
+ }
Review comment:
I blocked a set of Port range from 18000 to 18050 and configure web or
to start at 18008 but it eventually Started at 18051
**Properties Set: spark.ui.port=18018 spark.port.maxRetries=60**
Successful Case:
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18018. Attempting port 18019.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18019. Attempting port 18020.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18020. Attempting port 18021.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18021. Attempting port 18022.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18022. Attempting port 18023.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18023. Attempting port 18024.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18024. Attempting port 18025.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18025. Attempting port 18026.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18026. Attempting port 18027.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18027. Attempting port 18028.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18028. Attempting port 18029.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18029. Attempting port 18030.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18030. Attempting port 18031.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18031. Attempting port 18032.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18032. Attempting port 18033.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18033. Attempting port 18034.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18034. Attempting port 18035.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18035. Attempting port 18036.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18036. Attempting port 18037.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18037. Attempting port 18038.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18038. Attempting port 18039.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18039. Attempting port 18040.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18040. Attempting port 18041.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18041. Attempting port 18042.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18042. Attempting port 18043.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18043. Attempting port 18044.
20/06/23 22:18:43 WARN util.Utils: Service 'SparkUI' could not bind on port
18044. Attempting port 18045.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port
18045. Attempting port 18046.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port
18046. Attempting port 18047.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port
18047. Attempting port 18048.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port
18048. Attempting port 18049.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port
18049. Attempting port 18050.
20/06/23 22:18:44 WARN util.Utils: Service 'SparkUI' could not bind on port
18050. Attempting port 18051.
20/06/23 22:18:44 INFO server.AbstractConnector: Started
ServerConnector@6f330eb9{HTTP/1.1,[http/1.1]}{0.0.0.0:18051}
20/06/23 22:18:44 INFO util.Utils: Successfully started service 'SparkUI' on
port 18051.
And Job succeeded**
Failure Case
**Properties Set: spark.ui.port=18018 spark.port.maxRetries=6**
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port
18018. Attempting port 18019.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port
18019. Attempting port 18020.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port
18020. Attempting port 18021.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port
18021. Attempting port 18022.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port
18022. Attempting port 18023.
20/06/23 22:16:15 WARN util.Utils: Service 'SparkUI' could not bind on port
18023. Attempting port 18024.
**20/06/23 22:16:15 ERROR ui.SparkUI: Failed to bind SparkUI
java.net.BindException: Address already in use: Service 'SparkUI' failed
after 6 retries (starting from 18018)! Consider explicitly setting the
appropriate port for the service 'SparkUI' (for example spark.ui.port for
SparkUI) to an available port or increasing spark.port.maxRetries.
at sun.nio.ch.Net.bind0(Native Method)
Job Failed ...**
@tgravescs @Ngone51
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
