Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/3841#issuecomment-68418574
I set `spark.port.maxRetries` to 1 in spark-defaults.conf and ran 3 SparkPi
example using yarn-client mode in one node. Logs in 3 drivers' side were:
>14/12/31 10:09:26 INFO Utils: Successfully started service 'SparkUI' on
port 23000.
14/12/31 10:09:26 INFO SparkUI: Started SparkUI at
http://dc1-rack1-host2:23000
>14/12/31 10:09:33 WARN Utils: Service 'SparkUI' could not bind on port
23000. Attempting port 23001.
14/12/31 10:09:34 INFO Utils: Successfully started service 'SparkUI' on
port 23001.
14/12/31 10:09:34 INFO SparkUI: Started SparkUI at
http://dc1-rack1-host2:23001
>14/12/31 10:09:36 WARN Utils: Service 'SparkUI' could not bind on port
23000. Attempting port 23001.
14/12/31 10:09:36 WARN Utils: Service 'SparkUI' could not bind on port
23001. Attempting port 23002.
14/12/31 10:09:36 INFO Utils: Successfully started service 'SparkUI' on
port 23002.
14/12/31 10:09:36 INFO SparkUI: Started SparkUI at
http://dc1-rack1-host2:23002
We could see the third application retried 3 times before it giving up. So
the spark.port.maxRetries didn't work here.
In another node, I set this config's value to 20 and it could only launch
16 applications.
>14/12/31 17:43:55 WARN Utils: Service 'SparkUI' could not bind on port
23015.
14/12/31 17:43:55 ERROR SparkUI: Failed to bind SparkUI
java.net.BindException: Address already in use: Service 'SparkUI' failed
after 16 retries!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
