Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3314#discussion_r20495676
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1652,25 +1652,28 @@ private[spark] object Utils extends Logging {
* Attempt to start a service on the given port, or fail after a number
of attempts.
* Each subsequent attempt uses 1 + the port used in the previous
attempt (unless the port is 0).
*
- * @param startPort The initial port to start the service on.
+ * @param port The initial port to start the service on.
* @param maxRetries Maximum number of retries to attempt.
* A value of 3 means attempting ports n, n+1, n+2,
and n+3, for example.
* @param startService Function to start service on a given port.
* This is expected to throw java.net.BindException
on port collision.
*/
def startServiceOnPort[T](
- startPort: Int,
+ port: Int,
startService: Int => (T, Int),
+ conf: SparkConf = new SparkConf(),
serviceName: String = "",
maxRetries: Int = portMaxRetries): (T, Int) = {
val serviceString = if (serviceName.isEmpty) "" else s" '$serviceName'"
+ val startPort = conf.getInt("spark.port.min", 0)
+ val endPort = conf.getInt("spark.port.max", 65536)
for (offset <- 0 to maxRetries) {
- // Do not increment port if startPort is 0, which is treated as a
special port
- val tryPort = if (startPort == 0) {
- startPort
+ // Do not increment port if port is 0, which is treated as a special
port
+ val tryPort = if (port == 0) {
+ (startPort + Math.random() * (endPort - startPort)).toInt
--- End diff --
0 is passed through to the start function unchanged.
This looks better, but I think we have to be careful with the semantics.
"spark.port.max" sounds like a maximum allowable port, inclusive, but the way
it's used here it's exclusive. That is, if I want to use ports 9000-9005, I
would have to set min=9000 and max=9006, which isn't intuitive. You may need
`endPort+1` and document this.
This doesn't work if `port` isn't in the range, right? that seems like a
problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
