Github user WangTaoTheTonic commented on a diff in the pull request:
https://github.com/apache/spark/pull/5722#discussion_r29572401
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1959,30 +1959,41 @@ private[spark] object Utils extends Logging {
* Attempt to start a service on the given port, or fail after a number
of attempts.
* Each subsequent attempt uses 1 + the port used in the previous
attempt (unless the port is 0).
*
- * @param startPort The initial port to start the service on.
+ * @param port The minimum and maximum port to start the service on,
separated by colon.
+ * If just set one number, take it as the minimum.
* @param startService Function to start service on a given port.
* This is expected to throw java.net.BindException
on port collision.
* @param conf A SparkConf used to get the maximum number of retries
when binding to a port.
* @param serviceName Name of the service.
*/
def startServiceOnPort[T](
- startPort: Int,
+ port: String,
startService: Int => (T, Int),
conf: SparkConf,
serviceName: String = ""): (T, Int) = {
-
- require(startPort == 0 || (1024 <= startPort && startPort < 65536),
- "startPort should be between 1024 and 65535 (inclusive), or 0 for a
random free port.")
+ val ports = port.split(":", 2)
+ val (minPort, maxPort) = if (ports.length == 2) {
+ (ports(0).toInt, ports(1).toInt)
+ } else {
+ (ports(0).toInt, 65535)
+ }
+ require(minPort == 0 || (1024 <= minPort && minPort <= 65535),
+ s"Minimum port ${minPort} should be between 1024 and 65535
(inclusive)," +
+ " or 0 for a random free port.")
+ require(maxPort == 0 || (1024 <= maxPort && maxPort <= 65535),
+ s"Maximum port ${maxPort} should be between 1024 and 65535
(inclusive)," +
+ " or 0 for a random free port.")
+ require(minPort <= maxPort, s"Minimum ${minPort} port should not be" +
+ s" less than the maximum ${maxPort}.")
val serviceString = if (serviceName.isEmpty) "" else s" '$serviceName'"
val maxRetries = portMaxRetries(conf)
--- End diff --
Ehh..I think I couldn't understand you totally. what did you mean by `that
will potentially throw an exception the user doesn't expect`? If `maxPort -
minPort > maxRetries` then port will retry between `minPort` and `minPort +
maxRetires - 1` and throw an exception if all ports are not available in that
range.
I think `minPort` and `maxPort` is the range that port will limited to (if
successfully), and `maxRetries` is the max retry times to try to bind.
But take another thought, these three configs are proposed for same reason
(limit port range https://github.com/apache/spark/pull/1777), and they maybe
overlap but `maxRetires` is kinda global setting.
Anyway, I will `set maxPort to minPort + maxRetries if it's not defined`
per your comments as `65535` is sort of too big for the default value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
