Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/11157#discussion_r69723776
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
---
@@ -356,4 +374,233 @@ private[mesos] trait MesosSchedulerUtils extends
Logging {
sc.conf.getTimeAsSeconds("spark.mesos.rejectOfferDurationForReachedMaxCores",
"120s")
}
+ /**
+ * Checks executor ports if they are within some range of the offered
list of ports ranges,
+ *
+ * @param sc the Spark Context
+ * @param ports the list of ports to check
+ * @return true if ports are within range false otherwise
+ */
+ protected def checkPorts(sc: SparkContext, ports: List[(Long, Long)]):
Boolean = {
+
+ def checkIfInRange(port: Long, ps: List[(Long, Long)]): Boolean = {
+ ps.exists(r => r._1 <= port & r._2 >= port)
+ }
+
+ val portsToCheck = ManagedPorts.getPortValues(sc.conf)
+ val nonZeroPorts = portsToCheck.filter(_ != 0)
+ val withinRange = nonZeroPorts.forall(p => checkIfInRange(p, ports))
+ // make sure we have enough ports to allocate per offer
+ ports.map(r => r._2 - r._1 + 1).sum >= portsToCheck.size && withinRange
+ }
+
+ /**
+ * Partitions port resources.
+ *
+ * @param conf the spark config
+ * @param ports the ports offered
+ * @return resources left, port resources to be used and the list of
assigned ports
+ */
+ def partitionPorts(
+ conf: SparkConf,
--- End diff --
If we do that then at some higher level like within the
MesosCoarseGrainedSchedulerBackend code
we need to call ManagedPorts.getPortValues(conf) to get the port list.
Although carrying the conf info there is unnecessary, in general i see this
call as low level so wanted to hide it within the logic which does the port
process stuff. Ideally in a different design of the spark core there should be
a module to query to get net ports configured, config or elsewhere it does not
matter it would have been hidden.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
