Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8100#discussion_r43577930
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala ---
@@ -534,6 +549,40 @@ private[yarn] class YarnAllocator(
private[yarn] def getNumUnexpectedContainerRelease =
numUnexpectedContainerRelease
+ /**
+ * Split the pending container requests into 3 groups based on current
localities of pending
+ * tasks.
+ * @param hostToLocalTaskCount a map of preferred hostname to possible
task counts to be used as
+ * container placement hint.
+ * @param pendingAllocations A sequence of pending allocation container
request.
+ * @return A tuple of 3 sequences, first is a sequence of locality
matched container
+ * requests, second is a sequence of locality unmatched
container requests, and third is a
+ * sequence of locality free container requests.
+ */
+ private def splitPendingAllocationsByLocality(
+ hostToLocalTaskCount: Map[String, Int],
+ pendingAllocations: Seq[ContainerRequest]
+ ): (Seq[ContainerRequest], Seq[ContainerRequest],
Seq[ContainerRequest]) = {
+ val localityMatched = ArrayBuffer[ContainerRequest]()
+ val localityUnMatched = ArrayBuffer[ContainerRequest]()
+ val localityFree = ArrayBuffer[ContainerRequest]()
+
+ val preferredHosts = hostToLocalTaskCount.keySet
+ pendingAllocations.foreach { cr =>
+ val nodes = cr.getNodes
+ if (nodes == null) {
+ localityFree += cr
+ } else {
--- End diff --
minor: cleaner if you use `} else if (...) {`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
