Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/6394#discussion_r32243839
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala ---
@@ -242,4 +245,154 @@ class YarnAllocatorSuite extends SparkFunSuite with
Matchers with BeforeAndAfter
assert(pmemMsg.contains("2.1 MB of 2 GB physical memory used."))
}
+ test("allocate locality preferred containers with enough resource and no
matched existed " +
+ "containers") {
+ // 1. All the locations of current containers cannot satisfy the new
requirements
+ // 2. Current requested container number can fully satisfy the pending
tasks.
+
+ val handler = createAllocator(2)
+ handler.updateResourceRequests()
+ handler.handleAllocatedContainers(Array(
+ createContainer("host1"),
+ createContainer("host2")))
+
+ handler.requestTotalExecutorsWithPreferredLocalities(
+ 5, 15, Map("host3" -> 15, "host4" -> 15, "host5" -> 10))
+ handler.updateResourceRequests()
+ val nodesOfRequests = handler.containerRequests.toArray.map { request
=>
+ if (request.getNodes == null) {
+ null
+ } else {
+ request.getNodes.toArray(new Array[String](request.getNodes.size))
+ }
+ }
+
+ assert(nodesOfRequests === Array(
+ null, // container1 request
+ null, // container2 request
--- End diff --
I am a little confused by the comments -- I thought that the `nodes` in the
request will be null when you are requesting new containers, but there is
absolutely no preference for which node the container ends up on? The comment
makes me think this is a request which is somehow related to the existing
container1 on host1.
(but again I'm kinda new to this code ... so this might just show my
general level of confusion ...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
