Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/6394#discussion_r32284228
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala ---
@@ -242,4 +245,154 @@ class YarnAllocatorSuite extends SparkFunSuite with
Matchers with BeforeAndAfter
assert(pmemMsg.contains("2.1 MB of 2 GB physical memory used."))
}
+ test("allocate locality preferred containers with enough resource and no
matched existed " +
+ "containers") {
+ // 1. All the locations of current containers cannot satisfy the new
requirements
+ // 2. Current requested container number can fully satisfy the pending
tasks.
+
+ val handler = createAllocator(2)
+ handler.updateResourceRequests()
+ handler.handleAllocatedContainers(Array(
+ createContainer("host1"),
+ createContainer("host2")))
+
+ handler.requestTotalExecutorsWithPreferredLocalities(
+ 5, 15, Map("host3" -> 15, "host4" -> 15, "host5" -> 10))
+ handler.updateResourceRequests()
+ val nodesOfRequests = handler.containerRequests.toArray.map { request
=>
+ if (request.getNodes == null) {
+ null
+ } else {
+ request.getNodes.toArray(new Array[String](request.getNodes.size))
+ }
+ }
+
+ assert(nodesOfRequests === Array(
+ null, // container1 request
+ null, // container2 request
--- End diff --
Hi @squito , sorry for it, the first and second `null` is the node list of
`ContainerRequest` here:
```
val handler = createAllocator(2)
handler.updateResourceRequests()
handler.handleAllocatedContainers(Array(
createContainer("host1"),
createContainer("host2")))
```
Here at start, I created two `ContainerRequest` with preferred node list
`null`, but actually allocated on "host1" and "host2", so the preferred node
list of these two `ContainerRequest` are `null`, I'm not sure if I explained
clearly, I will add more comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
