tgravescs commented on a change in pull request #27583: [SPARK-29149][YARN]
Update YARN cluster manager For Stage Level Scheduling
URL: https://github.com/apache/spark/pull/27583#discussion_r385729849
##########
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
##########
@@ -336,7 +338,7 @@ private[yarn] class YarnAllocator(
val resource = Resource.newInstance(totalMem, cores)
ResourceRequestHelper.setResourceRequests(customResources.toMap,
resource)
logDebug(s"Created resource capability: $resource")
- rpIdToYarnResource(rp.id) = resource
+ rpIdToYarnResource.putIfAbsent(rp.id, resource)
Review comment:
no not at the moment anyway, this function is synchronized and no where else
adds it so only one can run at a time. I put in putIfAbsent but it doesn't
really matter. ResourceProfile ids are unique and ResourceProfiles are
immutable. Even if this code ran in multiple threads at the same time the
result should be exactly the same so we would put the same thing in twice and
it wouldn't matter which one got inserted first.
Strictly speaking that doesn't need to be a concurrent hashmap due to
locking of the calling functions but to be more strict on it and ot help with
future changes I made it one.
If you think its more clear one way or another let me know and I can modify.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
