abellina commented on a change in pull request #26078: [SPARK-29151][CORE]
Support fractional resources for task resource scheduling
URL: https://github.com/apache/spark/pull/26078#discussion_r341745050
##########
File path:
core/src/main/scala/org/apache/spark/resource/ResourceAllocator.scala
##########
@@ -86,8 +103,8 @@ trait ResourceAllocator {
s"address $address doesn't exist.")
}
val isAvailable = addressAvailabilityMap(address)
- if (!isAvailable) {
- addressAvailabilityMap(address) = true
+ if (isAvailable < slotsPerAddress) {
+ addressAvailabilityMap(address) = addressAvailabilityMap(address) + 1
Review comment:
`ResourceAllocator` _should_ only used from a single thread (it is
documented that way).
For its public interface, here is the access pattern:
- `CoarseGrainedSchedulerBackend` and `Master` event loop are calling into
`acquire` and `release`. They are also using `availableAddrs`, and are calling
`toBuffer` on it before handing off that state. This is a single thread pattern.
- A class extending `ResourceAllocator` (`WorkerResourceInfo`), has some
potential issues: The `WorkerInfo` class is calling into `availableAddrs` and
`assignedAddrs` but those calls appear to be coming from the UI (looking at the
`resourcesInfo*` functions), e.g. `JsonProtocol` and `MasterPage` call this.
There are other calls to the `resourceInfo*` functions, but those are from the
event loop. Is this what you were worried about @jiangxb1987?
Note this issue is not directly related to this PR, as this PR didn't change
that access pattern, or the datastructure. That said, I do believe that a
request from the UI could result in at least an exception (if not worse). Let
me know and I can handle here, or open another PR that would address it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
