tgravescs commented on code in PR #43494:
URL: https://github.com/apache/spark/pull/43494#discussion_r1381839965
##########
core/src/main/scala/org/apache/spark/resource/ResourceAllocator.scala:
##########
@@ -29,59 +65,54 @@ private[spark] trait ResourceAllocator {
protected def resourceName: String
protected def resourceAddresses: Seq[String]
- protected def slotsPerAddress: Int
/**
- * Map from an address to its availability, a value > 0 means the address is
available,
- * while value of 0 means the address is fully assigned.
- *
- * For task resources ([[org.apache.spark.scheduler.ExecutorResourceInfo]]),
this value
- * can be a multiple, such that each address can be allocated up to
[[slotsPerAddress]]
- * times.
+ * Map from an address to its availability default to RESOURCE_TOTAL_AMOUNT,
a value > 0 means
+ * the address is available, while value of 0 means the address is fully
assigned.
*/
private lazy val addressAvailabilityMap = {
- mutable.HashMap(resourceAddresses.map(_ -> slotsPerAddress): _*)
+ mutable.HashMap(resourceAddresses.map(address => address ->
RESOURCE_TOTAL_AMOUNT): _*)
}
/**
- * Sequence of currently available resource addresses.
- *
- * With [[slotsPerAddress]] greater than 1, [[availableAddrs]] can contain
duplicate addresses
- * e.g. with [[slotsPerAddress]] == 2, availableAddrs for addresses 0 and 1
can look like
- * Seq("0", "0", "1"), where address 0 has two assignments available, and 1
has one.
+ * Get the resources and its amounts.
+ * @return the resources amounts
+ */
+ def resourcesAmounts: Map[String, Double] = addressAvailabilityMap.map {
Review Comment:
I just saying if we are storing things in long format (10000000000000000L *
number of resources requested). Be sure that isn't going to overflow. Since I
said its a generic comment all over, just make sure it isn't going to happen.
If the values we store are always < (10000000000000000L * 1) its not a problem.
Overflow means you have a value larger then 2^64, which then isn't positive
anymore
```
scala> 10000000000000000L * 2500l
res7: Long = 6553255926290448384
scala> 10000000000000000L * 25000l
res8: Long = -8254417031933722624
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
