wbo4958 commented on code in PR #44690:
URL: https://github.com/apache/spark/pull/44690#discussion_r1480798101
##########
core/src/main/scala/org/apache/spark/resource/ResourceAllocator.scala:
##########
@@ -20,6 +20,49 @@ package org.apache.spark.resource
import scala.collection.mutable
import org.apache.spark.SparkException
+import org.apache.spark.resource.ResourceAmountUtils.ONE_ENTIRE_RESOURCE
+
+private[spark] object ResourceAmountUtils {
+ /**
+ * Using "double" to do the resource calculation may encounter a problem of
precision loss. Eg
+ *
+ * scala> val taskAmount = 1.0 / 9
+ * taskAmount: Double = 0.1111111111111111
+ *
+ * scala> var total = 1.0
+ * total: Double = 1.0
+ *
+ * scala> for (i <- 1 to 9 ) {
+ * | if (total >= taskAmount) {
+ * | total -= taskAmount
+ * | println(s"assign $taskAmount for task $i, total left: $total")
+ * | } else {
+ * | println(s"ERROR Can't assign $taskAmount for task $i, total
left: $total")
+ * | }
+ * | }
+ * assign 0.1111111111111111 for task 1, total left: 0.8888888888888888
+ * assign 0.1111111111111111 for task 2, total left: 0.7777777777777777
+ * assign 0.1111111111111111 for task 3, total left: 0.6666666666666665
+ * assign 0.1111111111111111 for task 4, total left: 0.5555555555555554
+ * assign 0.1111111111111111 for task 5, total left: 0.44444444444444425
+ * assign 0.1111111111111111 for task 6, total left: 0.33333333333333315
+ * assign 0.1111111111111111 for task 7, total left: 0.22222222222222204
+ * assign 0.1111111111111111 for task 8, total left: 0.11111111111111094
+ * ERROR Can't assign 0.1111111111111111 for task 9, total left:
0.11111111111111094
+ *
+ * So we multiply ONE_ENTIRE_RESOURCE to convert the double to long to avoid
this limitation.
+ * Double can display up to 16 decimal places, so we set the factor to
+ * 10, 000, 000, 000, 000, 000L.
+ */
+ final val ONE_ENTIRE_RESOURCE: Long = 10000000000000000L
Review Comment:
Hi @srowen,
The **using Long** way is quite the same with the BigDecimal. The default
scale of BigDecimal is 16, so this PR chooses (ONE_ENTIRE_RESOURCE =
1E16.toLong)
``` scala
scala> val ONE_ENTIRE_RESOURCE: Long = 1E16.toLong
| val taskAmount = 0.3333333333333334
|
| val usingLong = (taskAmount * ONE_ENTIRE_RESOURCE).toLong
|
| val bigDec = BigDecimal(taskAmount).toDouble
val ONE_ENTIRE_RESOURCE: Long = 10000000000000000
val taskAmount: Double = 0.3333333333333334
val usingLong: Long = 3333333333333334
val bigDec: Double = 0.3333333333333334
```
So if we need to ensure the input is small enough (<1/n) and we can set the
scale to be like 14 for BigDecimal, and similarly, to keep align with
BigDecimal, we can set `ONE_ENTIRE_RESOURCE = 1E14.toLong`
``` scala
scala> import scala.math.BigDecimal.RoundingMode
|
| val ONE_ENTIRE_RESOURCE: Long = 1E14.toLong
| val taskAmount = 0.3333333333333334
|
| val usingLong = (taskAmount * ONE_ENTIRE_RESOURCE).toLong
|
| val bigDec = BigDecimal(taskAmount).setScale(14,
RoundingMode.DOWN).toDouble
import scala.math.BigDecimal.RoundingMode
val ONE_ENTIRE_RESOURCE: Long = 100000000000000
val taskAmount: Double = 0.3333333333333334
val usingLong: Long = 33333333333333
val bigDec: Double = 0.33333333333333
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
