wbo4958 commented on code in PR #44690:
URL: https://github.com/apache/spark/pull/44690#discussion_r1477679566
##########
core/src/main/scala/org/apache/spark/resource/ResourceAllocator.scala:
##########
@@ -20,6 +20,49 @@ package org.apache.spark.resource
import scala.collection.mutable
import org.apache.spark.SparkException
+import org.apache.spark.resource.ResourceAmountUtils.ONE_ENTIRE_RESOURCE
+
+private[spark] object ResourceAmountUtils {
+ /**
+ * Using "double" to do the resource calculation may encounter a problem of
precision loss. Eg
+ *
+ * scala> val taskAmount = 1.0 / 9
+ * taskAmount: Double = 0.1111111111111111
+ *
+ * scala> var total = 1.0
+ * total: Double = 1.0
+ *
+ * scala> for (i <- 1 to 9 ) {
+ * | if (total >= taskAmount) {
+ * | total -= taskAmount
+ * | println(s"assign $taskAmount for task $i, total left: $total")
+ * | } else {
+ * | println(s"ERROR Can't assign $taskAmount for task $i, total
left: $total")
+ * | }
+ * | }
+ * assign 0.1111111111111111 for task 1, total left: 0.8888888888888888
+ * assign 0.1111111111111111 for task 2, total left: 0.7777777777777777
+ * assign 0.1111111111111111 for task 3, total left: 0.6666666666666665
+ * assign 0.1111111111111111 for task 4, total left: 0.5555555555555554
+ * assign 0.1111111111111111 for task 5, total left: 0.44444444444444425
+ * assign 0.1111111111111111 for task 6, total left: 0.33333333333333315
+ * assign 0.1111111111111111 for task 7, total left: 0.22222222222222204
+ * assign 0.1111111111111111 for task 8, total left: 0.11111111111111094
+ * ERROR Can't assign 0.1111111111111111 for task 9, total left:
0.11111111111111094
+ *
+ * So we multiply ONE_ENTIRE_RESOURCE to convert the double to long to avoid
this limitation.
+ * Double can display up to 16 decimal places, so we set the factor to
+ * 10, 000, 000, 000, 000, 000L.
+ */
+ final val ONE_ENTIRE_RESOURCE: Long = 10000000000000000L
Review Comment:
Hi @srowen, Yeah, as long as we use double to do the calculation, we can't
avoid the precision issue. We can also check the absolute value of the
difference between two numbers, But it seems it doesn't make any difference?
``` scala
scala> val taskAmount = 0.05
taskAmount: Double = 0.05
scala> var total = 1.0
total: Double = 1.0
scala> for (i <- 1 to 20 ) {
| if (total >= taskAmount) {
| total -= taskAmount
| println(s"assign $taskAmount for task $i, total left:
$total")
| } else {
| println(s"ERROR Can't assign $taskAmount for task $i,
total left: $total")
| }
| }
assign 0.05 for task 1, total left: 0.95
assign 0.05 for task 2, total left: 0.8999999999999999
assign 0.05 for task 3, total left: 0.8499999999999999
assign 0.05 for task 4, total left: 0.7999999999999998
assign 0.05 for task 5, total left: 0.7499999999999998
assign 0.05 for task 6, total left: 0.6999999999999997
assign 0.05 for task 7, total left: 0.6499999999999997
assign 0.05 for task 8, total left: 0.5999999999999996
assign 0.05 for task 9, total left: 0.5499999999999996
assign 0.05 for task 10, total left: 0.4999999999999996
assign 0.05 for task 11, total left: 0.4499999999999996
assign 0.05 for task 12, total left: 0.39999999999999963
assign 0.05 for task 13, total left: 0.34999999999999964
assign 0.05 for task 14, total left: 0.29999999999999966
assign 0.05 for task 15, total left: 0.24999999999999967
assign 0.05 for task 16, total left: 0.19999999999999968
assign 0.05 for task 17, total left: 0.1499999999999997
assign 0.05 for task 18, total left: 0.09999999999999969
assign 0.05 for task 19, total left: 0.049999999999999684
ERROR Can't assign 0.05 for task 20, total left: 0.049999999999999684
```
Take the above code as an example, taskAmount=0.05, which I think is a very
normal value, but we can't assign the resources to a total of 20 tasks due to
precision issues.
If we convert double to Long, it can work
``` scala
scala> val ONE_ENTIRE_RESOURCE: Long = 10000000000000000L
ONE_ENTIRE_RESOURCE: Long = 10000000000000000
scala>
scala> val taskAmount = 0.05
taskAmount: Double = 0.05
scala> var total = 1.0 * ONE_ENTIRE_RESOURCE
total: Double = 1.0E16
scala> for (i <- 1 to 20 ) {
| if (total >= taskAmount * ONE_ENTIRE_RESOURCE) {
| total -= taskAmount * ONE_ENTIRE_RESOURCE
| println(s"assign $taskAmount for task $i, total left:
$total")
| } else {
| println(s"ERROR Can't assign $taskAmount for task $i,
total left: $total")
| }
| }
assign 0.05 for task 1, total left: 9.5E15
assign 0.05 for task 2, total left: 9.0E15
assign 0.05 for task 3, total left: 8.5E15
assign 0.05 for task 4, total left: 8.0E15
assign 0.05 for task 5, total left: 7.5E15
assign 0.05 for task 6, total left: 7.0E15
assign 0.05 for task 7, total left: 6.5E15
assign 0.05 for task 8, total left: 6.0E15
assign 0.05 for task 9, total left: 5.5E15
assign 0.05 for task 10, total left: 5.0E15
assign 0.05 for task 11, total left: 4.5E15
assign 0.05 for task 12, total left: 4.0E15
assign 0.05 for task 13, total left: 3.5E15
assign 0.05 for task 14, total left: 3.0E15
assign 0.05 for task 15, total left: 2.5E15
assign 0.05 for task 16, total left: 2.0E15
assign 0.05 for task 17, total left: 1.5E15
assign 0.05 for task 18, total left: 1.0E15
assign 0.05 for task 19, total left: 5.0E14
assign 0.05 for task 20, total left: 0.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
