Bobby Wang created SPARK-46721:
----------------------------------
Summary: Some tests failed when cherry-picking
https://github.com/apache/spark/pull/43494 back to branch 3.5
Key: SPARK-46721
URL: https://issues.apache.org/jira/browse/SPARK-46721
Project: Spark
Issue Type: Test
Components: Tests
Affects Versions: 4.0.0
Reporter: Bobby Wang
Fix For: 4.0.0
When cherry-picking https://github.com/apache/spark/pull/43494 back to branch
3.5 https://github.com/apache/spark/pull/44690,
I ran into the issue that some tests for Scala 2.12 failed when comparing two
maps. It turned out that the function
[compareMaps](https://github.com/apache/spark/pull/43494/files#diff-f205431247dd9446f4ce941e5a4620af438c242b9bdff6e7faa7df0194db49acR129)
is not so robust for scala 2.12 and scala 2.13.
- scala 2.13
``` scala
Welcome to Scala 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.9).
Type in expressions for evaluation. Or try :help.
scala> def compareMaps(lhs: Map[String, Double], rhs: Map[String, Double],
| eps: Double = 0.00000001): Boolean = {
| lhs.size == rhs.size &&
| lhs.zip(rhs).forall { case ((lName, lAmount), (rName, rAmount)) =>
| lName == rName && (lAmount - rAmount).abs < eps
| }
| }
|
| import scala.collection.mutable.HashMap
| val resources = Map("gpu" -> Map("a" -> 1.0, "b" -> 2.0, "c" -> 3.0,
"d"-> 4.0))
| val mapped = resources.map { case (rName, addressAmounts) =>
| rName -> HashMap(addressAmounts.toSeq.sorted: _*)
| }
|
| compareMaps(resources("gpu"), mapped("gpu").toMap)
def compareMaps(lhs: Map[String,Double], rhs: Map[String,Double], eps: Double):
Boolean
import scala.collection.mutable.HashMap
val resources:
scala.collection.immutable.Map[String,scala.collection.immutable.Map[String,Double]]
= Map(gpu -> Map(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0))
val mapped:
scala.collection.immutable.Map[String,scala.collection.mutable.HashMap[String,Double]]
= Map(gpu -> HashMap(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0))
val res0: Boolean = true
```
- scala 2.12
``` scala
Welcome to Scala 2.12.14 (OpenJDK 64-Bit Server VM, Java 17.0.9).
Type in expressions for evaluation. Or try :help.
scala> def compareMaps(lhs: Map[String, Double], rhs: Map[String, Double],
| eps: Double = 0.00000001): Boolean = {
| lhs.size == rhs.size &&
| lhs.zip(rhs).forall { case ((lName, lAmount), (rName, rAmount)) =>
| lName == rName && (lAmount - rAmount).abs < eps
| }
| }
compareMaps: (lhs: Map[String,Double], rhs: Map[String,Double], eps:
Double)Boolean
scala> import scala.collection.mutable.HashMap
import scala.collection.mutable.HashMap
scala> val resources = Map("gpu" -> Map("a" -> 1.0, "b" -> 2.0, "c" -> 3.0,
"d"-> 4.0))
resources:
scala.collection.immutable.Map[String,scala.collection.immutable.Map[String,Double]]
= Map(gpu -> Map(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0))
scala> val mapped = resources.map { case (rName, addressAmounts) =>
| rName -> HashMap(addressAmounts.toSeq.sorted: _*)
| }
mapped:
scala.collection.immutable.Map[String,scala.collection.mutable.HashMap[String,Double]]
= Map(gpu -> Map(b -> 2.0, d -> 4.0, a -> 1.0, c -> 3.0))
scala> compareMaps(resources("gpu"), mapped("gpu").toMap)
res0: Boolean = false
```
The same code bug got different results for Scala 2.12 and Scala 2.13. This PR
tried to rework compareMaps to make tests pass for both scala 2.12 and scala
2.13
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
