[ https://issues.apache.org/jira/browse/SPARK-46721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot reassigned SPARK-46721: -------------------------------------- Assignee: (was: Apache Spark) > Some tests failed when cherry-picking > https://github.com/apache/spark/pull/43494 back to branch 3.5 > ---------------------------------------------------------------------------------------------------- > > Key: SPARK-46721 > URL: https://issues.apache.org/jira/browse/SPARK-46721 > Project: Spark > Issue Type: Test > Components: Tests > Affects Versions: 4.0.0 > Reporter: Bobby Wang > Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > > When cherry-picking https://github.com/apache/spark/pull/43494 back to branch > 3.5 https://github.com/apache/spark/pull/44690, > I ran into the issue that some tests for Scala 2.12 failed when comparing two > maps. It turned out that the function > [compareMaps](https://github.com/apache/spark/pull/43494/files#diff-f205431247dd9446f4ce941e5a4620af438c242b9bdff6e7faa7df0194db49acR129) > is not so robust for scala 2.12 and scala 2.13. > - scala 2.13 > ``` scala > Welcome to Scala 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.9). > Type in expressions for evaluation. Or try :help. > scala> def compareMaps(lhs: Map[String, Double], rhs: Map[String, Double], > | eps: Double = 0.00000001): Boolean = { > | lhs.size == rhs.size && > | lhs.zip(rhs).forall { case ((lName, lAmount), (rName, rAmount)) > => > | lName == rName && (lAmount - rAmount).abs < eps > | } > | } > | > | import scala.collection.mutable.HashMap > | val resources = Map("gpu" -> Map("a" -> 1.0, "b" -> 2.0, "c" -> 3.0, > "d"-> 4.0)) > | val mapped = resources.map { case (rName, addressAmounts) => > | rName -> HashMap(addressAmounts.toSeq.sorted: _*) > | } > | > | compareMaps(resources("gpu"), mapped("gpu").toMap) > def compareMaps(lhs: Map[String,Double], rhs: Map[String,Double], eps: > Double): Boolean > import scala.collection.mutable.HashMap > val resources: > scala.collection.immutable.Map[String,scala.collection.immutable.Map[String,Double]] > = Map(gpu -> Map(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0)) > val mapped: > scala.collection.immutable.Map[String,scala.collection.mutable.HashMap[String,Double]] > = Map(gpu -> HashMap(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0)) > val res0: Boolean = true > ``` > - scala 2.12 > ``` scala > Welcome to Scala 2.12.14 (OpenJDK 64-Bit Server VM, Java 17.0.9). > Type in expressions for evaluation. Or try :help. > scala> def compareMaps(lhs: Map[String, Double], rhs: Map[String, Double], > | eps: Double = 0.00000001): Boolean = { > | lhs.size == rhs.size && > | lhs.zip(rhs).forall { case ((lName, lAmount), (rName, rAmount)) > => > | lName == rName && (lAmount - rAmount).abs < eps > | } > | } > compareMaps: (lhs: Map[String,Double], rhs: Map[String,Double], eps: > Double)Boolean > scala> import scala.collection.mutable.HashMap > import scala.collection.mutable.HashMap > scala> val resources = Map("gpu" -> Map("a" -> 1.0, "b" -> 2.0, "c" -> 3.0, > "d"-> 4.0)) > resources: > scala.collection.immutable.Map[String,scala.collection.immutable.Map[String,Double]] > = Map(gpu -> Map(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0)) > scala> val mapped = resources.map { case (rName, addressAmounts) => > | rName -> HashMap(addressAmounts.toSeq.sorted: _*) > | } > mapped: > scala.collection.immutable.Map[String,scala.collection.mutable.HashMap[String,Double]] > = Map(gpu -> Map(b -> 2.0, d -> 4.0, a -> 1.0, c -> 3.0)) > scala> compareMaps(resources("gpu"), mapped("gpu").toMap) > res0: Boolean = false > ``` > The same code bug got different results for Scala 2.12 and Scala 2.13. This > PR tried to rework compareMaps to make tests pass for both scala 2.12 and > scala 2.13 -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org