uros-b commented on code in PR #57701:
URL: https://github.com/apache/spark/pull/57701#discussion_r3699754521
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/InMemoryColumnarQuerySuite.scala:
##########
@@ -172,6 +172,16 @@ class InMemoryColumnarQuerySuite extends
SharedSparkSession with AdaptiveSparkPl
assert(df.filter("f <= 10.0").count() == 9)
}
+ test("SPARK-58482: cached table pruning should retain NaN values") {
+ val df = Seq((1, 1.5d), (2, Double.NaN), (3, 0.0d)).toDF("id", "v").cache()
+ try {
+ df.count()
+ checkAnswer(df.filter($"v" === Double.NaN).select("id"), Row(2))
+ } finally {
+ df.unpersist(blocking = true)
+ }
+ }
+
Review Comment:
Float regression test missing: the new end-to-end test "SPARK-58482: cached
table pruning should retain NaN values" exercises the row-based path (default
DefaultCachedBatchSerializer) with a Double.NaN equality filter only; there is
no symmetric test for FloatType. The same incorrect-prune bug exists for float
columns, and since the test exercises DefaultCachedBatchSerializer, no existing
coverage closes this gap. Adding a float case (e.g. Seq((1, 1.5f), (2,
Float.NaN), (3, 0.0f)) with filter($"v" === Float.NaN)) would complete the
regression coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
