This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 947e271402f [SPARK-39612][SQL][TESTS] DataFrame.exceptAll followed by
count should work
947e271402f is described below
commit 947e271402f749f6f58b79fecd59279eaf86db57
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Tue Jul 5 20:44:43 2022 +0900
[SPARK-39612][SQL][TESTS] DataFrame.exceptAll followed by count should work
### What changes were proposed in this pull request?
This PR adds a test case broken by
https://github.com/apache/spark/commit/4b9343593eca780ca30ffda45244a71413577884
which was reverted in
https://github.com/apache/spark/commit/161c596cafea9c235b5c918d8999c085401d73a9.
### Why are the changes needed?
To prevent a regression in the future. This was a regression in Apache
Spark 3.3 that used to work in Apache Spark 3.2.
### Does this PR introduce _any_ user-facing change?
Yes, it makes `DataFrame.exceptAll` followed by `count` working.
### How was this patch tested?
The unit test was added.
Closes #37084 from HyukjinKwon/SPARK-39612.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index 4daa0a1b3b6..41593c701a7 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
@@ -3239,6 +3239,11 @@ class DataFrameSuite extends QueryTest
}
}
}
+
+ test("SPARK-39612: exceptAll with following count should work") {
+ val d1 = Seq("a").toDF
+ assert(d1.exceptAll(d1).count() === 0)
+ }
}
case class GroupByKey(a: Int, b: Int)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
