This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 37d19b9ee0e4 [SPARK-46237][SQL][TESTS] Make `HiveDDLSuite`
independently testable
37d19b9ee0e4 is described below
commit 37d19b9ee0e4e100e37358e71d771a2e42d01d88
Author: yangjie01 <yangji...@baidu.com>
AuthorDate: Mon Dec 4 08:52:23 2023 -0800
[SPARK-46237][SQL][TESTS] Make `HiveDDLSuite` independently testable
### What changes were proposed in this pull request?
When I test `HiveDDLSuite` with
```
build/sbt "hive/testOnly org.apache.spark.sql.hive.execution.HiveDDLSuite"
-Phive
```
This test throws an error:
```
[info] - SPARK-34261: Avoid side effect if create exists temporary function
*** FAILED *** (4 milliseconds)
[info] java.util.NoSuchElementException: key not found: default
[info] at scala.collection.MapOps.default(Map.scala:274)
[info] at scala.collection.MapOps.default$(Map.scala:273)
[info] at scala.collection.AbstractMap.default(Map.scala:405)
[info] at scala.collection.MapOps.apply(Map.scala:176)
[info] at scala.collection.MapOps.apply$(Map.scala:175)
[info] at scala.collection.AbstractMap.apply(Map.scala:405)
[info] at
org.apache.spark.sql.hive.execution.HiveDDLSuite.$anonfun$new$445(HiveDDLSuite.scala:3275)
[info] at
org.apache.spark.sql.test.SQLTestUtilsBase.withUserDefinedFunction(SQLTestUtils.scala:256)
[info] at
org.apache.spark.sql.test.SQLTestUtilsBase.withUserDefinedFunction$(SQLTestUtils.scala:254)
[info] at
org.apache.spark.sql.execution.command.DDLSuite.withUserDefinedFunction(DDLSuite.scala:326)
[info] at
org.apache.spark.sql.hive.execution.HiveDDLSuite.$anonfun$new$444(HiveDDLSuite.scala:3267)
```
I manually printed the content of `spark.sparkContext.addedJars`, which is
an empty `Map`.
However, when I execute
```
build/sbt "hive/testOnly org.apache.spark.sql.hive.execution.SQLQuerySuite
org.apache.spark.sql.hive.execution.HiveDDLSuite" -Phive
```
All tests pass, and the content of `spark.sparkContext.addedJars` is
```
Map(default -> Map(spark://localhost:54875/jars/SPARK-21101-1.0.jar ->
1701676986594, spark://localhost:54875/jars/hive-contrib-2.3.9.jar ->
1701676944590, spark://localhost:54875/jars/TestUDTF.jar -> 1701676921340))
```
The reason why this failure is not reproduced in the GitHub Action test is
because `SQLQuerySuite` is indeed executed before `HiveDDLSuite`.
So in the current PR, I change to use
`.get("default").foreach(_.remove(k))` that the remove operation is only
performed when `.get("default")` is not `None`.
### Why are the changes needed?
Make `HiveDDLSuite` independently testable.
### Does this PR introduce _any_ user-facing change?
No, just for test
### How was this patch tested?
- Pass Github Actions
- Manual check `HiveDDLSuite` with this pr and all test passed
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #44153 from LuciferYang/HiveDDLSuite.
Authored-by: yangjie01 <yangji...@baidu.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.../test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
index c3a528da382a..2f5d1fcbb540 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
@@ -3270,7 +3270,7 @@ class HiveDDLSuite
val jarName = "TestUDTF.jar"
val jar =
spark.asInstanceOf[TestHiveSparkSession].getHiveFile(jarName).toURI.toString
spark.sparkContext.allAddedJars.keys.find(_.contains(jarName))
- .foreach(spark.sparkContext.addedJars("default").remove)
+ .foreach(k =>
spark.sparkContext.addedJars.get("default").foreach(_.remove(k)))
assert(!spark.sparkContext.listJars().exists(_.contains(jarName)))
val e = intercept[AnalysisException] {
sql("CREATE TEMPORARY FUNCTION f1 AS " +
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
