This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 8fc2bc805880 [SPARK-60328][SQL][TEST] Use a unique jar copy in the
HiveDDLSuite SPARK-34261 test
8fc2bc805880 is described below
commit 8fc2bc805880a7d0a142ac924aae90b1f3a69108
Author: John Zhuge <[email protected]>
AuthorDate: Mon Nov 2 14:17:06 2026 +0900
[SPARK-60328][SQL][TEST] Use a unique jar copy in the HiveDDLSuite
SPARK-34261 test
### What changes were proposed in this pull request?
The HiveDDLSuite test SPARK-34261: Avoid side effect if create exists
temporary function now uses its own copy of TestUDTFJar.jar, with a name unique
to this test. The copy cannot already be in `listJars()`, so the cleanup that
removed the shared jar from `SparkContext.addedJars` is gone.
### Why are the changes needed?
The old cleanup depends on how map `addedJars` is keyed. It clears only the
"default" bucket and only the first matching key, but `listJars()` flattens
every bucket. If the shared jar is registered under another bucket or a second
key, the first assertion fails depending on test order. Clearing more buckets
would also drop the jar from other live sessions. This line already needed a
fix once when jar tracking changed (SPARK-46237).
With a unique jar, the test only checks what it is meant to check: a
failing `CREATE TEMPORARY FUNCTION ... USING JAR` does not add the jar, and
`CREATE OR REPLACE` does.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
```
build/sbt -Phive 'hive/testOnly *HiveDDLSuite -- -z "SPARK-34261"'
[info] - SPARK-34261: Avoid side effect if create exists temporary function
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
```
It also passes when the shared `TestUDTFJar.jar` is added with `ADD JAR`
before the test body, which shows that leftover state from the shared jar no
longer affects it.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code 2.1.301 (Claude Opus 5.5)
Closes #59587 from jzhuge/SPARK-34261-followup-clear-all-buckets.
Authored-by: John Zhuge <[email protected]>
Signed-off-by: Kousuke Saruta <[email protected]>
(cherry picked from commit 9e771a7bef9a5f5bba407dec3f5aebd5ec784adc)
Signed-off-by: Kousuke Saruta <[email protected]>
---
.../scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
index 54d5cc734fd2..0f2cb6ab33e3 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.hive.execution
import java.io.File
import java.net.URI
+import java.nio.file.{Files, StandardCopyOption}
import java.time.{LocalDateTime, LocalTime}
import java.util.Locale
@@ -3318,11 +3319,11 @@ class HiveDDLSuite
withUserDefinedFunction("f1" -> true) {
sql("CREATE TEMPORARY FUNCTION f1 AS
'org.apache.hadoop.hive.ql.udf.UDFUUID'")
- val udtfJar = TestUDTFJar.jar
+ // A copy unique to this test cannot be registered by an earlier test.
+ val udtfJar = File.createTempFile("SPARK-34261-", ".jar",
Utils.createTempDir())
+ Files.copy(TestUDTFJar.jar.toPath, udtfJar.toPath,
StandardCopyOption.REPLACE_EXISTING)
val jarName = udtfJar.getName
val jar = udtfJar.toURI.toString
- spark.sparkContext.allAddedJars.keys.find(_.contains(jarName))
- .foreach(k =>
spark.sparkContext.addedJars.get("default").foreach(_.remove(k)))
assert(!spark.sparkContext.listJars().exists(_.contains(jarName)))
val e = intercept[AnalysisException] {
sql("CREATE TEMPORARY FUNCTION f1 AS " +
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]