jzhuge opened a new pull request, #59587:
URL: https://github.com/apache/spark/pull/59587
### What changes were proposed in this pull request?
The `HiveDDLSuite` test `SPARK-34261: Avoid side effect if create exists
temporary function` starts by removing `TestUDTF.jar` from the driver, so that
its later assertions are not affected by earlier tests. Today that cleanup only
looks in the `"default"` bucket of `SparkContext.addedJars`.
This PR removes the jar from every bucket:
```scala
// before
.foreach(k =>
spark.sparkContext.addedJars.get("default").foreach(_.remove(k)))
// after
.foreach(k => spark.sparkContext.addedJars.values.foreach(_.remove(k)))
```
### Why are the changes needed?
`addedJars` is keyed by job artifact UUID. The cleanup and the assertion
right after it read different views of that map:
- **Cleanup** removes the jar from `addedJars("default")` only.
- **Assertion** checks `listJars()`, which is `allAddedJars.keySet`, and
`allAddedJars` flattens every bucket.
A jar does not have to be in the `"default"` bucket. When a session's
`ArtifactManager` state is active, `SparkContext.addJar` stores the jar under
that session's UUID. SQL commands run inside `artifactManager.withResources`
(see `SQLExecution`), and `spark.sql.artifact.isolation.enabled` defaults to
`true`. So if an earlier test in the same JVM left `TestUDTF.jar` under a
session bucket, the cleanup removes nothing and
`assert(!listJars().exists(...))` fails. Whether this happens depends on which
tests ran before this one in the JVM, so it shows up as a flaky failure that is
hard to reproduce.
SPARK-46237 already made this line null-safe (`addedJars.get("default")`),
but the cleanup still misses non-default buckets.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Ran the updated test locally on master:
```
build/sbt -Phive 'hive/testOnly *HiveDDLSuite -- -z "SPARK-34261"'
...
[info] - SPARK-34261: Avoid side effect if create exists temporary function
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
```
When this test runs alone, no earlier test leaves the jar behind, so this
run only shows that the new cleanup does not break the test. It does not
reproduce the order-dependent failure. Existing CI covers the full
`HiveDDLSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code 2.1.301 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
