This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f6b670a650d [SPARK-45950][INFRA][CORE] Fix
`IvyTestUtils#createIvyDescriptor` function and make `common-utils` module can
run tests on GitHub Action
f6b670a650d is described below
commit f6b670a650d23ef4b2be5ed5c903091361a26a63
Author: yangjie01 <yangji...@baidu.com>
AuthorDate: Fri Nov 17 08:41:14 2023 +0900
[SPARK-45950][INFRA][CORE] Fix `IvyTestUtils#createIvyDescriptor` function
and make `common-utils` module can run tests on GitHub Action
### What changes were proposed in this pull request?
This PR mainly does two things:
1. It revert a line of code in `IvyTestUtils.scala` that was mistakenly
deleted in SPARK-45506 | https://github.com/apache/spark/pull/43354 to ensure
that the `ivy.xml` file generated by `IvyTestUtils#createIvyDescriptor` is
complete. Before this PR, the generated `ivy.xml` file would missing the
`</ivy-module>` end tag, which would cause two test cases in `MavenUtilsSuite`
to fail. We can reproduce the problem by executing the `build/sbt
"common-utils/test"` command:
```
[info] MavenUtilsSuite:
[info] - incorrect maven coordinate throws error (8 milliseconds)
[info] - create repo resolvers (24 milliseconds)
[info] - create additional resolvers (3 milliseconds)
:: loading settings :: url =
jar:file:/Users/yangjie01/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/ivy/ivy/2.5.1/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
[info] - add dependencies works correctly (35 milliseconds)
[info] - excludes works correctly (2 milliseconds)
[info] - ivy path works correctly (3 seconds, 759 milliseconds)
[info] - search for artifact at local repositories *** FAILED *** (2
seconds, 833 milliseconds)
[info] java.lang.RuntimeException: [unresolved dependency:
my.great.lib#mylib;0.1: java.text.ParseException: [[Fatal Error]
ivy-0.1.xml.original:22:18: XML document structures must start and end within
the same entity. in
file:/SourceCode/git/spark-mine-sbt/target/tmp/ivy-8b860aca-a9c4-4af9-b15a-ac8c6049b773/cache/my.great.lib/mylib/ivy-0.1.xml.original
[info] ]]
[info] at
org.apache.spark.util.MavenUtils$.resolveMavenCoordinates(MavenUtils.scala:459)
[info] at
org.apache.spark.util.MavenUtilsSuite.$anonfun$new$25(MavenUtilsSuite.scala:173)
[info] at
org.apache.spark.util.MavenUtilsSuite.$anonfun$new$25$adapted(MavenUtilsSuite.scala:172)
[info] at
org.apache.spark.util.IvyTestUtils$.withRepository(IvyTestUtils.scala:373)
[info] at
org.apache.spark.util.MavenUtilsSuite.$anonfun$new$18(MavenUtilsSuite.scala:172)
[info] at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] at org.scalatest.Transformer.apply(Transformer.scala:22)
[info] at org.scalatest.Transformer.apply(Transformer.scala:20)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[info] at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
[info] at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
[info] at
org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[info] at
org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[info] at
org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info] at scala.collection.immutable.List.foreach(List.scala:333)
[info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info] at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[info] at
org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[info] at org.scalatest.Suite.run(Suite.scala:1114)
[info] at org.scalatest.Suite.run$(Suite.scala:1096)
[info] at
org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
[info] at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
[info] at
org.apache.spark.util.MavenUtilsSuite.org$scalatest$BeforeAndAfterAll$$super$run(MavenUtilsSuite.scala:36)
[info] at
org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at
org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at
org.apache.spark.util.MavenUtilsSuite.run(MavenUtilsSuite.scala:36)
[info] at
org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:833)
[info] - dependency not found throws RuntimeException (2 seconds, 274
milliseconds)
[info] - neglects Spark and Spark's dependencies (1 second, 934
milliseconds)
[info] - exclude dependencies end to end (953 milliseconds)
:: loading settings :: file =
/Users/yangjie01/SourceCode/git/spark-mine-sbt/target/tmp/ivy-8b860aca-a9c4-4af9-b15a-ac8c6049b773/ivysettings.xml
[info] - load ivy settings file *** FAILED *** (167 milliseconds)
[info] java.lang.RuntimeException: [unresolved dependency:
my.great.lib#mylib;0.1: java.text.ParseException: [[Fatal Error]
ivy-0.1.xml.original:22:18: XML document structures must start and end within
the same entity. in
file:/SourceCode/git/spark-mine-sbt/target/tmp/ivy-8b860aca-a9c4-4af9-b15a-ac8c6049b773/cache/my.great.lib/mylib/ivy-0.1.xml.original
[info] ]]
[info] at
org.apache.spark.util.MavenUtils$.resolveMavenCoordinates(MavenUtils.scala:459)
[info] at
org.apache.spark.util.MavenUtilsSuite.$anonfun$new$40(MavenUtilsSuite.scala:260)
[info] at
org.apache.spark.util.MavenUtilsSuite.$anonfun$new$40$adapted(MavenUtilsSuite.scala:259)
[info] at
org.apache.spark.util.IvyTestUtils$.withRepository(IvyTestUtils.scala:373)
[info] at
org.apache.spark.util.MavenUtilsSuite.$anonfun$new$39(MavenUtilsSuite.scala:259)
[info] at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] at org.scalatest.Transformer.apply(Transformer.scala:22)
[info] at org.scalatest.Transformer.apply(Transformer.scala:20)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[info] at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
[info] at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
[info] at
org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[info] at
org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[info] at
org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info] at scala.collection.immutable.List.foreach(List.scala:333)
[info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info] at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[info] at
org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[info] at org.scalatest.Suite.run(Suite.scala:1114)
[info] at org.scalatest.Suite.run$(Suite.scala:1096)
[info] at
org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
[info] at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
[info] at
org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
[info] at
org.apache.spark.util.MavenUtilsSuite.org$scalatest$BeforeAndAfterAll$$super$run(MavenUtilsSuite.scala:36)
[info] at
org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at
org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at
org.apache.spark.util.MavenUtilsSuite.run(MavenUtilsSuite.scala:36)
[info] at
org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info] at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info] at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:833)
[info] - SPARK-10878: test resolution files cleaned after resolving
artifact (880 milliseconds)
[info] - SPARK-34624: should ignore non-jar dependencies (145 milliseconds)
[info] Run completed in 13 seconds, 493 milliseconds.
[info] Total number of tests run: 13
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 11, failed 2, canceled 0, ignored 0, pending 0
[info] *** 2 TESTS FAILED ***
[error] Failed tests:
[error] org.apache.spark.util.MavenUtilsSuite
```
2. Added `sbt_test_goals` to the `utils` module in `modules.py` to ensure
that the unit tests in the `common-utils` module will be verified by GitHub
Action.
### Why are the changes needed?
Fix the failed test cases in `MavenUtilsSuite` and let GitHub Action verify
the unit tests in the `common-utils` module.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Monitor GA, the unit tests in the common-utils module should be run, and
`MavenUtilsSuite` should test successfully.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #43834 from LuciferYang/utils-ga-test.
Authored-by: yangjie01 <yangji...@baidu.com>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
common/utils/src/main/scala/org/apache/spark/util/IvyTestUtils.scala | 1 +
dev/sparktestsupport/modules.py | 3 +++
2 files changed, 4 insertions(+)
diff --git
a/common/utils/src/main/scala/org/apache/spark/util/IvyTestUtils.scala
b/common/utils/src/main/scala/org/apache/spark/util/IvyTestUtils.scala
index 545693665e6..50312646bdb 100644
--- a/common/utils/src/main/scala/org/apache/spark/util/IvyTestUtils.scala
+++ b/common/utils/src/main/scala/org/apache/spark/util/IvyTestUtils.scala
@@ -226,6 +226,7 @@ private[spark] object IvyTestUtils {
val inside = deps.map(ivyArtifactWriter).mkString("\n")
"\n <dependencies>\n" + inside + "\n </dependencies>"
}.getOrElse("")
+ content += "\n</ivy-module>"
writeFile(dir, "ivy.xml", content.trim)
}
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 8aa93821637..c8981c779ca 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -119,6 +119,9 @@ utils = Module(
source_file_regexes=[
"common/utils/",
],
+ sbt_test_goals=[
+ "common-utils/test",
+ ],
)
kvstore = Module(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
