This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 104a546f6d9 [SPARK-42334][CONNECT][BUILD] Make sure connect client
assembly and sql package is built before running client tests - SBT
104a546f6d9 is described below
commit 104a546f6d9e1d2d7c63ced9cbbdb588110252e9
Author: yangjie01 <[email protected]>
AuthorDate: Sat Feb 4 21:42:38 2023 -0400
[SPARK-42334][CONNECT][BUILD] Make sure connect client assembly and sql
package is built before running client tests - SBT
### What changes were proposed in this pull request?
`build/sbt clean "connect-client-jvm/test"` will fail after SPARK-42172
merged, so this pr make sure sbt assembles the connect client assembly jar and
sbt package the sql jar before we run `CompatibilitySuite` of client module.
### Why are the changes needed?
Similar as SPARK-42284, it makes it easier to develop and test the JVM
client for Spark Connect.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually tested.
```
build/sbt clean "connect-client-jvm/test"
```
**Before**
```
[info] - compatibility MiMa tests *** FAILED *** (34 milliseconds)
[info] java.lang.AssertionError: assertion failed: Failed to find the jar
inside folder:
/Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/client/jvm/target
[info] at scala.Predef$.assert(Predef.scala:223)
[info] at
org.apache.spark.sql.connect.client.util.IntegrationTestUtils$.findJar(IntegrationTestUtils.scala:67)
[info] at
org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar$lzycompute(CompatibilitySuite.scala:57)
[info] at
org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar(CompatibilitySuite.scala:53)
[info] at
org.apache.spark.sql.connect.client.CompatibilitySuite.$anonfun$new$1(CompatibilitySuite.scala:69)
....
[info] *** 2 TESTS FAILED ***
[error] Failed tests:
[error] org.apache.spark.sql.connect.client.CompatibilitySuite
[error] (connect-client-jvm / Test / test) sbt.TestsFailedException: Tests
unsuccessful
[error] Total time: 196 s (03:16), completed 2023-2-3 17:20:40
```
**After**
```
[info] Run completed in 20 seconds, 652 milliseconds.
[info] Total number of tests run: 31
[info] Suites: completed 6, aborted 0
[info] Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 230 s (03:50), completed 2023-2-3 17:35:37
```
Closes #39874 from LuciferYang/make-test.
Authored-by: yangjie01 <[email protected]>
Signed-off-by: Herman van Hovell <[email protected]>
---
project/SparkBuild.scala | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 4eb17e88d4d..a4c8d62dd6e 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -829,6 +829,7 @@ object SparkConnect {
object SparkConnectClient {
import BuildCommons.protoVersion
+ val buildTestDeps = TaskKey[Unit]("buildTestDeps", "Build needed
dependencies for test.")
lazy val settings = Seq(
// For some reason the resolution from the imported Maven build does not
work for some
@@ -851,8 +852,14 @@ object SparkConnectClient {
)
},
+ buildTestDeps := {
+ (LocalProject("sql") / Compile / Keys.`package`).value
+ (LocalProject("connect") / assembly).value
+ (LocalProject("connect-client-jvm") / assembly).value
+ },
+
// Make sure the connect server assembly jar is available for testing.
- test := ((Test / test) dependsOn (LocalProject("connect") /
assembly)).value,
+ test := ((Test / test) dependsOn (buildTestDeps)).value,
(assembly / test) := { },
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
