gh-yzou commented on code in PR #4588:
URL: https://github.com/apache/polaris/pull/4588#discussion_r3366043072
##########
plugins/spark/v3.5/integration/build.gradle.kts:
##########
@@ -162,3 +171,69 @@ tasks.named<Test>("intTest").configure {
// For Spark integration tests
addSparkJvmOptions()
}
+
+// Bundle-jar sanity test
+testing {
+ suites {
+ register<JvmTestSuite>("sparkBundleTest") {
+ useJUnitJupiter()
+ dependencies {
+
implementation("org.apache.spark:spark-sql_${scalaVersion}:${spark35Version}") {
+ exclude("org.apache.logging.log4j", "log4j-slf4j2-impl")
+ exclude("org.apache.logging.log4j", "log4j-1.2-api")
+ exclude("org.apache.logging.log4j", "log4j-core")
+ exclude("org.slf4j", "jul-to-slf4j")
+ }
+ implementation(
+
"org.apache.iceberg:iceberg-spark-runtime-${sparkMajorVersion}_${scalaVersion}:${icebergVersion}"
+ )
+
implementation(enforcedPlatform("org.scala-lang:scala-library:${scalaLibraryVersion}"))
+
implementation(enforcedPlatform("org.scala-lang:scala-reflect:${scalaLibraryVersion}"))
+ implementation(libs.antlr4.runtime)
+ implementation(libs.javax.servlet.api)
+ runtimeOnly("org.apache.logging.log4j:log4j-core:2.26.0")
+
+ // Bundle jar as a file artifact so the shadow jar is the only source
of
+ // polaris-spark/polaris-core classes
+ runtimeOnly(files(sparkBundleJarTask.flatMap { it.archiveFile }))
Review Comment:
Yes, the purpose is to test both --jar (spark.jars) and --package
(spark.packages) workflow. --jar workload requires gradlew to build the bundle
jar, --package requires publishToMavenLocal so that it can do the installation
through local maven. If spark.jars doesn't work, I don't think
publishToMavenLocal would help there, because spark.jars only require the
bundle jar to be built.
Current regression test framework is the closet to actual user workflow,
which mimics the actual workflow and tests all dependency installation. I am
curious what is the concern of staying with the same framework as before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
