This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch branch-1.9
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/branch-1.9 by this push:
new 673ac78c5 [KYUUBI #6441] Kyuubi Spark TPC-DS/H Connector cross version
test
673ac78c5 is described below
commit 673ac78c51408d16e0b2282b3aed6ca2fa24a07d
Author: zhouyifan279 <[email protected]>
AuthorDate: Mon Jun 3 11:08:47 2024 +0800
[KYUUBI #6441] Kyuubi Spark TPC-DS/H Connector cross version test
# :mag: Description
## Issue References ๐
This pull request adds cross-version tests for Kyuubi Spark TPC-DS
Connector and TPC-H Connector.
## Describe Your Solution ๐ง
Add TPC-DS Connector and TPC-H Connector into GitHub Actions job
`spark-connector-cross-version-test`.
## Types of changes :bookmark:
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
## Test Plan ๐งช
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklist ๐
- [x] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6441 from zhouyifan279/tcp-ds/h-cross-version.
Closes #6441
c2abc468a [zhouyifan279] Kyuubi Spark TPC-DS/H Connector cross version test
Authored-by: zhouyifan279 <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 7bf0f57239958fccd5bdcd76f2a953623a5d2575)
Signed-off-by: Cheng Pan <[email protected]>
---
.github/workflows/master.yml | 17 ++++-----
.../spark/kyuubi-spark-connector-tpcds/pom.xml | 41 ++++++++++++++++++++++
.../spark/kyuubi-spark-connector-tpch/pom.xml | 41 ++++++++++++++++++++++
3 files changed, 91 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index 8658aaf25..434308ff3 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -195,6 +195,9 @@ jobs:
comment: [ "normal" ]
env:
SPARK_LOCAL_IP: localhost
+ TEST_MODULES: "extensions/spark/kyuubi-spark-connector-hive,\
+ extensions/spark/kyuubi-spark-connector-tpcds,\
+ extensions/spark/kyuubi-spark-connector-tpch"
steps:
- uses: actions/checkout@v4
- name: Free up disk space
@@ -210,16 +213,14 @@ jobs:
check-latest: false
- name: Setup Maven
uses: ./.github/actions/setup-maven
- - name: Build Kyuubi Spark Hive Connector with Spark-${{
matrix.spark-compile }}
+ - name: Build Kyuubi Spark Connectors with Spark-${{
matrix.spark-compile }}
run: |
- ./build/mvn clean install ${MVN_OPT} -pl
extensions/spark/kyuubi-spark-connector-hive -am \
- -Pjava-${{ matrix.java }} -Pscala-${{ matrix.scala }} -Pspark-${{
matrix.spark-compile }} \
- -DskipTests
- - name: Test Kyuubi Spark Hive Connector with Spark-${{
matrix.spark-runtime }}
+ ./build/mvn clean install ${MVN_OPT} -pl ${TEST_MODULES} -am
-Pjava-${{ matrix.java }} \
+ -Pscala-${{ matrix.scala }} -Pspark-${{ matrix.spark-compile }}
-DskipTests
+ - name: Test Kyuubi Spark Connectors with Spark-${{ matrix.spark-runtime
}}
run: |
- ./build/mvn test ${MVN_OPT} -pl
extensions/spark/kyuubi-spark-connector-hive \
- -Pjava-${{ matrix.java }} -Pscala-${{ matrix.scala }} -Pspark-${{
matrix.spark-runtime }} \
- -Pcross-version-test
+ ./build/mvn test ${MVN_OPT} -pl ${TEST_MODULES} -Pjava-${{
matrix.java }} \
+ -Pscala-${{ matrix.scala }} -Pspark-${{ matrix.spark-runtime }}
-Pcross-version-test
- name: Upload test logs
if: failure()
uses: actions/upload-artifact@v3
diff --git a/extensions/spark/kyuubi-spark-connector-tpcds/pom.xml
b/extensions/spark/kyuubi-spark-connector-tpcds/pom.xml
index ccf90c597..86b6d6690 100644
--- a/extensions/spark/kyuubi-spark-connector-tpcds/pom.xml
+++ b/extensions/spark/kyuubi-spark-connector-tpcds/pom.xml
@@ -202,4 +202,45 @@
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
+
+ <profiles>
+ <profile>
+ <id>cross-version-test</id>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.kyuubi</groupId>
+
<artifactId>kyuubi-spark-connector-tpcds-local_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>system</scope>
+
<systemPath>${project.basedir}/target/kyuubi-spark-connector-tpcds_${scala.binary.version}-${project.version}.jar</systemPath>
+ </dependency>
+ </dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-clean-plugin</artifactId>
+ <configuration>
+
<excludeDefaultDirectories>true</excludeDefaultDirectories>
+ <filesets>
+ <fileset>
+
<directory>target/scala-${scala.binary.version}/classes</directory>
+ <includes>**/*.*</includes>
+ </fileset>
+ </filesets>
+ </configuration>
+ <executions>
+ <execution>
+ <id>clean
target/scala-${scala.binary.version}/classes</id>
+ <goals>
+ <goal>clean</goal>
+ </goals>
+ <phase>process-test-classes</phase>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
diff --git a/extensions/spark/kyuubi-spark-connector-tpch/pom.xml
b/extensions/spark/kyuubi-spark-connector-tpch/pom.xml
index 37b78cfdb..29f0a5945 100644
--- a/extensions/spark/kyuubi-spark-connector-tpch/pom.xml
+++ b/extensions/spark/kyuubi-spark-connector-tpch/pom.xml
@@ -208,4 +208,45 @@
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
+
+ <profiles>
+ <profile>
+ <id>cross-version-test</id>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.kyuubi</groupId>
+
<artifactId>kyuubi-spark-connector-tpch-local_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>system</scope>
+
<systemPath>${project.basedir}/target/kyuubi-spark-connector-tpch_${scala.binary.version}-${project.version}.jar</systemPath>
+ </dependency>
+ </dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-clean-plugin</artifactId>
+ <configuration>
+
<excludeDefaultDirectories>true</excludeDefaultDirectories>
+ <filesets>
+ <fileset>
+
<directory>target/scala-${scala.binary.version}/classes</directory>
+ <includes>**/*.*</includes>
+ </fileset>
+ </filesets>
+ </configuration>
+ <executions>
+ <execution>
+ <id>clean
target/scala-${scala.binary.version}/classes</id>
+ <goals>
+ <goal>clean</goal>
+ </goals>
+ <phase>process-test-classes</phase>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
</project>
