This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 4530b12090e [SPARK-45356][INFRA] Adjust the test configuration of
`maven_test.yml`
4530b12090e is described below
commit 4530b12090eddf2d01394ab7b6980d0e2721434a
Author: yangjie01 <[email protected]>
AuthorDate: Fri Nov 24 17:01:05 2023 +0800
[SPARK-45356][INFRA] Adjust the test configuration of `maven_test.yml`
### What changes were proposed in this pull request?
This pr has made the following changes to the configuration file
`maven_test.yml` used for Maven daily testing:
1. Added tests for the `common/utils` and `sql/api` modules
2. Referring to `build_and_test.yml`, moved the `mllib-local` and `mllib`
modules to a new group
3. Using `Maven local repository Cache` instead of `Coursier local
repository Cache`
4. Referring to `build_and_test.yml`, switched to using zulu OpenJDK model
for testing
5. Referring to `build_and_test.yml`, installed python test dependencies
for the `connect` test group
### Why are the changes needed?
Keep the configuration of Maven daily testing basically consistent with sbt
test
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Monitor maven daily test
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #43999 from LuciferYang/SPARK-45356-2.
Authored-by: yangjie01 <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
.github/workflows/maven_test.yml | 22 ++++++++++++----------
1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/.github/workflows/maven_test.yml b/.github/workflows/maven_test.yml
index b43df059b0a..89d46e7961e 100644
--- a/.github/workflows/maven_test.yml
+++ b/.github/workflows/maven_test.yml
@@ -57,15 +57,17 @@ jobs:
- hive2.3
modules:
- >-
-
core,launcher,common#unsafe,common#kvstore,common#network-common,common#network-shuffle,common#sketch
+
core,launcher,common#unsafe,common#kvstore,common#network-common,common#network-shuffle,common#sketch,common#utils
- >-
- graphx,streaming,mllib-local,mllib,hadoop-cloud
+ graphx,streaming,hadoop-cloud
+ - >-
+ mllib-local,mllib
- >-
repl,sql#hive-thriftserver
- >-
connector#kafka-0-10,connector#kafka-0-10-sql,connector#kafka-0-10-token-provider,connector#spark-ganglia-lgpl,connector#protobuf,connector#avro
- >-
-
sql#catalyst,resource-managers#yarn,resource-managers#kubernetes#core
+
sql#api,sql#catalyst,resource-managers#yarn,resource-managers#kubernetes#core
- >-
connect
# Here, we split Hive and SQL tests into some of slow ones and the
rest of them.
@@ -140,29 +142,29 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties',
'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash',
'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
+ - name: Cache Maven local repository
uses: actions/cache@v3
with:
- path: ~/.cache/coursier
- key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{
hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ path: ~/.m2/repository
+ key: java${{ matrix.java }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
- ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ java${{ matrix.java }}-maven-
- name: Install Java ${{ matrix.java }}
uses: actions/setup-java@v3
with:
- distribution: temurin
+ distribution: zulu
java-version: ${{ matrix.java }}
- name: Install Python 3.9
uses: actions/setup-python@v4
# We should install one Python that is higher than 3+ for SQL and Yarn
because:
# - SQL component also has Python related tests, for example,
IntegratedUDFTestUtils.
# - Yarn has a Python specific test too, for example, YarnClusterSuite.
- if: contains(matrix.modules, 'resource-managers#yarn') ||
(contains(matrix.modules, 'sql#core'))
+ if: contains(matrix.modules, 'resource-managers#yarn') ||
(contains(matrix.modules, 'sql#core')) || contains(matrix.modules, 'connect')
with:
python-version: '3.9'
architecture: x64
- name: Install Python packages (Python 3.9)
- if: (contains(matrix.modules, 'sql#core'))
+ if: (contains(matrix.modules, 'sql#core')) || contains(matrix.modules,
'connect')
run: |
python3.9 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy
unittest-xml-reporting 'grpcio==1.59.3' 'grpcio-status==1.59.3'
'protobuf==4.25.1'
python3.9 -m pip list
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
