This is an automated email from the ASF dual-hosted git repository. ruifengz pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push: new a8539688186 [SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests` a8539688186 is described below commit a8539688186be40c81c39050e70a49a9ef01519f Author: Ruifeng Zheng <ruife...@apache.org> AuthorDate: Wed Jul 26 15:52:38 2023 +0800 [SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests` ### What changes were proposed in this pull request? it seems that `run_python_packaging_tests` requires some disk space and cause some pyspark modules fail, this PR is to make `run_python_packaging_tests` only enabled within `pyspark-errors` (which is the smallest pyspark test module) ![image](https://github.com/apache/spark/assets/7322292/2d37c141-15b8-4d9f-bfbd-4dd7782ab62e) ### Why are the changes needed? 1, it seems it is the `run_python_packaging_tests` that cause the `No space left` error; 2, the `run_python_packaging_tests` is tested in all `pyspark-*` test modules, should be deduplicated; ### Does this PR introduce _any_ user-facing change? no, infra-only ### How was this patch tested? updated CI Closes #42146 from zhengruifeng/infra_skip_py_packing_tests. Authored-by: Ruifeng Zheng <ruife...@apache.org> Signed-off-by: Ruifeng Zheng <ruife...@apache.org> (cherry picked from commit 748eaff4e2177466dd746f6fbb82de8544bc7168) Signed-off-by: Ruifeng Zheng <ruife...@apache.org> --- .github/workflows/build_and_test.yml | 16 ++++++++++++++-- dev/run-tests.py | 2 +- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index 54fe9f38ddd..1fcca7e4c39 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -204,6 +204,7 @@ jobs: HIVE_PROFILE: ${{ matrix.hive }} GITHUB_PREV_SHA: ${{ github.event.before }} SPARK_LOCAL_IP: localhost + SKIP_PACKAGING: true steps: - name: Checkout Spark repository uses: actions/checkout@v3 @@ -343,6 +344,8 @@ jobs: java: - ${{ inputs.java }} modules: + - >- + pyspark-errors - >- pyspark-sql, pyspark-mllib, pyspark-resource, pyspark-testing - >- @@ -352,7 +355,7 @@ jobs: - >- pyspark-pandas-slow - >- - pyspark-connect, pyspark-errors + pyspark-connect - >- pyspark-pandas-connect - >- @@ -365,6 +368,7 @@ jobs: SPARK_LOCAL_IP: localhost SKIP_UNIDOC: true SKIP_MIMA: true + SKIP_PACKAGING: true METASPACE_SIZE: 1g steps: - name: Checkout Spark repository @@ -413,14 +417,20 @@ jobs: python3.9 -m pip list pypy3 -m pip list - name: Install Conda for pip packaging test + if: ${{ matrix.modules == 'pyspark-errors' }} run: | curl -s https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > miniconda.sh bash miniconda.sh -b -p $HOME/miniconda # Run the tests. - name: Run tests env: ${{ fromJSON(inputs.envs) }} + shell: 'script -q -e -c "bash {0}"' run: | - export PATH=$PATH:$HOME/miniconda/bin + if [[ "$MODULES_TO_TEST" == "pyspark-errors" ]]; then + export PATH=$PATH:$HOME/miniconda/bin + export SKIP_PACKAGING=false + echo "Python Packaging Tests Enabled!" + fi ./dev/run-tests --parallelism 1 --modules "$MODULES_TO_TEST" - name: Upload coverage to Codecov if: fromJSON(inputs.envs).PYSPARK_CODECOV == 'true' @@ -456,6 +466,7 @@ jobs: GITHUB_PREV_SHA: ${{ github.event.before }} SPARK_LOCAL_IP: localhost SKIP_MIMA: true + SKIP_PACKAGING: true steps: - name: Checkout Spark repository uses: actions/checkout@v3 @@ -900,6 +911,7 @@ jobs: SPARK_LOCAL_IP: localhost ORACLE_DOCKER_IMAGE_NAME: gvenzl/oracle-xe:21.3.0 SKIP_MIMA: true + SKIP_PACKAGING: true steps: - name: Checkout Spark repository uses: actions/checkout@v3 diff --git a/dev/run-tests.py b/dev/run-tests.py index c0c281b549e..9bf3095edb7 100755 --- a/dev/run-tests.py +++ b/dev/run-tests.py @@ -395,7 +395,7 @@ def run_python_tests(test_modules, parallelism, with_coverage=False): def run_python_packaging_tests(): - if not os.environ.get("SPARK_JENKINS"): + if not os.environ.get("SPARK_JENKINS") and os.environ.get("SKIP_PACKAGING", "false") != "true": set_title_and_block("Running PySpark packaging tests", "BLOCK_PYSPARK_PIP_TESTS") command = [os.path.join(SPARK_HOME, "dev", "run-pip-tests")] run_cmd(command) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org