This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8b8d92334fa [SPARK-42167][INFRA] Improve GitHub Action `lint` job to
stop on failures earlier
8b8d92334fa is described below
commit 8b8d92334faee889d4106a665ce769403e098240
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Jan 24 09:55:46 2023 -0800
[SPARK-42167][INFRA] Improve GitHub Action `lint` job to stop on failures
earlier
### What changes were proposed in this pull request?
This PR aims to improve `GitHub Action` **lint** job to stop on failure
earlier.
### Why are the changes needed?
Currently, we designed to put all installation steps before testing.
However, we don't need to install `SparkR` dependency if the PR fails at
`LICENSE` check.
By pairing the installation and testing steps, we can skip as much as
possible in case of failures.
### Does this PR introduce _any_ user-facing change?
No. This is a INFRA change.
### How was this patch tested?
Manual review.
Closes #39716 from dongjoon-hyun/SPARK-42167.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_and_test.yml | 53 ++++++++++++++++++------------------
1 file changed, 27 insertions(+), 26 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 46cb7fe27db..07c5ddf5c73 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -550,6 +550,19 @@ jobs:
key: docs-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
docs-maven-
+ - name: Install Java 8
+ uses: actions/setup-java@v3
+ with:
+ distribution: temurin
+ java-version: 8
+ - name: License test
+ run: ./dev/check-license
+ - name: Dependencies test
+ run: ./dev/test-dependencies.sh
+ - name: Scala linter
+ run: ./dev/lint-scala
+ - name: Java linter
+ run: ./dev/lint-java
- name: Install Python linter dependencies
run: |
# TODO(SPARK-32407): Sphinx 3.1+ does not correctly index nested
classes.
@@ -557,13 +570,24 @@ jobs:
# Jinja2 3.0.0+ causes error when building with Sphinx.
# See also https://issues.apache.org/jira/browse/SPARK-35375.
python3.9 -m pip install 'flake8==3.9.0' pydata_sphinx_theme
'mypy==0.920' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' numpydoc
'jinja2<3.0.0' 'black==22.6.0'
- python3.9 -m pip install 'pandas-stubs==1.2.0.53'
+ python3.9 -m pip install 'pandas-stubs==1.2.0.53' ipython
'grpcio==1.48.1' 'grpc-stubs==1.24.11' 'googleapis-common-protos-stubs==2.2.0'
+ - name: Python linter
+ run: PYTHON_EXECUTABLE=python3.9 ./dev/lint-python
- name: Install dependencies for Python code generation check
run: |
# See more in "Installation"
https://docs.buf.build/installation#tarball
curl -LO
https://github.com/bufbuild/buf/releases/download/v1.12.0/buf-Linux-x86_64.tar.gz
mkdir -p $HOME/buf
tar -xvzf buf-Linux-x86_64.tar.gz -C $HOME/buf --strip-components 1
+ python3.9 -m pip install 'protobuf==3.19.5' 'mypy-protobuf==3.3.0'
+ - name: Python code generation check
+ run: if test -f ./dev/connect-check-protos.py; then
PATH=$PATH:$HOME/buf/bin PYTHON_EXECUTABLE=python3.9
./dev/connect-check-protos.py; fi
+ - name: Install JavaScript linter dependencies
+ run: |
+ apt update
+ apt-get install -y nodejs npm
+ - name: JS linter
+ run: ./dev/lint-js
- name: Install R linter dependencies and SparkR
run: |
apt update
@@ -573,10 +597,6 @@ jobs:
Rscript -e "install.packages(c('devtools'),
repos='https://cloud.r-project.org/')"
Rscript -e "devtools::install_version('lintr', version='2.0.1',
repos='https://cloud.r-project.org')"
./R/install-dev.sh
- - name: Install JavaScript linter dependencies
- run: |
- apt update
- apt-get install -y nodejs npm
- name: Install dependencies for documentation generation
run: |
# pandoc is required to generate PySpark APIs as well in nbsphinx.
@@ -587,9 +607,9 @@ jobs:
# See also https://issues.apache.org/jira/browse/SPARK-35375.
# Pin the MarkupSafe to 2.0.1 to resolve the CI error.
# See also https://issues.apache.org/jira/browse/SPARK-38279.
- python3.9 -m pip install 'sphinx<3.1.0' mkdocs pydata_sphinx_theme
ipython nbsphinx numpydoc 'jinja2<3.0.0' 'markupsafe==2.0.1' 'pyzmq<24.0.0'
+ python3.9 -m pip install 'sphinx<3.1.0' mkdocs pydata_sphinx_theme
nbsphinx numpydoc 'jinja2<3.0.0' 'markupsafe==2.0.1' 'pyzmq<24.0.0'
python3.9 -m pip install ipython_genutils # See SPARK-38517
- python3.9 -m pip install sphinx_plotly_directive 'numpy>=1.20.0'
pyarrow pandas 'plotly>=4.8' 'grpcio==1.48.1' 'protobuf==3.19.5'
'mypy-protobuf==3.3.0' 'grpc-stubs==1.24.11'
'googleapis-common-protos-stubs==2.2.0'
+ python3.9 -m pip install sphinx_plotly_directive 'numpy>=1.20.0'
pyarrow pandas 'plotly>=4.8'
python3.9 -m pip install 'docutils<0.18.0' # See SPARK-39421
apt-get update -y
apt-get install -y ruby ruby-dev
@@ -599,27 +619,8 @@ jobs:
gem install bundler
cd docs
bundle install
- - name: Install Java 8
- uses: actions/setup-java@v3
- with:
- distribution: temurin
- java-version: 8
- - name: Scala linter
- run: ./dev/lint-scala
- - name: Java linter
- run: ./dev/lint-java
- - name: Python linter
- run: PYTHON_EXECUTABLE=python3.9 ./dev/lint-python
- - name: Python code generation check
- run: if test -f ./dev/connect-check-protos.py; then
PATH=$PATH:$HOME/buf/bin PYTHON_EXECUTABLE=python3.9
./dev/connect-check-protos.py; fi
- name: R linter
run: ./dev/lint-r
- - name: JS linter
- run: ./dev/lint-js
- - name: License test
- run: ./dev/check-license
- - name: Dependencies test
- run: ./dev/test-dependencies.sh
- name: Run documentation build
run: |
cd docs
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
