This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new db5cbf104a92 [SPARK-57782][INFRA][DOC] Make pages.yml reuse the doc
image
db5cbf104a92 is described below
commit db5cbf104a92c7fb2263a4562e82e3db6d33618c
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Wed Jul 15 16:54:33 2026 +0800
[SPARK-57782][INFRA][DOC] Make pages.yml reuse the doc image
### What changes were proposed in this pull request?
Run the "GitHub Pages deployment" documentation job inside the prebuilt
documentation container image
`ghcr.io/apache/spark/apache-spark-github-action-image-docs-cache:master-static`
-- the same image that the documentation job in `build_and_test.yml` builds
and runs in. That image is produced from `dev/spark-test-image/docs/Dockerfile`
and published by `build_infra_images_cache.yml`.
As a result, the following steps now come from the image and are removed
from `pages.yml`:
- `Install Python 3.11` and `Install Python dependencies` (the pinned
Sphinx/pandas/grpcio pip list)
- `Install Ruby for documentation generation`
- `Install Pandoc`
Companion changes required to build inside a container, mirroring the
documentation job in `build_and_test.yml`:
- set `LC_ALL`/`LANG` to `C.UTF-8`
- add a `git config --global --add safe.directory ${GITHUB_WORKSPACE}` step
(the doc build invokes git as root inside the container)
- run `dev/free_disk_space_container` to reclaim runner disk now that the
image also occupies it
- keep `setup-java` (Java 17) so `JAVA_HOME` is set for the Scala/SQL doc
generation, and align the Bundler install with `build_and_test.yml`
### Why are the changes needed?
`pages.yml` duplicated the documentation toolchain setup -- a long pinned
Python dependency list, Ruby, and Pandoc -- that is already captured in
`dev/spark-test-image/docs/Dockerfile` and published as a reusable image.
Reusing that image keeps the documentation dependencies in a single source of
truth, removes the duplicated install steps, and avoids reinstalling the
toolchain on every run.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Validated by running the updated workflow end-to-end on a fork. Since the
workflow only triggers on push to `master` in `apache/spark`, it was
temporarily enabled for the PR branch (commit c1f16f3508d, reverted in
821a563e919, so the final diff is unchanged), with only the two Pages deploy
steps skipped because Pages is not enabled on the fork.
Successful run:
https://github.com/zhengruifeng/spark/actions/runs/27413443557
The run pulled `apache-spark-github-action-image-docs-cache:master-static`,
built the full documentation inside the container (`SKIP_RDOC=1 bundle exec
jekyll build` against `apache/sparkmaster` sources, ~27 minutes), and uploaded
the built site as a 114 MB `github-pages` artifact. Total run time (~30
minutes) is consistent with recent runs of the current workflow on
`apache/spark` (~31-65 minutes), with the environment setup reduced to a ~50
second image pull plus a ~20 second `bundl [...]
The deployment has not been checked. I am not sure how to check it, so I
will monitor the Apache Spark CI run after this change is merged.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (model: claude-opus-4-8)
Closes #56393 from zhengruifeng/pages-reuse-doc-image-dev1.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.github/workflows/pages.yml | 35 +++++++++++++----------------------
1 file changed, 13 insertions(+), 22 deletions(-)
diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml
index 49b342f1dad3..dd7b2beffb92 100644
--- a/.github/workflows/pages.yml
+++ b/.github/workflows/pages.yml
@@ -37,9 +37,13 @@ jobs:
pages: write
environment:
name: github-pages # https://github.com/actions/deploy-pages/issues/271
+ container:
+ image:
ghcr.io/apache/spark/apache-spark-github-action-image-docs-cache:master-static
env:
SPARK_TESTING: 1 # Reduce some noise in the logs
RELEASE_VERSION: 'In-Progress'
+ LC_ALL: C.UTF-8
+ LANG: C.UTF-8
if: github.repository == 'apache/spark'
steps:
- name: Checkout Spark repository
@@ -47,36 +51,23 @@ jobs:
with:
repository: apache/spark
ref: 'master'
+ - name: Add GITHUB_WORKSPACE to git trust safe.directory
+ run: |
+ git config --global --add safe.directory ${GITHUB_WORKSPACE}
+ - name: Free up disk space
+ run: ./dev/free_disk_space_container
- name: Install Java 17
uses: actions/setup-java@v5
with:
distribution: zulu
java-version: 17
- - name: Install Python 3.11
- uses: actions/setup-python@v6
- with:
- python-version: '3.11'
- architecture: x64
- cache: 'pip'
- - name: Install Python dependencies
- run: |
- pip install 'sphinx==8.2.3' mkdocs 'pydata_sphinx_theme>=0.13'
sphinx-copybutton nbsphinx numpydoc jinja2 markupsafe 'pyzmq<24.0.0' \
- ipython ipython_genutils sphinx_plotly_directive 'numpy>=1.23.2'
pyarrow 'pandas==2.3.3' 'plotly>=4.8' 'docutils' \
- 'flake8==3.9.0' 'mypy==1.8.0' 'pytest==7.1.3'
'pytest-mypy-plugins==1.9.3' 'ruff==0.14.8' \
- 'pandas-stubs==1.2.0.53' 'grpcio==1.76.0' 'grpcio-status==1.76.0'
'protobuf==6.33.5' 'grpc-stubs==1.24.11'
'googleapis-common-protos-stubs==2.2.0' \
- - name: Install Ruby for documentation generation
- uses: ruby/setup-ruby@4dc28cf14d77b0afa6832d9765ac422dbf0dfedd # v1
- with:
- ruby-version: '3.3'
- bundler-cache: true
- - name: Install Pandoc
- run: |
- sudo apt-get update -y
- sudo apt-get install pandoc
- name: Install dependencies for documentation generation
run: |
+ # Keep the version of Bundler here in sync with the following
locations:
+ # - dev/create-release/spark-rm/Dockerfile
+ # - docs/README.md
+ gem install bundler -v 2.4.22
cd docs
- gem install bundler -v 2.4.22 -n /usr/local/bin
bundle install --retry=100
- name: Run documentation build
run: |
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
