This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/spark-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 59719624d8 Add workflow to automatically generate and commit site HTML
(#697)
59719624d8 is described below
commit 59719624d8f4bb548e1042654e60cb787b2824cf
Author: Nicholas Chammas <[email protected]>
AuthorDate: Sat Jul 11 12:07:07 2026 -0400
Add workflow to automatically generate and commit site HTML (#697)
* add workflow to generate and commit HTML
* ignore `site/`
* allow pushes on my branch for testing
* [html] allow pushes on my branch for testing
* make sitemap generation more deterministic
* [html] make sitemap generation more deterministic
* eliminate extraneous whitespace
* [html] eliminate extraneous whitespace
* remove content/ symlink
* update docs on building site
* split pr vs asf-site workflows
* clarify purpose of workflow condition
* capitalization
* add concurrency group
* add note about `[html]`
* check commit author + factor out common steps
* `shell` is required in composite actions
* latest build passes; remove testing branch
---------
Co-authored-by: github-actions[bot]
<github-actions[bot]@users.noreply.github.com>
---
.github/CONTRIBUTING.md | 6 ++-
.github/PULL_REQUEST_TEMPLATE.md | 1 -
.github/actions/build-html/action.yml | 21 ++++++++++
.github/workflows/doc_gen.yml | 76 -----------------------------------
.github/workflows/html-build.yml | 16 ++++++++
.github/workflows/html-push.yml | 43 ++++++++++++++++++++
.gitignore | 2 +-
README.md | 28 +++++--------
_config.yml | 2 +-
content | 1 -
site/sitemap.xml | 39 +++++++++---------
sitemap.xml | 15 +++++--
12 files changed, 127 insertions(+), 123 deletions(-)
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
index 7b5dc0b348..ab2057b8e7 100644
--- a/.github/CONTRIBUTING.md
+++ b/.github/CONTRIBUTING.md
@@ -1 +1,5 @@
-Make sure that you generate site HTML with `jekyll build`, and include the
changes to the HTML in your pull request also. See README.md for more
information.
+# Contributing to this repo
+
+Build and test your changes locally according to the instructions in
[README](../README.md).
+
+Once you've done that, submit a pull request with your changes. You only need
to commit your changes to the Markdown source. A GitHub Actions workflow will
[generate the corresponding HTML and push it for
you](./workflows/html-push.yml).
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
deleted file mode 100644
index 9e1ffb348a..0000000000
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ /dev/null
@@ -1 +0,0 @@
-<!-- *Make sure that you generate site HTML with `bundle exec jekyll build`,
and include the changes to the HTML in your pull request. See README.md for
more information.* -->
diff --git a/.github/actions/build-html/action.yml
b/.github/actions/build-html/action.yml
new file mode 100644
index 0000000000..76f7767416
--- /dev/null
+++ b/.github/actions/build-html/action.yml
@@ -0,0 +1,21 @@
+name: Build HTML
+
+description: Set up Ruby and run the Jekyll build.
+
+inputs:
+ ruby-version:
+ description: Ruby version to use.
+ required: false
+ default: "3.4"
+
+runs:
+ using: composite
+ steps:
+ - name: Set up Ruby and Bundler
+ uses: ruby/setup-ruby@v1
+ with:
+ ruby-version: ${{ inputs.ruby-version }}
+ bundler-cache: true
+ - name: Run documentation build
+ shell: bash
+ run: bundle exec jekyll build
diff --git a/.github/workflows/doc_gen.yml b/.github/workflows/doc_gen.yml
deleted file mode 100644
index 01478f9273..0000000000
--- a/.github/workflows/doc_gen.yml
+++ /dev/null
@@ -1,76 +0,0 @@
-name: Check document generation
-
-on:
- push:
- branches:
- - asf-site
- pull_request:
- branches:
- - asf-site
-
-jobs:
- lint:
- name: check whether all documentation was generated with the right Jekyll
version
- runs-on: ubuntu-24.04
- steps:
- - name: Free up disk space
- shell: 'script -q -e -c "bash {0}"'
- run: |
- echo "=================================="
- echo "Free up disk space on CI system"
- echo "=================================="
-
- echo "Listing top 100 largest packages (from large to small)"
- printf "Installed-Size\tPackage\n"
- dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n -r | head
-n 100
- df -h
-
- echo "Removing large packages"
- rm -rf /__t/CodeQL
- rm -rf /__t/go
- rm -rf /__t/node
-
- apt-get remove --purge -y '^aspnet.*' || true
- apt-get remove --purge -y '^dotnet-.*' || true
- apt-get remove --purge -y '^llvm-.*' || true
- apt-get remove --purge -y 'php.*' || true
- apt-get remove --purge -y '^mongodb-.*' || true
- apt-get remove --purge -y 'gfortran-11' || true
- apt-get remove --purge -y 'humanity-icon-theme' || true
- apt-get remove --purge -y 'nodejs-doc' || true
- apt-get remove --purge -y snapd google-chrome-stable
microsoft-edge-stable firefox || true
- apt-get remove --purge -y azure-cli google-cloud-sdk mono-devel
powershell libgl1-mesa-dri || true
- apt-get autoremove --purge -y
- apt-get clean
-
- df -h
- - name: Checkout Spark Website repository
- uses: actions/checkout@v2
- - name: Install dependencies for documentation generation
- run: |
- sudo apt-get update -y
- sudo apt-get install -y ruby ruby-dev
- sudo gem install bundler --version 2.4.19
- bundle install
- - name: Run documentation build
- run: |
- export LC_ALL=C.UTF-8
- export LANG=C.UTF-8
- OLD_IFS=$IFS
- IFS=
- GEN_ERRORS=$(bundle exec jekyll build 3>&2 2>&1 1>&3)
- if [ $(echo $GEN_ERRORS| grep -v -e '^$'| grep -c -v
"rubygems_integration") -ne 0 ]; then
- echo "Error during document generation:"
- echo $GEN_ERRORS
- exit 1
- fi
- IFS=$OLD_IFS
- CHANGED_FILE=( $(git ls-files --modified --other --exclude-standard
--directory | grep -v sitemap.xml | grep -v llms.txt) )
- if [ ${#CHANGED_FILE[@]} -ne 0 ]; then
- echo "Not all documentation was generated and/or not the right
Jekyll version was used! Modified / untracked files (excluding sitemap.xml):"
- echo ${CHANGED_FILE[*]}
- echo "Git diff (excluding sitemap.xml):"
- git diff -- . ':(exclude)site/sitemap.xml'
- exit 1
- fi
- shell: /bin/bash {0}
diff --git a/.github/workflows/html-build.yml b/.github/workflows/html-build.yml
new file mode 100644
index 0000000000..b94dfebe2b
--- /dev/null
+++ b/.github/workflows/html-build.yml
@@ -0,0 +1,16 @@
+name: Build HTML
+
+on:
+ pull_request:
+ branches:
+ - asf-site
+
+jobs:
+ build:
+ name: Build HTML
+ runs-on: ubuntu-24.04
+ steps:
+ - name: Checkout Spark Website repository
+ uses: actions/checkout@v7
+ - name: Build HTML
+ uses: ./.github/actions/build-html
diff --git a/.github/workflows/html-push.yml b/.github/workflows/html-push.yml
new file mode 100644
index 0000000000..15160e7b4c
--- /dev/null
+++ b/.github/workflows/html-push.yml
@@ -0,0 +1,43 @@
+name: Build and Push HTML
+
+on:
+ push:
+ branches:
+ - asf-site
+
+jobs:
+ commit:
+ name: Build and commit HTML to `asf-site`
+ # This condition is important. We don't want to trigger this job if the
last
+ # commit was created _by_ this job!
+ if: >-
+ !(
+ contains(github.event.head_commit.message, '[html]') &&
+ github.event.head_commit.author.name == 'github-actions[bot]'
+ )
+ # Not technically necessary, but helps avoid spurious failures if multiple
+ # commits are pushed in rapid succession.
+ concurrency:
+ group: html-push-${{ github.ref }}
+ cancel-in-progress: true
+ runs-on: ubuntu-24.04
+ permissions:
+ contents: write
+ steps:
+ - name: Checkout Spark Website repository
+ uses: actions/checkout@v7
+ - name: Build HTML
+ uses: ./.github/actions/build-html
+ - name: Commit and push generated HTML
+ run: |
+ git config user.name "github-actions[bot]"
+ git config user.email "github-actions[bot]@users.noreply.github.com"
+ # `-f` because we told git to otherwise ignore `site/`
+ git add -f site/
+ if git diff --cached --quiet; then
+ echo "No changes to commit."
+ else
+ COMMIT_TITLE=$(git log -1 --pretty=%s)
+ git commit -m "[html] $COMMIT_TITLE"
+ git push
+ fi
diff --git a/.gitignore b/.gitignore
index d0d5f1c267..90ef9e9205 100644
--- a/.gitignore
+++ b/.gitignore
@@ -4,4 +4,4 @@ target/
.jekyll-cache/
.jekyll-metadata
.local_ruby_bundle
-site/python
+site/
diff --git a/README.md b/README.md
index bef477a201..7b77603936 100644
--- a/README.md
+++ b/README.md
@@ -1,4 +1,12 @@
-## Generating the website HTML
+# Apache Spark Main Website
+
+This repository captures the main Apache Spark website located at
https://spark.apache.org. The programming docs are [in the main Spark repo][1],
not here.
+
+[1]: https://github.com/apache/spark/tree/master/docs
+
+To contribute changes, see [CONTRIBUTING](.github/CONTRIBUTING.md).
+
+## Generating the website HTML locally
In this directory you will find text files formatted using Markdown, with an
`.md` suffix.
@@ -28,22 +36,6 @@ of Spark from the Spark source repository and then copied to
the website under t
directory. See the instructions for building those in the readme in the Spark
project's `/docs` directory.
-## Rouge and Pygments
-
-We also use [Rouge](https://github.com/rouge-ruby/rouge) for syntax
highlighting in documentation Markdown pages.
-Its HTML output is compatible with CSS files designed for
[Pygments](https://pygments.org/).
-
-To mark a block of code in your Markdown to be syntax highlighted by `jekyll`
during the
-compile phase, use the following syntax:
-
- {% highlight scala %}
- // Your Scala code goes here, you can replace Scala with many other
- // supported languages too.
- {% endhighlight %}
-
-You probably don't need to install that unless you want to regenerate the
Pygments CSS file.
-It requires Python, and can be installed by running `sudo easy_install
Pygments`.
-
## Merge PR
-To merge pull request, use the `merge_pr.py` script which also squashes the
commits.
+To merge a pull request, use the `merge_pr.py` script. This script also
squashes the commits.
diff --git a/_config.yml b/_config.yml
index 98f05068c0..5e5be4c0c8 100644
--- a/_config.yml
+++ b/_config.yml
@@ -4,6 +4,6 @@ kramdown:
entity_output: symbol
permalink: none
destination: site
-exclude: ['README.md', 'content', 'LICENSE', 'merge_pr.py', 'Gemfile',
'Gemfile.lock']
+exclude: ['README.md', 'LICENSE', 'merge_pr.py', 'Gemfile', 'Gemfile.lock']
keep_files: ['docs', 'static', 'llms.txt']
url: https://spark.apache.org
diff --git a/content b/content
deleted file mode 120000
index d97e100637..0000000000
--- a/content
+++ /dev/null
@@ -1 +0,0 @@
-site
\ No newline at end of file
diff --git a/site/sitemap.xml b/site/sitemap.xml
index 3ef7e411a1..19f2ddf2f1 100644
--- a/site/sitemap.xml
+++ b/site/sitemap.xml
@@ -1163,7 +1163,10 @@
<loc>https://spark.apache.org/releases/spark-release-0-3.html</loc>
<changefreq>weekly</changefreq>
</url>
-
+<url>
+ <loc>https://spark.apache.org/</loc>
+ <changefreq>weekly</changefreq>
+</url>
<url>
<loc>https://spark.apache.org/404.html</loc>
<changefreq>weekly</changefreq>
@@ -1205,23 +1208,19 @@
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/history.html</loc>
- <changefreq>weekly</changefreq>
-</url>
-<url>
- <loc>https://spark.apache.org/improvement-proposals.html</loc>
+ <loc>https://spark.apache.org/graphx/</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/spark-connect/</loc>
+ <loc>https://spark.apache.org/history.html</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/pandas-on-spark/</loc>
+ <loc>https://spark.apache.org/improvement-proposals.html</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/graphx/</loc>
+ <loc>https://spark.apache.org/mailing-lists.html</loc>
<changefreq>weekly</changefreq>
</url>
<url>
@@ -1229,46 +1228,45 @@
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/streaming/</loc>
+ <loc>https://spark.apache.org/news/</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/news/</loc>
+ <loc>https://spark.apache.org/pandas-on-spark/</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/screencasts/</loc>
+ <loc>https://spark.apache.org/powered-by.html</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/sql/</loc>
+ <loc>https://spark.apache.org/release-process.html</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/</loc>
+ <loc>https://spark.apache.org/research.html</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/mailing-lists.html</loc>
+ <loc>https://spark.apache.org/screencasts/</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/powered-by.html</loc>
+ <loc>https://spark.apache.org/security.html</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/release-process.html</loc>
+ <loc>https://spark.apache.org/spark-connect/</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/research.html</loc>
+ <loc>https://spark.apache.org/sql/</loc>
<changefreq>weekly</changefreq>
</url>
<url>
- <loc>https://spark.apache.org/security.html</loc>
+ <loc>https://spark.apache.org/streaming/</loc>
<changefreq>weekly</changefreq>
</url>
-
<url>
<loc>https://spark.apache.org/third-party-projects.html</loc>
<changefreq>weekly</changefreq>
@@ -1281,5 +1279,4 @@
<loc>https://spark.apache.org/versioning-policy.html</loc>
<changefreq>weekly</changefreq>
</url>
-
</urlset>
diff --git a/sitemap.xml b/sitemap.xml
index 257359ef91..71fb8b093d 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -151,9 +151,18 @@ sitemap: false
<changefreq>weekly</changefreq>
</url>
{% endfor %}
-{% for page in site.pages %}{% if page.sitemap != false %}<url>
+{%- comment -%}
+Explicitly sort `site.pages` so that the order is consistent and we don't get
spurious git diffs.
+`site.posts` doesn't have this issue because it's already sorted.
+See: https://jekyllrb.com/docs/variables/#site-variables
+{%- endcomment -%}
+{%- assign sorted_pages = site.pages | sort: "url" -%}
+{%- for page in sorted_pages -%}
+{%- if page.sitemap != false -%}
+<url>
<loc>{{ site.url }}{{ page.url }}</loc>
<changefreq>weekly</changefreq>
-</url>{% endif %}
-{% endfor %}
+</url>
+{% endif %}
+{%- endfor -%}
</urlset>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
