This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-25525-918ab2c99844e4b6f79e17f275343b7c59c139f7 in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 99af44a5e620f899bb6796b9fbdcb6b00d33c6e1 Author: Kumar Ujjawal <[email protected]> AuthorDate: Sun Sep 20 14:32:05 2026 +0000 dev: include generated config and function docs in the local lint suite (#25525) ## Which issue does this PR close? - Part of #21048. ## Rationale for this change `dev/rust_lint.sh` does not check the generated `configs.md` and function pages. A contributor finds a stale page only in the `config-docs-check` job. That job also regenerates the tracked pages in place and then runs `git diff`, so the same check cannot run in a working tree with other changes. ## What changes are included in this PR? - `ci/scripts/check_config_function_docs.sh` regenerates the four pages into per-run scratch directories, compares each with its working file, and prints a diff for each stale page. Check mode never rewrites a page. `--write` replaces the four pages, with the same clean-tree rule as `rust_fmt.sh`. - `dev/update_config_docs.sh` and `dev/update_function_docs.sh` gain `--output-dir DIR`, so the checker reuses the existing generators and headers. Without the option, both scripts behave as before. - `dev/rust_lint.sh` runs the checker as a write step. - The `config-docs-check` job runs the checker and keeps its `git diff --exit-code` guard. The job identity and setup do not change. - `docs/source/contributor-guide/testing.md` documents the check and the update command. ## What is the testing strategy for this PR? - In a clean clone with the real generators: the no-argument scripts and `--output-dir` produce byte-identical pages, the four committed pages pass, a committed stale page fails with its diff, `--write` restores it byte for byte, the dirty-tree gate refuses, and staged edits survive check mode. No lockfile changed. - A fixture with stub `cargo` and `npx` covered each page stale on its own, generator and formatter failures in both modes, cleanup after failure and interruption, concurrent runs, missing `npx`, comparison errors, and argument errors. No page changed in any failure case. - The parsed workflow matches `main` except the replaced steps. The full `./dev/rust_lint.sh` passes. ## Are there any user-facing changes? No. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- .github/workflows/rust.yml | 10 +- ci/scripts/check_generated_docs.sh | 152 +++++++++++++++++++++++++++++++ dev/rust_lint.sh | 1 + dev/update_config_docs.sh | 34 ++++++- dev/update_function_docs.sh | 38 +++++++- docs/source/contributor-guide/testing.md | 16 ++++ 6 files changed, 239 insertions(+), 12 deletions(-) diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index a2314098f8..73d1ff3de5 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -855,15 +855,9 @@ jobs: - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: node-version: "20" - - name: Check if configs.md has been modified + - name: Check generated config and function docs run: | - # If you encounter an error, run './dev/update_config_docs.sh' and commit - ./dev/update_config_docs.sh - git diff --exit-code - - name: Check if any of the ***_functions.md has been modified - run: | - # If you encounter an error, run './dev/update_function_docs.sh' and commit - ./dev/update_function_docs.sh + ./ci/scripts/check_generated_docs.sh git diff --exit-code # This job ensures `datafusion-examples/README.md` stays in sync with the source code: diff --git a/ci/scripts/check_generated_docs.sh b/ci/scripts/check_generated_docs.sh new file mode 100755 index 0000000000..fc9d31c745 --- /dev/null +++ b/ci/scripts/check_generated_docs.sh @@ -0,0 +1,152 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Regenerates the configuration and function documentation pages and checks +# that the committed pages match, the same way the "check configs.md and +# ***_functions.md is up-to-date" job does. With `--write`, replaces the pages +# with the generated ones. + +set -euo pipefail + +SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT_DIR="$(cd "${SCRIPT_DIR}/../.." && pwd)" + +source "${SCRIPT_DIR}/utils/git.sh" + +CONFIG_DOCS_DIR="docs/source/user-guide" +FUNCTION_DOCS_DIR="docs/source/user-guide/sql" + +MODE="check" +ALLOW_DIRTY=0 + +usage() { + cat >&2 <<USAGE +Usage: $0 [--write] [--allow-dirty] + +Checks that docs/source/user-guide/configs.md and the aggregate, scalar, and window +function pages under docs/source/user-guide/sql/ match the output of +dev/update_config_docs.sh and dev/update_function_docs.sh. +--write Replace the pages with the generated ones (requires a clean git worktree, no uncommitted changes). +--allow-dirty Allow \`--write\` to run even when the git worktree has uncommitted changes. +USAGE + exit 1 +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --write) + MODE="write" + ;; + --allow-dirty) + ALLOW_DIRTY=1 + ;; + -h|--help) + usage + ;; + *) + usage + ;; + esac + shift +done + +cd "${ROOT_DIR}" + +if [[ "$MODE" == "write" && $ALLOW_DIRTY -eq 0 ]]; then + require_clean_work_tree "$SCRIPT_NAME" || exit 1 +fi + +if ! command -v npx >/dev/null 2>&1; then + echo "[${SCRIPT_NAME}] npx is required to run the prettier check. Install Node.js (e.g., brew install node) and re-run." >&2 + exit 1 +fi + +# One scratch directory beneath each documentation directory, so Prettier finds +# the same configuration as for the committed pages. +SCRATCH_DIRS=() +cleanup() { + rm -rf ${SCRATCH_DIRS[@]+"${SCRATCH_DIRS[@]}"} +} +trap cleanup EXIT +trap 'exit 130' INT +trap 'exit 143' TERM +CONFIG_SCRATCH="$(mktemp -d "${ROOT_DIR}/${CONFIG_DOCS_DIR}/.config-docs-check.XXXXXX")" +SCRATCH_DIRS+=("${CONFIG_SCRATCH}") +FUNCTION_SCRATCH="$(mktemp -d "${ROOT_DIR}/${FUNCTION_DOCS_DIR}/.function-docs-check.XXXXXX")" +SCRATCH_DIRS+=("${FUNCTION_SCRATCH}") + +./dev/update_config_docs.sh --output-dir "${CONFIG_SCRATCH}" +./dev/update_function_docs.sh --output-dir "${FUNCTION_SCRATCH}" + +GENERATED=( + "${CONFIG_SCRATCH}/configs.md" + "${FUNCTION_SCRATCH}/aggregate_functions.md" + "${FUNCTION_SCRATCH}/scalar_functions.md" + "${FUNCTION_SCRATCH}/window_functions.md" +) +COMMITTED=( + "${CONFIG_DOCS_DIR}/configs.md" + "${FUNCTION_DOCS_DIR}/aggregate_functions.md" + "${FUNCTION_DOCS_DIR}/scalar_functions.md" + "${FUNCTION_DOCS_DIR}/window_functions.md" +) + +if [[ "$MODE" == "write" ]]; then + for i in "${!COMMITTED[@]}"; do + cp "${GENERATED[$i]}" "${COMMITTED[$i]}" || { + echo "[${SCRIPT_NAME}] failed to copy the generated page to ${COMMITTED[$i]}" >&2 + exit 1 + } + echo "✅ ${COMMITTED[$i]} updated." + done + exit 0 +fi + +stale_count=0 +for i in "${!COMMITTED[@]}"; do + diff_status=0 + diff -u -L "${COMMITTED[$i]} (committed)" -L "${COMMITTED[$i]} (generated)" \ + "${COMMITTED[$i]}" "${GENERATED[$i]}" > "${GENERATED[$i]}.diff" || diff_status=$? + case "${diff_status}" in + 0) + echo "✅ ${COMMITTED[$i]} is up-to-date." + ;; + 1) + stale_count=$((stale_count + 1)) + echo "" + echo "❌ ${COMMITTED[$i]} is out of date." + echo "------------------------------------------------------------" + cat "${GENERATED[$i]}.diff" + echo "------------------------------------------------------------" + ;; + *) + echo "❌ diff exited with status ${diff_status} while comparing ${COMMITTED[$i]}; no comparison result." >&2 + exit "${diff_status}" + ;; + esac +done + +if [[ ${stale_count} -gt 0 ]]; then + echo "" + echo "${stale_count} generated page(s) out of date. To update them, run:" + echo "" + echo " ./ci/scripts/check_generated_docs.sh --write" + exit 1 +fi diff --git a/dev/rust_lint.sh b/dev/rust_lint.sh index 77bed06457..74d5fc16d7 100755 --- a/dev/rust_lint.sh +++ b/dev/rust_lint.sh @@ -130,6 +130,7 @@ declare -a WRITE_STEPS=( "ci/scripts/typos_check.sh|true" "ci/scripts/doc_prettier_check.sh|true" "ci/scripts/check_examples_docs.sh|true" + "ci/scripts/check_generated_docs.sh|true" ) declare -a READONLY_STEPS=( diff --git a/dev/update_config_docs.sh b/dev/update_config_docs.sh index df40c65210..5bba39625f 100755 --- a/dev/update_config_docs.sh +++ b/dev/update_config_docs.sh @@ -26,7 +26,39 @@ cd "${ROOT_DIR}" # Load centralized tool versions source "${ROOT_DIR}/ci/scripts/utils/tool_versions.sh" -TARGET_FILE="docs/source/user-guide/configs.md" +# `--output-dir DIR` writes the generated page into DIR instead of +# docs/source/user-guide. A relative DIR is taken from the repository root. +OUTPUT_DIR="docs/source/user-guide" + +usage() { + cat >&2 <<USAGE +Usage: $0 [--output-dir DIR] + +Regenerates docs/source/user-guide/configs.md. +--output-dir DIR Write the generated page into DIR instead of docs/source/user-guide (relative to the repository root). +USAGE + exit 1 +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --output-dir) + [[ $# -ge 2 ]] || usage + OUTPUT_DIR="$2" + shift + ;; + -h|--help) + usage + ;; + *) + usage + ;; + esac + shift +done + +mkdir -p "${OUTPUT_DIR}" +TARGET_FILE="${OUTPUT_DIR}/configs.md" PRINT_CONFIG_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --features docs_generation --bin print_config_docs" PRINT_RUNTIME_CONFIG_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --features docs_generation --bin print_runtime_config_docs" diff --git a/dev/update_function_docs.sh b/dev/update_function_docs.sh index 005122c33e..9bbe857a53 100755 --- a/dev/update_function_docs.sh +++ b/dev/update_function_docs.sh @@ -26,7 +26,39 @@ cd "${ROOT_DIR}" # Load centralized tool versions source "${ROOT_DIR}/ci/scripts/utils/tool_versions.sh" -TARGET_FILE="docs/source/user-guide/sql/aggregate_functions.md" +# `--output-dir DIR` writes the generated pages into DIR instead of +# docs/source/user-guide/sql. A relative DIR is taken from the repository root. +OUTPUT_DIR="docs/source/user-guide/sql" + +usage() { + cat >&2 <<USAGE +Usage: $0 [--output-dir DIR] + +Regenerates the aggregate, scalar, and window function pages under docs/source/user-guide/sql. +--output-dir DIR Write the generated pages into DIR instead of docs/source/user-guide/sql (relative to the repository root). +USAGE + exit 1 +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --output-dir) + [[ $# -ge 2 ]] || usage + OUTPUT_DIR="$2" + shift + ;; + -h|--help) + usage + ;; + *) + usage + ;; + esac + shift +done + +mkdir -p "${OUTPUT_DIR}" +TARGET_FILE="${OUTPUT_DIR}/aggregate_functions.md" PRINT_AGGREGATE_FUNCTION_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --features docs_generation --bin print_functions_docs -- aggregate" echo "Inserting header" @@ -120,7 +152,7 @@ npx "prettier@${PRETTIER_VERSION}" --write "$TARGET_FILE" echo "'$TARGET_FILE' successfully updated!" -TARGET_FILE="docs/source/user-guide/sql/scalar_functions.md" +TARGET_FILE="${OUTPUT_DIR}/scalar_functions.md" PRINT_SCALAR_FUNCTION_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --features docs_generation --bin print_functions_docs -- scalar" echo "Inserting header" @@ -164,7 +196,7 @@ npx "prettier@${PRETTIER_VERSION}" --write "$TARGET_FILE" echo "'$TARGET_FILE' successfully updated!" -TARGET_FILE="docs/source/user-guide/sql/window_functions.md" +TARGET_FILE="${OUTPUT_DIR}/window_functions.md" PRINT_WINDOW_FUNCTION_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --features docs_generation --bin print_functions_docs -- window" echo "Inserting header" diff --git a/docs/source/contributor-guide/testing.md b/docs/source/contributor-guide/testing.md index 6983c84da3..89dc7969b3 100644 --- a/docs/source/contributor-guide/testing.md +++ b/docs/source/contributor-guide/testing.md @@ -304,6 +304,22 @@ the README: ./ci/scripts/check_examples_docs.sh --write ``` +## Config and Function Docs Check + +[`configs.md`], [`aggregate_functions.md`], [`scalar_functions.md`], and +[`window_functions.md`] are generated by [`dev/update_config_docs.sh`] and +[`dev/update_function_docs.sh`]. To check that they are up to date, run the +[`ci/scripts/check_generated_docs.sh`] script, which also runs as part of +`./dev/rust_lint.sh`. Run it with `--write` to update the pages. + +[`configs.md`]: https://github.com/apache/datafusion/blob/main/docs/source/user-guide/configs.md +[`aggregate_functions.md`]: https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/aggregate_functions.md +[`scalar_functions.md`]: https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/scalar_functions.md +[`window_functions.md`]: https://github.com/apache/datafusion/blob/main/docs/source/user-guide/sql/window_functions.md +[`dev/update_config_docs.sh`]: https://github.com/apache/datafusion/blob/main/dev/update_config_docs.sh +[`dev/update_function_docs.sh`]: https://github.com/apache/datafusion/blob/main/dev/update_function_docs.sh +[`ci/scripts/check_generated_docs.sh`]: https://github.com/apache/datafusion/blob/main/ci/scripts/check_generated_docs.sh + ## Benchmarks ### Criterion Benchmarks --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
