alamb commented on code in PR #27199: URL: https://github.com/apache/datafusion/pull/27199#discussion_r4501495053
########## docs/source/contributor-guide/testing_non_functional.md: ########## @@ -0,0 +1,287 @@ +<!-- +SPDX-License-Identifier: Apache-2.0 +SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership +SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements +--> + +# Non-Functional Tests + +This page covers checks for CI workflows, documentation, repository maintenance, and benchmarks. For tests of DataFusion's query behavior, see [Testing](testing.md). + +## CI Workflow Tests + +The Rust workflow validator and fork status bridge have regression tests. +These tests cover skip conditions, fork run recovery, polling frequency, and +concurrency groups for fork pushes and merge queue runs. +They run in the `asf-yaml-check` CI job and in `./dev/rust_lint.sh`. +Run them locally with Python, PyYAML, and Node.js 24: + +```shell +python3 ci/scripts/test_rust_workflow_skips.py +node --test ci/scripts/test_fork_ci_status.cjs +``` + +The validator tests remove guards and add an invalid job dependency in memory. Review Comment: I also think we could remove this stuff about implementation details of how the validator works ########## docs/source/contributor-guide/testing_non_functional.md: ########## @@ -0,0 +1,287 @@ +<!-- +SPDX-License-Identifier: Apache-2.0 +SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership +SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements +--> + +# Non-Functional Tests Review Comment: I normally think of non functional tests as those for performance, etc. I wonder if "Testing Infrastructure" or something would be a clearer name ########## docs/source/contributor-guide/testing_non_functional.md: ########## @@ -0,0 +1,287 @@ +<!-- +SPDX-License-Identifier: Apache-2.0 +SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership +SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements +--> + +# Non-Functional Tests + +This page covers checks for CI workflows, documentation, repository maintenance, and benchmarks. For tests of DataFusion's query behavior, see [Testing](testing.md). + +## CI Workflow Tests + +The Rust workflow validator and fork status bridge have regression tests. +These tests cover skip conditions, fork run recovery, polling frequency, and +concurrency groups for fork pushes and merge queue runs. +They run in the `asf-yaml-check` CI job and in `./dev/rust_lint.sh`. +Run them locally with Python, PyYAML, and Node.js 24: + +```shell +python3 ci/scripts/test_rust_workflow_skips.py +node --test ci/scripts/test_fork_ci_status.cjs +``` + +The validator tests remove guards and add an invalid job dependency in memory. +The bridge tests use stubbed GitHub API responses and make no network requests. + +## Documentation Examples + +We use Rust [doctest] to verify examples from the documentation are correct and +up-to-date. These tests are run as part of our CI and you can run them +locally with the following command: + +```shell +cargo test --doc +``` + +### API Documentation Examples + +As with other Rust projects, examples in doc comments in `.rs` files are +automatically checked to ensure they work and evolve along with the code. + +Prefix a line with `# ` (a `#` followed by a space) to keep it compiled and +run, but hidden from the rendered documentation. This is most useful for +`use` statements, since readers care about the API being demonstrated, not +its import boilerplate: + +````rust +/// ``` +/// # use std::sync::Arc; +/// let value = Arc::new(42); +/// ``` +```` + +Only the `let value = ...` line renders. + +### User Guide Documentation + +Rust example code from the user guide (anything marked with \`\`\`rust) is also +tested in the same way using the [doc_comment] crate. See the end of +[core/src/lib.rs] for more details. + +[doctest]: https://doc.rust-lang.org/rust-by-example/testing/doc_testing.html +[doc_comment]: https://docs.rs/doc-comment/latest/doc_comment +[core/src/lib.rs]: https://github.com/apache/datafusion/blob/main/datafusion/core/src/lib.rs#L583 + +## Documentation Link Checks + +`./dev/rust_lint.sh` runs the internal markdown link check. If `lychee` is +missing, the script installs the version pinned in +`ci/scripts/utils/tool_versions.sh`. It uses an existing installation as is, +even if the version differs from the pin. + +To run the check on its own: + +```shell +source ci/scripts/utils/tool_versions.sh +cargo install lychee --locked --version "${LYCHEE_VERSION}" +bash ci/scripts/markdown_link_check.sh +``` + +Notes: + +- The script is run with `bash` and is compatible with the default Bash on macOS (no `mapfile` dependency). +- The CI configuration currently checks internal markdown links only. External `http(s)` and `mailto` links are excluded to avoid flaky failures. +- The check only reports broken links. `./dev/rust_lint.sh --write` does not change them. + +When a link is broken, lychee prints the file and URL/path that failed. For example: + +```text +[docs/source/user-guide/cli/overview.md]: + [ERROR] file:///.../docs/source/user-guide/cli/missing-page.md | Cannot find file: File not found. Check if file exists and path is correct +``` + +Rust doc comments are validated by rustdoc in CI and can be checked locally with: + +```shell +bash ci/scripts/rust_docs.sh +``` + +## ASF Status Check Validation + +`ci/scripts/check_asf_yaml_status_checks.py` checks that every required status +check in `.asf.yaml` matches a job in `.github/workflows`. +`ci/scripts/check_rust_workflow_skips.py` checks that `rust.yml` skips only its +listed jobs on pushes to `main` and on pull requests from forks. +`./dev/rust_lint.sh` runs both and needs `python3` with [PyYAML]. The [uv] +workspace provides both: + +```shell +uv run ./dev/rust_lint.sh +``` + +To run the checks on their own: + +```shell +uv run python3 ci/scripts/check_asf_yaml_status_checks.py +uv run python3 ci/scripts/check_rust_workflow_skips.py +``` + +[pyyaml]: https://pypi.org/project/PyYAML/ +[uv]: https://docs.astral.sh/uv/ + +## Security Audit + +`ci/scripts/security_audit.sh` runs `cargo audit` on the root `Cargo.lock` with +the advisory exceptions that CI uses. `./dev/rust_lint.sh` runs it and installs +[cargo-audit] if it is missing. To run the audit on its own: + +```shell +./ci/scripts/security_audit.sh +``` + +The audit fetches the RustSec advisory database. A new advisory or a different +`cargo-audit` version can change the result without any change to the repository. + +[cargo-audit]: https://github.com/rustsec/rustsec/blob/main/cargo-audit/README.md + +## Large File Check + +`ci/scripts/check_large_files.sh` fails if any file committed between a base +ref and a head ref is larger than 1.5 MB, the same check the "Large files PR +check" workflow runs on pull requests. `./dev/rust_lint.sh` runs it against +the merge base of `HEAD` and `main` on the remote that points at +`apache/datafusion`, or on `origin` when there is no such remote. To run the +check on its own, or against a different range: Review Comment: The detail of how the script works I think should go in the script and here just a pointer to it ```suggestion check" workflow runs on pull requests. To run the check on its own, or against a different range: ``` ########## docs/source/contributor-guide/testing_non_functional.md: ########## @@ -0,0 +1,287 @@ +<!-- +SPDX-License-Identifier: Apache-2.0 +SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership +SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements +--> + +# Non-Functional Tests + +This page covers checks for CI workflows, documentation, repository maintenance, and benchmarks. For tests of DataFusion's query behavior, see [Testing](testing.md). + +## CI Workflow Tests + +The Rust workflow validator and fork status bridge have regression tests. +These tests cover skip conditions, fork run recovery, polling frequency, and +concurrency groups for fork pushes and merge queue runs. +They run in the `asf-yaml-check` CI job and in `./dev/rust_lint.sh`. +Run them locally with Python, PyYAML, and Node.js 24: + +```shell +python3 ci/scripts/test_rust_workflow_skips.py +node --test ci/scripts/test_fork_ci_status.cjs +``` + +The validator tests remove guards and add an invalid job dependency in memory. +The bridge tests use stubbed GitHub API responses and make no network requests. + +## Documentation Examples + +We use Rust [doctest] to verify examples from the documentation are correct and +up-to-date. These tests are run as part of our CI and you can run them +locally with the following command: + +```shell +cargo test --doc +``` + +### API Documentation Examples + +As with other Rust projects, examples in doc comments in `.rs` files are +automatically checked to ensure they work and evolve along with the code. + +Prefix a line with `# ` (a `#` followed by a space) to keep it compiled and +run, but hidden from the rendered documentation. This is most useful for +`use` statements, since readers care about the API being demonstrated, not +its import boilerplate: + +````rust +/// ``` +/// # use std::sync::Arc; +/// let value = Arc::new(42); +/// ``` +```` + +Only the `let value = ...` line renders. + +### User Guide Documentation + +Rust example code from the user guide (anything marked with \`\`\`rust) is also +tested in the same way using the [doc_comment] crate. See the end of +[core/src/lib.rs] for more details. + +[doctest]: https://doc.rust-lang.org/rust-by-example/testing/doc_testing.html +[doc_comment]: https://docs.rs/doc-comment/latest/doc_comment +[core/src/lib.rs]: https://github.com/apache/datafusion/blob/main/datafusion/core/src/lib.rs#L583 + +## Documentation Link Checks + +`./dev/rust_lint.sh` runs the internal markdown link check. If `lychee` is +missing, the script installs the version pinned in +`ci/scripts/utils/tool_versions.sh`. It uses an existing installation as is, +even if the version differs from the pin. + +To run the check on its own: + +```shell +source ci/scripts/utils/tool_versions.sh +cargo install lychee --locked --version "${LYCHEE_VERSION}" +bash ci/scripts/markdown_link_check.sh +``` + +Notes: + +- The script is run with `bash` and is compatible with the default Bash on macOS (no `mapfile` dependency). +- The CI configuration currently checks internal markdown links only. External `http(s)` and `mailto` links are excluded to avoid flaky failures. +- The check only reports broken links. `./dev/rust_lint.sh --write` does not change them. + +When a link is broken, lychee prints the file and URL/path that failed. For example: + +```text +[docs/source/user-guide/cli/overview.md]: + [ERROR] file:///.../docs/source/user-guide/cli/missing-page.md | Cannot find file: File not found. Check if file exists and path is correct +``` + +Rust doc comments are validated by rustdoc in CI and can be checked locally with: + +```shell +bash ci/scripts/rust_docs.sh +``` + +## ASF Status Check Validation + +`ci/scripts/check_asf_yaml_status_checks.py` checks that every required status +check in `.asf.yaml` matches a job in `.github/workflows`. +`ci/scripts/check_rust_workflow_skips.py` checks that `rust.yml` skips only its +listed jobs on pushes to `main` and on pull requests from forks. +`./dev/rust_lint.sh` runs both and needs `python3` with [PyYAML]. The [uv] +workspace provides both: + +```shell +uv run ./dev/rust_lint.sh +``` + +To run the checks on their own: + +```shell +uv run python3 ci/scripts/check_asf_yaml_status_checks.py +uv run python3 ci/scripts/check_rust_workflow_skips.py +``` + +[pyyaml]: https://pypi.org/project/PyYAML/ +[uv]: https://docs.astral.sh/uv/ + +## Security Audit + +`ci/scripts/security_audit.sh` runs `cargo audit` on the root `Cargo.lock` with +the advisory exceptions that CI uses. `./dev/rust_lint.sh` runs it and installs +[cargo-audit] if it is missing. To run the audit on its own: + +```shell +./ci/scripts/security_audit.sh +``` + +The audit fetches the RustSec advisory database. A new advisory or a different +`cargo-audit` version can change the result without any change to the repository. + +[cargo-audit]: https://github.com/rustsec/rustsec/blob/main/cargo-audit/README.md + +## Large File Check + +`ci/scripts/check_large_files.sh` fails if any file committed between a base +ref and a head ref is larger than 1.5 MB, the same check the "Large files PR +check" workflow runs on pull requests. `./dev/rust_lint.sh` runs it against +the merge base of `HEAD` and `main` on the remote that points at +`apache/datafusion`, or on `origin` when there is no such remote. To run the +check on its own, or against a different range: + +```shell +./ci/scripts/check_large_files.sh +./ci/scripts/check_large_files.sh --base upstream/main --head my-branch +``` + +Only committed files are checked. Commit a change before running it. + +## Dependency Checks + +CI runs two dependency checks, and `./dev/rust_lint.sh` runs both: + +- `ci/scripts/check_circular_dependencies.sh` fails on dependency cycles + between DataFusion crates, including one through a dev-dependency, which + Cargo allows when building but which prevents publishing the crates. It runs + `cargo publish --workspace --dry-run --no-verify`, which packages every crate + that does not set `publish = false` without building or uploading anything, + so it also fails on any other problem that would stop the crates from being + published to crates.io. It reads the crates.io index and accepts uncommitted + changes. +- `ci/scripts/check_unused_dependencies.sh` runs `cargo machete --with-metadata` + from the repository root. The lint suite installs [cargo-machete] with the + version in `ci/scripts/utils/tool_versions.sh` if it is missing. + +To run either check on its own: + +```shell +./ci/scripts/check_circular_dependencies.sh +./ci/scripts/check_unused_dependencies.sh +``` + +[cargo-machete]: https://github.com/bnjbvr/cargo-machete + +## Examples README Check + +`datafusion-examples/README.md` is generated from the documentation comments in +`datafusion-examples/examples/<group>/main.rs`. `ci/scripts/check_examples_docs.sh` +regenerates it and fails if the committed file differs. `./dev/rust_lint.sh` +runs it and needs `cargo` and `npx`. To run the check on its own, or to update +the README: Review Comment: ```suggestion `datafusion-examples/README.md` is generated from the documentation comments in `datafusion-examples/examples/<group>/main.rs`. `ci/scripts/check_examples_docs.sh` regenerates it and fails if the committed file differs. To run the check on its own, or to update the README: ``` ########## docs/source/contributor-guide/testing_non_functional.md: ########## @@ -0,0 +1,287 @@ +<!-- +SPDX-License-Identifier: Apache-2.0 +SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership +SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements +--> + +# Non-Functional Tests + +This page covers checks for CI workflows, documentation, repository maintenance, and benchmarks. For tests of DataFusion's query behavior, see [Testing](testing.md). + +## CI Workflow Tests + +The Rust workflow validator and fork status bridge have regression tests. +These tests cover skip conditions, fork run recovery, polling frequency, and +concurrency groups for fork pushes and merge queue runs. +They run in the `asf-yaml-check` CI job and in `./dev/rust_lint.sh`. +Run them locally with Python, PyYAML, and Node.js 24: Review Comment: "Run them locally with Python, PyYAML, and Node.js 24:" --> "Run them locally" All the rest of that is irrelevant detail I hink ########## docs/source/contributor-guide/testing_non_functional.md: ########## @@ -0,0 +1,287 @@ +<!-- +SPDX-License-Identifier: Apache-2.0 +SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership +SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements +--> + +# Non-Functional Tests + +This page covers checks for CI workflows, documentation, repository maintenance, and benchmarks. For tests of DataFusion's query behavior, see [Testing](testing.md). + +## CI Workflow Tests + +The Rust workflow validator and fork status bridge have regression tests. +These tests cover skip conditions, fork run recovery, polling frequency, and +concurrency groups for fork pushes and merge queue runs. +They run in the `asf-yaml-check` CI job and in `./dev/rust_lint.sh`. +Run them locally with Python, PyYAML, and Node.js 24: + +```shell +python3 ci/scripts/test_rust_workflow_skips.py +node --test ci/scripts/test_fork_ci_status.cjs +``` + +The validator tests remove guards and add an invalid job dependency in memory. +The bridge tests use stubbed GitHub API responses and make no network requests. + +## Documentation Examples + +We use Rust [doctest] to verify examples from the documentation are correct and +up-to-date. These tests are run as part of our CI and you can run them +locally with the following command: + +```shell +cargo test --doc +``` + +### API Documentation Examples + +As with other Rust projects, examples in doc comments in `.rs` files are +automatically checked to ensure they work and evolve along with the code. + +Prefix a line with `# ` (a `#` followed by a space) to keep it compiled and +run, but hidden from the rendered documentation. This is most useful for +`use` statements, since readers care about the API being demonstrated, not +its import boilerplate: + +````rust +/// ``` +/// # use std::sync::Arc; +/// let value = Arc::new(42); +/// ``` +```` + +Only the `let value = ...` line renders. + +### User Guide Documentation + +Rust example code from the user guide (anything marked with \`\`\`rust) is also +tested in the same way using the [doc_comment] crate. See the end of +[core/src/lib.rs] for more details. + +[doctest]: https://doc.rust-lang.org/rust-by-example/testing/doc_testing.html +[doc_comment]: https://docs.rs/doc-comment/latest/doc_comment +[core/src/lib.rs]: https://github.com/apache/datafusion/blob/main/datafusion/core/src/lib.rs#L583 + +## Documentation Link Checks + +`./dev/rust_lint.sh` runs the internal markdown link check. If `lychee` is +missing, the script installs the version pinned in +`ci/scripts/utils/tool_versions.sh`. It uses an existing installation as is, +even if the version differs from the pin. + +To run the check on its own: + +```shell +source ci/scripts/utils/tool_versions.sh +cargo install lychee --locked --version "${LYCHEE_VERSION}" +bash ci/scripts/markdown_link_check.sh +``` + +Notes: + +- The script is run with `bash` and is compatible with the default Bash on macOS (no `mapfile` dependency). +- The CI configuration currently checks internal markdown links only. External `http(s)` and `mailto` links are excluded to avoid flaky failures. +- The check only reports broken links. `./dev/rust_lint.sh --write` does not change them. + +When a link is broken, lychee prints the file and URL/path that failed. For example: + +```text +[docs/source/user-guide/cli/overview.md]: + [ERROR] file:///.../docs/source/user-guide/cli/missing-page.md | Cannot find file: File not found. Check if file exists and path is correct +``` + +Rust doc comments are validated by rustdoc in CI and can be checked locally with: + +```shell +bash ci/scripts/rust_docs.sh +``` + +## ASF Status Check Validation + +`ci/scripts/check_asf_yaml_status_checks.py` checks that every required status +check in `.asf.yaml` matches a job in `.github/workflows`. +`ci/scripts/check_rust_workflow_skips.py` checks that `rust.yml` skips only its +listed jobs on pushes to `main` and on pull requests from forks. +`./dev/rust_lint.sh` runs both and needs `python3` with [PyYAML]. The [uv] +workspace provides both: + +```shell +uv run ./dev/rust_lint.sh +``` + +To run the checks on their own: + +```shell +uv run python3 ci/scripts/check_asf_yaml_status_checks.py +uv run python3 ci/scripts/check_rust_workflow_skips.py +``` + +[pyyaml]: https://pypi.org/project/PyYAML/ +[uv]: https://docs.astral.sh/uv/ + +## Security Audit + +`ci/scripts/security_audit.sh` runs `cargo audit` on the root `Cargo.lock` with +the advisory exceptions that CI uses. `./dev/rust_lint.sh` runs it and installs +[cargo-audit] if it is missing. To run the audit on its own: + +```shell +./ci/scripts/security_audit.sh +``` + +The audit fetches the RustSec advisory database. A new advisory or a different +`cargo-audit` version can change the result without any change to the repository. + +[cargo-audit]: https://github.com/rustsec/rustsec/blob/main/cargo-audit/README.md + +## Large File Check + +`ci/scripts/check_large_files.sh` fails if any file committed between a base +ref and a head ref is larger than 1.5 MB, the same check the "Large files PR +check" workflow runs on pull requests. `./dev/rust_lint.sh` runs it against +the merge base of `HEAD` and `main` on the remote that points at +`apache/datafusion`, or on `origin` when there is no such remote. To run the +check on its own, or against a different range: + +```shell +./ci/scripts/check_large_files.sh +./ci/scripts/check_large_files.sh --base upstream/main --head my-branch +``` + +Only committed files are checked. Commit a change before running it. + +## Dependency Checks + +CI runs two dependency checks, and `./dev/rust_lint.sh` runs both: + +- `ci/scripts/check_circular_dependencies.sh` fails on dependency cycles + between DataFusion crates, including one through a dev-dependency, which + Cargo allows when building but which prevents publishing the crates. It runs + `cargo publish --workspace --dry-run --no-verify`, which packages every crate + that does not set `publish = false` without building or uploading anything, + so it also fails on any other problem that would stop the crates from being + published to crates.io. It reads the crates.io index and accepts uncommitted + changes. +- `ci/scripts/check_unused_dependencies.sh` runs `cargo machete --with-metadata` + from the repository root. The lint suite installs [cargo-machete] with the + version in `ci/scripts/utils/tool_versions.sh` if it is missing. Review Comment: Again I think we can make this much easier to read by avoiding irrelevant implementation detail ```suggestion - `ci/scripts/check_circular_dependencies.sh` fails on dependency cycles between DataFusion crates, including one through a dev-dependency, which Cargo allows when building but which prevents publishing the crates. - `ci/scripts/check_unused_dependencies.sh` runs `cargo machete --with-metadata` from the repository root. ``` ########## docs/source/contributor-guide/testing_non_functional.md: ########## @@ -0,0 +1,287 @@ +<!-- +SPDX-License-Identifier: Apache-2.0 +SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership +SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements +--> + +# Non-Functional Tests + +This page covers checks for CI workflows, documentation, repository maintenance, and benchmarks. For tests of DataFusion's query behavior, see [Testing](testing.md). + +## CI Workflow Tests + +The Rust workflow validator and fork status bridge have regression tests. +These tests cover skip conditions, fork run recovery, polling frequency, and +concurrency groups for fork pushes and merge queue runs. +They run in the `asf-yaml-check` CI job and in `./dev/rust_lint.sh`. +Run them locally with Python, PyYAML, and Node.js 24: + +```shell +python3 ci/scripts/test_rust_workflow_skips.py +node --test ci/scripts/test_fork_ci_status.cjs +``` + +The validator tests remove guards and add an invalid job dependency in memory. +The bridge tests use stubbed GitHub API responses and make no network requests. + +## Documentation Examples + +We use Rust [doctest] to verify examples from the documentation are correct and +up-to-date. These tests are run as part of our CI and you can run them +locally with the following command: + +```shell +cargo test --doc +``` + +### API Documentation Examples + +As with other Rust projects, examples in doc comments in `.rs` files are +automatically checked to ensure they work and evolve along with the code. + +Prefix a line with `# ` (a `#` followed by a space) to keep it compiled and +run, but hidden from the rendered documentation. This is most useful for +`use` statements, since readers care about the API being demonstrated, not +its import boilerplate: + +````rust +/// ``` +/// # use std::sync::Arc; +/// let value = Arc::new(42); +/// ``` +```` + +Only the `let value = ...` line renders. + +### User Guide Documentation + +Rust example code from the user guide (anything marked with \`\`\`rust) is also +tested in the same way using the [doc_comment] crate. See the end of +[core/src/lib.rs] for more details. + +[doctest]: https://doc.rust-lang.org/rust-by-example/testing/doc_testing.html +[doc_comment]: https://docs.rs/doc-comment/latest/doc_comment +[core/src/lib.rs]: https://github.com/apache/datafusion/blob/main/datafusion/core/src/lib.rs#L583 + +## Documentation Link Checks + +`./dev/rust_lint.sh` runs the internal markdown link check. If `lychee` is +missing, the script installs the version pinned in +`ci/scripts/utils/tool_versions.sh`. It uses an existing installation as is, +even if the version differs from the pin. + +To run the check on its own: + +```shell +source ci/scripts/utils/tool_versions.sh +cargo install lychee --locked --version "${LYCHEE_VERSION}" +bash ci/scripts/markdown_link_check.sh +``` + +Notes: + +- The script is run with `bash` and is compatible with the default Bash on macOS (no `mapfile` dependency). Review Comment: these notes seem like they should be in the scripts themselves -- it is irrelevant detail here ########## docs/source/contributor-guide/testing_non_functional.md: ########## @@ -0,0 +1,287 @@ +<!-- +SPDX-License-Identifier: Apache-2.0 +SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership +SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements +--> + +# Non-Functional Tests + +This page covers checks for CI workflows, documentation, repository maintenance, and benchmarks. For tests of DataFusion's query behavior, see [Testing](testing.md). + +## CI Workflow Tests + +The Rust workflow validator and fork status bridge have regression tests. +These tests cover skip conditions, fork run recovery, polling frequency, and +concurrency groups for fork pushes and merge queue runs. +They run in the `asf-yaml-check` CI job and in `./dev/rust_lint.sh`. +Run them locally with Python, PyYAML, and Node.js 24: + +```shell +python3 ci/scripts/test_rust_workflow_skips.py +node --test ci/scripts/test_fork_ci_status.cjs +``` + +The validator tests remove guards and add an invalid job dependency in memory. +The bridge tests use stubbed GitHub API responses and make no network requests. + +## Documentation Examples + +We use Rust [doctest] to verify examples from the documentation are correct and +up-to-date. These tests are run as part of our CI and you can run them +locally with the following command: + +```shell +cargo test --doc +``` + +### API Documentation Examples + +As with other Rust projects, examples in doc comments in `.rs` files are +automatically checked to ensure they work and evolve along with the code. + +Prefix a line with `# ` (a `#` followed by a space) to keep it compiled and +run, but hidden from the rendered documentation. This is most useful for +`use` statements, since readers care about the API being demonstrated, not +its import boilerplate: + +````rust +/// ``` +/// # use std::sync::Arc; +/// let value = Arc::new(42); +/// ``` +```` + +Only the `let value = ...` line renders. + +### User Guide Documentation + +Rust example code from the user guide (anything marked with \`\`\`rust) is also +tested in the same way using the [doc_comment] crate. See the end of +[core/src/lib.rs] for more details. + +[doctest]: https://doc.rust-lang.org/rust-by-example/testing/doc_testing.html +[doc_comment]: https://docs.rs/doc-comment/latest/doc_comment +[core/src/lib.rs]: https://github.com/apache/datafusion/blob/main/datafusion/core/src/lib.rs#L583 + +## Documentation Link Checks + +`./dev/rust_lint.sh` runs the internal markdown link check. If `lychee` is +missing, the script installs the version pinned in +`ci/scripts/utils/tool_versions.sh`. It uses an existing installation as is, +even if the version differs from the pin. + +To run the check on its own: + +```shell +source ci/scripts/utils/tool_versions.sh +cargo install lychee --locked --version "${LYCHEE_VERSION}" +bash ci/scripts/markdown_link_check.sh +``` + +Notes: + +- The script is run with `bash` and is compatible with the default Bash on macOS (no `mapfile` dependency). +- The CI configuration currently checks internal markdown links only. External `http(s)` and `mailto` links are excluded to avoid flaky failures. +- The check only reports broken links. `./dev/rust_lint.sh --write` does not change them. + +When a link is broken, lychee prints the file and URL/path that failed. For example: + +```text +[docs/source/user-guide/cli/overview.md]: + [ERROR] file:///.../docs/source/user-guide/cli/missing-page.md | Cannot find file: File not found. Check if file exists and path is correct +``` + +Rust doc comments are validated by rustdoc in CI and can be checked locally with: + +```shell +bash ci/scripts/rust_docs.sh +``` + +## ASF Status Check Validation + +`ci/scripts/check_asf_yaml_status_checks.py` checks that every required status +check in `.asf.yaml` matches a job in `.github/workflows`. +`ci/scripts/check_rust_workflow_skips.py` checks that `rust.yml` skips only its +listed jobs on pushes to `main` and on pull requests from forks. +`./dev/rust_lint.sh` runs both and needs `python3` with [PyYAML]. The [uv] Review Comment: listing requirements would be better in the script I think ########## docs/source/contributor-guide/testing_non_functional.md: ########## @@ -0,0 +1,287 @@ +<!-- +SPDX-License-Identifier: Apache-2.0 +SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership +SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements +--> + +# Non-Functional Tests + +This page covers checks for CI workflows, documentation, repository maintenance, and benchmarks. For tests of DataFusion's query behavior, see [Testing](testing.md). + +## CI Workflow Tests + +The Rust workflow validator and fork status bridge have regression tests. Review Comment: I think we could omit `These tests cover skip conditions, fork run recovery, polling frequency, and concurrency groups for fork pushes and merge queue runs.` and simply link to the scripts which would document their own behavior -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
