This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-24333-110a90aeebe6cad3e2963f46463fb1e7c4e2cca7 in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit bbf739a3cfe1d5cc7646e37250c947c6a56b5b0e Author: Narendra <[email protected]> AuthorDate: Sun Aug 16 14:20:33 2026 +0000 feat: Add robots.txt to site (#24333) ## Which issue does this PR close? - Closes #23258 ## What changes are included in this PR? There are 2 PR's for this issue. This one contains the changes for _robots.txt_ - added the robots.txt , added license header and allow rules. ## Are these changes tested? Yes. Tested by building the docs and verifying the files getting generated in the build/ ## Are there any user-facing changes? No. --------- Co-authored-by: Jeffrey Vo <[email protected]> --- dev/release/rat_exclude_files.txt | 1 + docs/source/conf.py | 2 +- docs/source/robots.txt | 4 ++++ 3 files changed, 6 insertions(+), 1 deletion(-) diff --git a/dev/release/rat_exclude_files.txt b/dev/release/rat_exclude_files.txt index 77da7db87e..74e4279a6a 100644 --- a/dev/release/rat_exclude_files.txt +++ b/dev/release/rat_exclude_files.txt @@ -61,6 +61,7 @@ datafusion/proto-common/src/generated/prost.rs .github/ISSUE_TEMPLATE/feature_request.yml .github/workflows/docs.yaml docs/source/llms.txt +docs/source/robots.txt **/node_modules/* datafusion/wasmtest/pkg/* clippy.toml diff --git a/docs/source/conf.py b/docs/source/conf.py index 81ffce3ad1..68ba628bba 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -112,7 +112,7 @@ html_static_path = ["_static"] # Copy agent-facing files (llms.txt) verbatim to the site root so they # resolve at the conventional URL `https://datafusion.apache.org/llms.txt`. -html_extra_path = ["llms.txt"] +html_extra_path = ["llms.txt", "robots.txt"] # Base URL of the published site. Required by sphinx-sitemap to build # absolute <loc> entries in the generated sitemap.xml. diff --git a/docs/source/robots.txt b/docs/source/robots.txt new file mode 100644 index 0000000000..9c80d8219c --- /dev/null +++ b/docs/source/robots.txt @@ -0,0 +1,4 @@ +User-agent: * +Allow: / + +Sitemap: https://datafusion.apache.org/sitemap.xml --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
