SameerMesiah97 opened a new pull request, #74536: URL: https://github.com/apache/airflow/pull/74536
**Description** This change adds automatic generation of Markdown pages, `llms.txt` and `llms-full.txt` for the Common AI provider documentation. It builds on the proof of concept in [this commit](https://github.com/apache/airflow/compare/main...astronomer:airflow:llms-txt-poc) using `sphinx-llm` to generate a `.html.md` copy alongside each documentation page. The Airflow extension groups the index by documentation navigation, uses versioned URLs and links example excerpts to their complete source files on GitHub. Markdown copies are also generated for redirect stubs. The build fails if a published HTML page has no Markdown copy, excluding `_modules/`, `genindex`, `search` and `py-modindex`. **Rationale** Coding agents consume Airflow documentation to write DAGs, but fetching HTML can lose code samples and other useful context during extraction or summarisation. Providing Markdown copies gives agents access to the documentation content and code samples in a simpler format. `llms.txt` helps them discover relevant pages, while `llms-full.txt` provides the indexed content in one file. Redirect coverage ensures that older documentation URLs also have a Markdown response when Markdown serving is enabled. **Tests** Added tests verifying that: - Markdown redirects resolve destinations correctly and preserve existing documentation. - Missing Markdown copies fail the build, with the intended exclusions. - Generated indexes follow navigation and use versioned URLs. - Example source references select the appropriate Git ref. - Example links, URL substitutions and comment removal behave correctly. - A Sphinx build renders diagrams, code captions, admonitions, tables, navigation captions and same-page links correctly. - Markdown builds use a separate doctree cache. **Notes** Markdown generation is enabled for `apache-airflow-providers-common-ai` only. Publishing changes and HTTP content negotiation are separate parts of #74376. The proof of concept’s generic rendering fixes and doctree-cache workaround remain in this branch pending upstream fixes in `sphinx-markdown-builder` and `sphinx-llm`. Their removal remains part of the issue’s pre-merge requirements. **Documentation** The documentation build generates `.html.md` pages, `llms.txt` and `llms-full.txt`, with alternate Markdown links in the HTML pages. **Backwards Compatibility** Existing HTML documentation remains available. Markdown generation is opt-in, so other provider packages retain their existing build behaviour. **Acknowledgements** This change builds on the [proof of concept](https://github.com/apache/airflow/compare/main...astronomer:airflow:llms-txt-poc) by @kaxil , which introduced Markdown pages, `llms.txt` and `llms-full.txt` generation for the Common AI provider documentation. Related: #74376. ###### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: [GPT 6.1] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
