Hi all, More and more people write Dags with coding agents (Claude Code, Cursor, Copilot, Codex), and those agents read our docs as HTML. A provider guide page is about 150 KB of HTML for about 20 KB of content, and fetch tools often summarise it down to a paragraph, code samples included. Pydantic AI, Ray, Camel, Astronomer and most of the popular sites & project publish an llms.txt index plus a Markdown copy of each page. We publish neither.
I've opened an issue with a plan and a working proof of concept, starting with the Common AI provider docs: https://github.com/apache/airflow/issues/74376 It has three parts: 1. Docs build: generate page.html.md next to every page, plus llms.txt and llms-full.txt, using sphinx-llm and a small extension. The proof of concept builds cleanly for Common AI with no change in build time. 2. Publishing: a couple of guards in breeze so stable/ always has the Markdown copies. 3. Serving: Claude Code, Cursor, Copilot and OpenCode already send "Accept: text/markdown" on every fetch. A short .htaccess rule in airflow-site can answer those requests with the Markdown copy, so agents get it from the URLs they already use. I tested this on airflow.staged.apache.org; ASF's Fastly cache keeps the HTML and Markdown answers apart. *I won't have time to build this or to review it, so it needs an owner who can see it through, including reviews.* The issue lists what's done, what's left, and the rollout constraints. It needs a committer, or a contributor pairing with one: testing the airflow-site change on staging, publishing docs and checking the live rollout all need committer access. This would be a high impact change, so thanks in advance for whoever picks it up. Regards, Kaxil
