kaxil opened a new pull request, #73477: URL: https://github.com/apache/airflow/pull/73477
The two documentation jobs in CI (`Build documentation (--docs-only)` and `(--spellcheck-only)`) take 25 to 35 minutes on `main`, and the spellcheck job took 93 minutes on a PR that adds a documentation page. This PR cuts the build itself by about 2.5x and removes the spellcheck job's worst case, without changing what gets built or checked. Measured on the same machine with the full set of 115 packages and 4 parallel jobs (the CI runner configuration): | | Before | After | |---|---|---| | Full `--docs-only` build, wall time | 645 s | 214 s | | CPU time across the 4 workers | 41.6 min | 14.2 min | | Median package | 17 s | 2 s | | 16 small providers, wall time | 66 s | 36 s | The build is now bound by the single largest package (google, 212 s of the 214 s); everything else fits around it on the other three workers. Output is unchanged: the generated HTML for a provider is byte-identical to the previous builder's (only the pagefind index file names differ, they are content hashes). ## Design rationale **Where the time went.** Per-package timings from the CI log show 98 of the 115 packages take under 40 s each but together cost 48 of the job's 73 CPU-minutes. Profiling one small provider: about 20 of its 22 seconds are fixed startup that has nothing to do with its own documentation. autoapi's astroid parsing of the `airflow`/`airflow.sdk` modules every provider imports (about 8.5 s), parsing and schema-validating all 108 `provider.yaml` files (about 3.4 s, done two or three times per build), importing `airflow` (1.4 s), and loading about 125 intersphinx inventories (1 s). Every package paid this in its own `sphinx-build` subprocess. **Sphinx now runs in-process inside each pool worker** (`sphinx.cmd.build.build_main` instead of `subprocess.run`), so a worker pays that startup once and then reuses the astroid parse cache, the imported `airflow`, the parsed `provider.yaml` files and the parsed inventories for the roughly 29 packages it builds. This is the same mode `sphinx-autobuild` and Sphinx's own test suite use, and Sphinx resets the docutils registries between apps. Three things had to be handled for it to be safe: - The per-package `conf.py` files mutate lists they import from `docs.utils.conf_constants` (`extensions.append`, `autoapi_ignore.extend`), so those modules are dropped from `sys.modules` before each build and re-executed fresh. Otherwise provider B inherits provider A's ignore patterns, including B's own package directory. - Redirecting stdout/stderr straight to the per-package log file breaks the next build: docutils' `Reporter` inside sphinx-argparse's nested parser keeps the stream it first saw, and once that file is closed the `fab` and `edge3` `cli-ref` pages fail with "I/O operation on closed file". Output goes through a proxy stream that lives as long as the worker and only switches its target. - The astroid cache is trimmed to drop the built package's own modules after each build, and nothing else. Deciding what to keep on the worker's resident memory does not work (a worker that built google stays at 3 GB because of the modules autodoc imported, so a memory check fires after every later build and the cache is never warm), and dropping third-party modules costs most of the gain because `airflow.sdk` pulls in attrs, pydantic and sqlalchemy that every provider then re-parses. The ASTs themselves are cheap: clearing the whole cache after an amazon build freed under 100 MB. `sphinx-autobuild` stays a subprocess (it is a long-running server), and the old 30-minute per-build timeout is kept through `SIGALRM`. **The spellcheck job's 93 minutes were three full passes.** The spelling builder inherits Sphinx's write phase, so it resolves cross-references and fails under `-W` on any label or page that is new in the PR, but it never writes an `objects.inv`. The docs build never hits this because the html builder writes inventories that later packages pick up. On the spellcheck side the failure triggered `rebuild_all_packages=True`: html for all 115 packages, then spelling for all 115 again. In the run linked below, one package (the providers summary) failed on the `operators/llm_batch` page this PR's neighbour added, and the job spent 57 of its 93 minutes rebuilding everything else. The spelling build now writes an inventory too (with the URIs the html build would produce), `airflow_intersphinx` prefers it over the downloaded one, and a retry only rebuilds the packages that failed, in the mode the run was started with. The `A -> B -> C` chain the extra passes were for is still covered by two retry passes, which are no-ops when nothing failed. **Scheduling.** Packages are handed out heaviest first with `imap_unordered(chunksize=1)`; `pool.map` pre-assigned fixed chunks of 8, which is why `task-sdk`, `helm-chart` and `docker-stack` were always the last three to start. **Runner disk cleanup.** The docs job passes `free-up-disk-space: false` to the shared prepare action (new input, default unchanged). The step removes pre-installed tooling for 2 minutes 43 seconds; the runner has 93 GB free before it and the docs job only needs the 8 GB CI image. `provider_yaml_utils` also stops re-validating the JSON schema on every one of the 108 files and uses the C YAML loader (2.6x faster per load); this matters less now that a worker loads once, but it is what every single-package local build pays. ## Gotchas - A worker's resident memory now stays at the peak of the largest package it built (about 3 GB after google) instead of dropping between packages. That peak is the same the previous per-package subprocess reached; what changes is that it persists for the rest of the run. With 4 workers on a 16 GB runner this leaves headroom, and the memory is held by imported modules, not by anything the builder could release. - The intersphinx reuse wraps a private Sphinx function (`sphinx.ext.intersphinx._load._fetch_inventory`). It exists in Sphinx 8.x, which is what CI's default Python 3.10 resolves from `uv.lock`; Sphinx 9 (Python 3.11+) split fetching from parsing, so there the wrapper is a guarded no-op, its tests skip, and each package spends about a second re-parsing inventories as before. Everything else in this PR is version-independent. Run that prompted this: https://github.com/apache/airflow/actions/runs/35612510405 (docs-only 29 min, spellcheck 93 min). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
