Hi, I’m building a (French LaTeX) FAQ <https://dbitouze.gitlab.io/test-faq-fr/index.html> as a static website with Sphinx-doc <https://www.sphinx-doc.org/>. In order to let other people easily contribute, I'm trying to regenerate the HTML pages corresponding to the changed source files thanks to:
- CD/CI - GitLab pages on a gilab.com instance. That works well, except that even if only a single source file is changed, the HTML pages of all the source files are regenerated and, since they are more than 1200 source files, that takes too much time (more than 15 minutes). In order to have a look at this issue, one can consider this minimal Sphinx-doc content <https://gitlab.com/gutenberg1/minimal-sphinx/-/tree/main>: - with mainly default Sphinx settings, - with a minimal conf.py config file, - but with many (100) test files in order to highlight the issue I'm facing. I first asked for help on the dedicated Discourse GitLab CI/CD forum <https://forum.gitlab.com/t/is-it-possible-to-have-a-kind-of-persistent-docker-image-for-gitlab-ci/89542?u=dbitouze> (https://forum.gitlab.com/t/is-it-possible-to-have-a-kind-of-persistent-docker-image-for-gitlab-ci/89542), and people there are pretty sure the sphinx-doc's cache should be restored and don't understand what's going on. I then got help outside this forum from a guy (but who has no clue about sphinx-doc). Here is what he said: [Y]ou are shooting yourself in your foot because you use make. make is utterly ineffective in CI pipelines because at the beginning of each pipeline the repo is cloned afresh, meaning the file modification dates of the source files are usually newer than the cached output files even if nothing did change. Because make only considers file modification dates/timestamps, make is definitely not helpful for the first invocation. The second invocation obviously behaves correctly because the first make invocation rebuilds all outputs. So you can reduce this minimal example by removing make and simply calling sphinx-build. And the pipelines for this new repository also show that a cache is pulled at the beginning and uploaded at the end of the pipeline. So GitLab’s caching is working. It’s now a question of what sphinx-build needs to determine whether to rebuild. If it works like make, you are out of luck because of file timestamps. If it works with another mechanism, you should look up where that is stored and check that all information needed for the rebuild are cached (maybe more is needed than just the doctrees directory). I told him that: - I used to try with sphinx-build instead of make but I didn’t work either, - I’m afraid sphinx-build doesn’t work with another mechanism and, regarding the cache, only the doctrees directory is involved. He answers then: So you need to use one of the solutions modifying the file timestamps to match the git commits to have an effective solution here. There are a variety of tools out there that do this. Just search the internet for git checkouts with mtime preserved. I had the hope that `git-restore-mtime` would be the solution: https://forum.gitlab.com/t/deploy-only-changes-with-lftp/76180 but hope dashed: https://gitlab.com/gutenberg1/minimal-sphinx/-/blob/main/.gitlab-ci.yml https://gitlab.com/gutenberg1/minimal-sphinx/-/jobs/4703827511#L52 The guy told me then: “Unfortunately, I have not been able to find any documentation on how sphinx's change detection works.“ Any help would be much appreciated. Thanks. -- You received this message because you are subscribed to the Google Groups "sphinx-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to sphinx-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/sphinx-users/161dbdc4-3628-494b-8ffc-e870ca256c8cn%40googlegroups.com.
