Hi! Ludovic Courtès <l...@gnu.org> skribis:
> Mathieu Othacehe <othac...@gnu.org> skribis: > >> A lot of builds, among them ~20 system tests[1], are failing with: >> "cannot build missing derivation >> ?/gnu/store/hs6kp1lqgymhyp3jndc0dsp0pn4psgv0-gui-installed-desktop-os-encrypted.drv?" >> errors. >> >> Those derivations are present on the CI head node. This means that the >> errors occur during substitution. This is most likely caused by some >> issue with the publish server, because: >> >> - The publish server serves a 404 error. We should get rid once and for >> all of this 404 thing, pushing something like: >> https://issues.guix.gnu.org/50040. >> >> or >> >> - The publish server is not fast enough and hits an Nginx timeout that >> closes the communication. > > Also being discussed at <https://issues.guix.gnu.org/48468#12>. I got confirmation that the cache-bypass-threshold hypothesis holds, at least for system tests. Namely, looking at <https://ci.guix.gnu.org/build/2258097/details>, which ends like this: --8<---------------cut here---------------start------------->8--- @ substituter-succeeded /gnu/store/qh2876i5l1wvxgwhg9fbl9zmb3px3n2m-gc-roots.drv fetching path `/gnu/store/fh9dnmrfsz429pwqmvsjnk0snlm959kc-xdg-mime-database-builder'... @ substituter-started /gnu/store/fh9dnmrfsz429pwqmvsjnk0snlm959kc-xdg-mime-database-builder substitute Downloading http://141.80.167.131/nar/lzip/fh9dnmrfsz429pwqmvsjnk0snlm959kc-xdg-mime-database-builder... .[K xdg-mime-database-builder 3.6MiB/s 00:00 | 3KiB transferred.[K xdg-mime-database-builder 1.9MiB/s 00:00 | 3KiB transferred @ substituter-succeeded /gnu/store/fh9dnmrfsz429pwqmvsjnk0snlm959kc-xdg-mime-database-builder cannot build missing derivation ‘/gnu/store/4r1wij3bzj9zv75ds82a93jl7bcman2x-installed-extlinux-os.drv’ --8<---------------cut here---------------end--------------->8--- Looking at the nginx and ‘guix publish’ logs, I found that the missing substitute is not that of 4r1wij3bzj9zv75ds82a93jl7bcman2x (the .drv itself) but rather that of a dependency of that .drv: [14/Oct/2023:23:22:09 +0200] "GET /wqqzcxrhbnv0nzg64iiiqd5grr4vk2zg.narinfo HTTP/1.1" 404 58 "-" "GNU Guile" That item’s size is above the cache bypass threshold of 100 MiB as currently configured on berlin: --8<---------------cut here---------------start------------->8--- $ du -hs /gnu/store/wqqzcxrhbnv0nzg64iiiqd5grr4vk2zg-guix-5a6b1a5 124M /gnu/store/wqqzcxrhbnv0nzg64iiiqd5grr4vk2zg-guix-5a6b1a5 --8<---------------cut here---------------end--------------->8--- The immediate fix/workaround is to raise that threshold. A better solution would be for system tests to depend on a fixed-output derivation for the Guix source instead of the “source” above (I use “source” as it is used in the context of <derivation>). Thanks, Ludo’.