Package: ftp.debian.org Severity: normal Existing files under http://archive.debian.org/debian/pool/main/ (bullseye) intermittently return 404 Not Found when a request misses the CDN cache. The 404 page is generated by the origin Apache, and the CDN then caches it. The same URL returns 206 or 404 on successive requests.
Impact apt in our CI (sbuild, bullseye chroots) fetches bullseye main from archive.debian.org. Between 2026-10-14 09:21 UTC and at least 2026-10-15 11:44 UTC, 8 builds failed with errors like: E: Failed to fetch http://archive.debian.org/debian/pool/main/p/python-wrapt/python3-wrapt_1.12.1-4%2bb1_amd64.deb 404 Not Found [IP: 151.101.66.132 80] A different file failed in each build. Seven distinct files were affected: python3-wrapt, python3-pystemd, python3-zipp, libelf1, libpython3-stdlib, libfile-stripnondeterminism-perl and autotools-dev. Retries sometimes fail again. Evidence All seven files exist: requests that were answered from cache returned 206 every time (455 of 455, all cache hits, from two CI hosts and one other machine). To force requests past the cache, I added a unique query string to the same URL (1-byte range requests, 7 files). 27 of 105 requests returned 404 and 78 returned 206. The 404s came from two CI hosts in different datacentres and from a third machine, via many different edge servers (cache-fra-*), and every one of the 7 files returned both 404 and 206. Sending Cache-Control: no-cache or Pragma: no-cache does not bypass the cache (35 of 35 requests were cache hits), so the query string is the only bypass I found. A 404 response, in full: HTTP/1.1 404 Not Found Server: Apache Content-Type: text/html; charset=iso-8859-1 Content-Length: 304 Age: 0 Date: Thu, 15 Oct 2026 14:45:52 GMT Via: 1.1 varnish X-Served-By: cache-fra-eddf8230227-FRA X-Cache: MISS X-Cache-Hits: 0 The body is the standard Apache page ending "Apache Server at archive.debian.org Port 80". The cache missed and the origin answered 404. The CDN also caches the 404. For 4 unique URLs that had just returned 404, I repeated the same request three times: every repeat returned 404 with X-Cache: HIT and a rising Age (1 to 3 seconds), served by different edge servers each time. I only tested this on unique query-string URLs. I did not request the real URLs repeatedly, to avoid caching a 404 for them. If the same happens for real URLs, a file that hits a bad back end once could keep returning 404 until the cached entry expires, which would match the repeat failures we saw. Reproduction (bash; about 15% of requests return 404, and each request is a single byte): url=http://archive.debian.org/debian/pool/main/p/python-wrapt/python3-wrapt_1.12.1-4%2bb1_amd64.deb for i in $(seq 1 20); do curl -s -o /dev/null -r 0-0 -w '%{http_code} ' "$url?cb=$RANDOM$RANDOM" done; echo Example output: 206 206 206 206 206 206 206 404 206 206 206 404 206 206 206 206 206 206 206 404 Question This looks like some back-end servers behind archive.debian.org missing parts of the bullseye pool, or not being in sync with the others. #1147093 (closed 2026-10-04) covers bullseye-security on archive.debian.org and mentions back-end servers missing files; I don't know whether the same servers are involved. Could someone check that all back ends serve the same bullseye main pool content? I can run further tests if useful. Michael Bennett Staff DevOps Engineer Release and Deployment IONOS Cloud GmbH | Revaler Str. 28-31 | 10245 Berlin | Germany E-Mail: [email protected] | Web: cloud.ionos.de General information about the company can be found here. A member of the IONOS Group
