Hi, while debugging rate-limiting issues on snapshot.d.o, I noticed that I run into the rate-limiting not due to the file-transfers, but due archive/.deb > /file/<hash> redirects. When apt accesses a file on the pool, this is not returned directly but via a 302 redirect (see curl log below). This redirect however has a max-age of just 10mins. The corresponding code is probably in [1]. This raises the following questions:
1. Why return a 302 (temporary moved) instead of a 301 (permanently moved)? 2. Why not use the same max-age (31536000)? Further, the rate-limits should be precisely documented so clients / caching proxies can adapt to this. The limits also need to match the retry-after header in the 429 responses. Currently s.d.o responds with retry-after 5 (seconds), which is by far to short to overcome the limit. If rate-limiting would be implemented correctly, downstream caches could properly cache the results and clients like apt could behave nicely. I further recommend to use WAY higher request limits in combination with a moving average limit on the amount of transferred data. By that, the cheap "is my cache still valid" requests could pass, while the more heavy payload transfers are avoided. Also clients could hit s.d.o without reduced transfer rates, hence reducing the amount of open handles on the server. [1] https://salsa.debian.org/snapshot-team/snapshot/-/blob/master/web/app/snapshot/views/archive.py?ref_type=heads#L96 curl -v -L http://snapshot.debian.org/archive/deb ian-security//20240702T082400Z/pool/updates/main/j/json-c/libjson-c- dev_0. 15-2%2bdeb11u1_amd64.deb > /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:- - 0* Trying 185.213.153.170:80... * Connected to snapshot.debian.org (185.213.153.170) port 80 (#0) > GET /archive/debian- security//20240702T082400Z/pool/updates/main/j/json-c/libjson-c- dev_0.15-2%2bdeb11u1_amd64.deb HTTP/1.1 > Host: snapshot.debian.org > User-Agent: curl/7.88.1 > Accept: */* > < HTTP/1.1 302 FOUND < Date: Tue, 05 Nov 2024 08:41:38 GMT < Server: Apache < X-Content-Type-Options: nosniff < X-Frame-Options: sameorigin < Referrer-Policy: no-referrer < X-Xss-Protection: 1 < Permissions-Policy: interest-cohort=() < Content-Length: 279 < Location: /file/a152d21f12e32c7f40cb03220502e238dbb264fd < Cache-Control: max-age=600, public < Expires: Tue, 05 Nov 2024 08:51:38 GMT < X-Clacks-Overhead: GNU Terry Pratchett < Content-Type: text/html; charset=utf-8 < X-Varnish: 135556047 < Age: 0 < Via: 1.1 varnish (Varnish/7.1) < Xonnection: close < Connection: Keep-Alive Best regards, Felix Moessbauer -- Siemens AG, Technology Linux Expert Center
