On 3/22/24 17:42, Adam D. Barratt wrote:
Control: tags -1 + moreinfo
On Wed, 2024-02-14 at 20:03 +, OVHcloud wrote:
Site: debian.mirrors.ovh.net
Archive-architecture: ALL amd64 arm64 armel armhf hurd-i386 hurd-
amd64 i386 mips mips64el mipsel powerpc ppc64el riscv64 s390x
Archive-http: /debian/
Maintainer: OVHcloud
Country: FR France
Location: Anycast (Gravelines, Roubaix and Strasbourg)
Hi Adam,
First, let me just explain in a bit more detail how requests are
handled. Our DNS A record for debian.mirrors.ovh.net only points to one
anycast IP address with 4 locations. Once it receives a request, the
following happens:
Load balancer (anycast IP) in Beauharnois (Canada)→ Caches in
Beauharnois → Backend in Canada if cache miss
Load balancer (anycast IP) in Gravelines (France)→ Caches in Gravelines
→ Backend in France if cache miss
Load balancer (anycast IP) in Roubaix(France)→Caches in Roubaix→ Backend
in France if cache miss
Load balancer (anycast IP) in Strasbourg(France)→Caches in Strasbourg→
Backend in France if cache miss
I know there was some discussion on IRC, so apologies if I'm rehashing
here, but:
- are the individual backends exposed in any way?
Not publicly, but if you were to give me me a source IP address, I could
provide you with an rsync account that has access to both our backends.
We already do this for other distributions.
- how do you ensure that the backends are in sync with each other?
We take ZFS snapshots after each sync and we send these to the backends.
Once the latest snapshot has been sent to both backends, they start
pointing to it. This last operation is performed simultaneously on both.
After this, we compile a list of changed URLs and issue a series of HTTP
PURGE requests to every cache server, simultaneously too.
We also have monitoring in place to detect discrepancies between the
backends for all our mirrors.
- what are the chances of users seeing inconsistent state if they hit
different backends which aren't at the same stage of updating?
I think the chances are pretty low as we try to run all phases of the
sync simultaneously. The most sensitive part would be the cache
invalidation process which might purge some URLs at slightly different
times.
All I can say is that, despite the high number of servers already
relying on our Debian mirror, we have never heard of errors caused by this.
Regards,
Adam
I hope this answers all your questions.
Have a nice day,
Louis
