Hi,
I'm copying here, for reference, an IRC log from #debian-admin:
19:38 <@jcristau> lucas: right, so the importer would indeed somehow need to
learn about copying stuff to other locations/backends.
the python/web bit in practice doesn't matter, i think,
because it's not used for /file/... requests, those are
serviced by apache directly
(https://salsa.debian.org/dsa-team/mirror/dsa-puppet/-/blob/production/modules/roles/templates/snapshot/snapshot.debian.org.conf.erb#L76)
19:42 <@jcristau> lucas: the way the copying of stuff to the replica happens
currently is very async
(https://salsa.debian.org/snapshot-team/snapshot/-/blob/master/master/import-run#L47-L52)
which means we just hope
things are eventually consistent, but there's a window where
the replica would 404 for new files. that might be ok,
because in practice users probably aren't requesting the very
latest mirror run, but
19:42 <@jcristau> it might be nice to somehow keep track of what's synced where
and what's new, also to reduce how much time we're
spending walking directories.
21:38 < lucas> ah right, apache could just redirect to S3 or act as a proxy, or
the web app could point to S3 directly
21:40 < lucas> re replica, I wonder if we shouldn't instead aim for several
parallel instances of snapshot (at least 2), each living
its own life (not in a master/slave setup)
03:46 <@pabs> lucas: then you would get different snapshot timestamps?
07:47 < lucas> pabs: do we care?
07:52 <@pabs> having multiple sets of timestamps served from different servers
seems confusing for end-users
08:38 < lucas> ah, my idea was more that they would be exposed as different
services, with one being used for snapshot.debian.org
08:40 < lucas> the snapshot services could actually share timestamps afterwards
(for example to cover a period of downtime)
15:26 <@jcristau> lucas: my idea would be to add the s3 bucket as a backend in
https://salsa.debian.org/dsa-team/mirror/cdn-fastly/-/blob/master/services/snapshot.yaml
and use that for /file/
21:14 < lucas> jcristau: I'm not familiar with how CDNs work. could you tell
fastly to fallback to other backends if the first one
replied with 404?
21:57 <@Mithrandir> we can do that
21:57 <@Mithrandir> it might not be wise
