Your message dated Tue, 5 Nov 2019 16:33:07 +0100
with message-id <20191105153307.GA19352@jcristau-z4>
and subject line Re: Bug#912524: snapshot.debian.org is unreachable from
(apparently) 18.128.0.0/9
has caused the Debian Bug report #912524,
regarding snapshot.debian.org is unreachable from (apparently) 18.128.0.0/9
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ow...@bugs.debian.org
immediately.)
--
912524: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=912524
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems
--- Begin Message ---
Package: snapshot.debian.org
Severity: important
Mozilla uses snapshot.debian.org as a way to generate deterministic
docker images for its CI. Those docker images are built from Amazon EC2
instance, and we've had recurring intermittent errors connecting to
snapshot.debian.org. After having been able to observe the problem first
hand (as in, with shell access), I've determined some interesting
information:
- While I originally was assuming that snapshot.debian.org was just
down at the time we were getting the errors, it turns out it *is*
available from other places of the internet (like other EC2 instances
or my own machine at home).
- When that happens, *both* snapshot.debian.org IPv4 addresses are
unreachable (193.62.202.27 and 185.17.185.185).
- Traceroute stops at the last router before those hosts. That is, the
first "* * *" hop is where, on a machine where it's reachable,
snapshot.debian.org would appear.
- Looking back at the logs from all the jobs we've had in the past
failing to reach snapshot.debian.org (or at least, marked as such),
the IP addresses of the hosts they were running on (as well as the IP
address of the host I had direct access to and that couldn't connect
to snapshot.debian.org) were all in the 18.128.0.0/9 block[1].
Mike
1. Full list:
18.144.32.96
18.144.63.37
18.206.136.206
18.207.181.113
18.212.146.108
18.212.202.120
18.212.204.30
18.212.240.147
18.232.99.231
18.236.139.10
18.236.215.218
--- End Message ---
--- Begin Message ---
On Mon, Oct 21, 2019 at 03:29:28PM +0200, Julien Cristau wrote:
> On Fri, Nov 02, 2018 at 09:15:12AM +0000, Peter Palfrader wrote:
> > On Thu, 01 Nov 2018, Noah Meyerhans wrote:
> >
> > > It was pointed out on IRC that this is intentional, per
> > > https://salsa.debian.org/dsa-team/mirror/dsa-puppet/blob/master/modules/roles/manifests/snapshot_web.pp
> > >
> > > IMO blocking random (and large) chunks of EC2 is not a good idea, as the
> > > collateral impact is potentially huge. I'd like to suggest a more
> > > targeted way of throttling individual clients that doesn't have such
> > > broad impact. The iptables connlimit module comes to mind, but there are
> > > undoubtedly other options.
> >
> > It's not random. Still, I agree that blocking large chunks is not
> > ideal.
> >
> > We would welcome you working with us on finding actual rate limiting
> > configurations that work. So far, many have suggested but nobody has
> > actually delivered anything.
>
> I have tentatively removed the block on AWS in
> https://salsa.debian.org/dsa-team/mirror/dsa-puppet/commit/6510538f5a1a525e62e85be0d887c1f1b3e0e3fd
>
> We'll see how that goes.
>
Things seem to hold up well enough so far; closing.
Cheers,
Julien
--- End Message ---
