What a day.. hope you are better now :)
On 6/12/2021 2:42 AM, Dan Mahoney wrote:
I only just now found this thread, so I'm sorry I'm late to the party,
but here, I put it on Medium.
https://gushi.medium.com/the-worst-day-ever-at-my-day-job-beff7f4170aa
<https://gushi.medium.com/the-worst-day-ever-at-my-day-job-beff7f4170aa>
On Mar 12, 2021, at 10:07 PM, Mark Tinka <mark@tinka.africa> wrote:
Hardly famous and not service-affecting in the end, but figured I'd
share an incident from our side that occurred back in 2018.
While commissioning a new node in our Metro-E network, an IPv6
point-to-point address was mis-typed. Instead of ending in /126, it
ended in /12. This happened in Johannesburg.
We actually came across this by chance while examining the IGP table
of another router located in Slough, and found an entry for 2c00::/12
floating around. That definitely looked out of place, as we never
carry parent blocks in our IGP.
Running the trace from Slough led us back to this one Metro-E device
in Jo'burg.
It took everyone nearly an hour to figure out the typo, because for
all the laser focus we had on the supposed link of the supposed box
that was creating this problem, we all overlooked the fact that the
/12 configured on the point-to-point link was actually supposed to
have been a /126.
The reason this never caused a service problem was because we do not
redistribute our IGP into BGP (not that anyone should). And even if
we did, there are a ton of filters and BGP communities on all devices
to ensure a route such as that would have never made it out of our AS.
Also, the IGP contains the most specific paths to every node in our
network, so the presence of the 2c00::/12 was mostly cosmetic. It
would have never been used for routing decisions.
Mark.
