Huge respect for admitting that ARIN uses Excel as an IPAM solution. You are braver than I...
I'll see myself out. On Fri, Dec 12, 2025 at 12:29 PM John Curran via NANOG < [email protected]> wrote: > Chase (and the NANOG operator community) – > > Apologies for not replying earlier, but I wanted to make sure we > understood exactly what went wrong and had that written up as an incident > report (i.e., rather than responding piecemeal and without full clarity). > > Short version – ARIN failed here (as you noted in your post). We’ve > published a public incident report that lays out what happened, the impact, > and what we’re changing: https://www.arin.net/announcements/20251212/ > > This came down to some remaining manual handling around NRPM 4.10 space, > which uses a sparse allocation model and wasn’t yet fully integrated into > our automated inventory. That gap allowed your already in-use IPv4 /24 > block to be mistaken as available and reissued to another customer. When it > was removed and reissued, your associated ROA was removed as well, along > with reverse DNS services, etc. > > We had plans to automate our NRPM 4.10 inventory management (largely for > efficiency reasons), but this incident and subsequent review showed that > the remaining manual steps pose more risk than is reasonable. As a result, > we’ve moved that work well up the priority list for development. In the > meantime, and as detailed in the incident report, we’ve put additional > controls in place – including a mandatory second review on any resource > deletion from an organization – to prevent this from happening again. > > I will also be reviewing our number resource inventory management > practices internally (and with the ARIN Board of Trustees) to ensure there > are not any other similar situations that might pose such a risk. My > deepest apologies for this incident; we are acutely aware that the > integrity of Internet number resources is essential to network operators, > and thus it must be inherent to ARIN’s performance at all times. > > Sincerely, > /John > > John Curran > President and CEO > American Registry for Internet Numbers > > On Dec 9, 2025, at 1:19 PM, Chase via NANOG <[email protected]> wrote: > > Hey NANOG, > > > > After receiving a BGPAlerter notification that one of our subnets ( > 23.150.164.0/24) had been hijacked, I checked and noticed the prefix in > question was missing RPKI. Assuming I had fat fingered something and > butchered the ROA, I logged into ARIN and found that the prefix was missing > from our resource list entirely, and had been reallocated to another > organization and announced from their network. I created a ticket in ARIN > and called immediately. > > > > They confirmed that our subnet had been accidentally reallocated to > another customer, and that they are currently working on returning it to > us. After a couple hours, they told us the other organization will stop > announcing the prefix, and WHOIS will be returned shortly. > > > > I’m guessing there’s no way to prevent this kind of thing on our side if > the RPKI ROA itself is removed along with the allocation? I’m planning on > adding checks to look for missing ROAs (in addition to invalid/expiring > ones), which I'm guessing would've caught this earlier. > > > > Have any of you had anything like this happen with ARIN or another RIR? > I’m especially curious what might have happened if we’d only noticed and > reached out a few weeks later instead of within a few minutes. > > > > Best, > > Chase Lauer > > GalaxyGate, AS397031 > > https://galaxygate.net > _______________________________________________ > NANOG mailing list > > https://lists.nanog.org/archives/list/[email protected]/message/5MCMSACQADNXE65BTK34MQ3PXY4PDETF/ > > _______________________________________________ > NANOG mailing list > > https://lists.nanog.org/archives/list/[email protected]/message/FY3SDD72W5OFTJHIPHMB46JBGQFE2G6G/ _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/GFXWK5YNHSOFDEYI63Q6WA5RIP454GFW/
