Sounds like you’re treating this as a simple error when the root cause is a lack of change management practices.
Having a supervisor approve something is meaningless if there is no actual cross-validation of the change and associated data sources. Frankly I agree with others that you are likely not treating this seriously enough. I’ll reiterate what others have said - your whole literal purpose is to manage these assignments. You failed at your fundamental mission. I would recommend an independent audit and review of practices, as well as seeking inspiration from documentation-driven aerospace maintenance practices. - bri On Sun, Dec 14, 2025 at 16:08 John Sweeting via NANOG <[email protected]> wrote: > See inline (ARIN) > > John S. > > From: Jon Lewis via NANOG <[email protected]> > Date: Sunday, December 14, 2025 at 6:55 PM > To: John Curran via NANOG <[email protected]> > Cc: John Curran <[email protected]>, Jon Lewis <[email protected]> > Subject: Re: Accidental ARIN Reallocation > > On Fri, 12 Dec 2025, John Curran via NANOG wrote: > > > Short version – ARIN failed here (as you noted in your post). We’ve > > published a public incident report that lays out what happened, the > > impact, and what we’re changing: > > https://www.arin.net/announcements/20251212/ > > This is a pretty epic failure considering ARIN's purpose is the assignment > of unique Internet numbers (and the necessary record keeping to facilitate > that function). > > I assume 23.128.0.0/10 records are maintained partially in the "off line > Excel file" because your primary system lacks necessary features to > differentiate free space / sparse allocations in 23.128.0.0/10 from > "regular" free space? I assume the sparse allocation strategy here is to > allow a member to come back for more 4.10 space and ideally just extend an > original /24 to a /23, perhaps getting the next /24 eventually, and > finally swap the adjacent /23 and /24 for the covering /22? > > (ARIN) correct, and the community developed policy requires sparse > allocation. From the policy > "A /24 will be allocated. ARIN should use sparse allocation when possible > within that /10 block. “ > > Things I'm curious about that are not mentioned in the incident report: > > 1) When the analyst looked in the e-black-book and selected 23.150.164.0 > (23.150.164.0/22, I presume) to be used to satisfy the request they > were working on, did they fail to see that this /22 was already in use > for a sparse assignment and 23.150.164.0/24 was already assigned, or > when 23.150.164.0/24 was originally assigned to AS397031, was the > e-black-book not properly updated to reflect the sparse assignment of > 23.150.164.0/22? > > (ARIN) The e-black-book (spreadsheet) was not properly updated which led > to the analyst selecting that block. > > 2) When assigning IP space, is it customary for the analyst to check > current/recent snapshots of the global BGP tables to see if the space > selected for assignment is currently or has recently been advertised? > It's not guaranteed that assigned space will be in the DFZ, but if it > is, that's a pretty good indicator that something is going wrong with > the current assignment of the space. > > (ARIN) Yes the steps call for the analyst to check for routing, this step > was missed. > > 3) Did the block split automatically delete any/all child subnets of > 23.150.164.0/22, or did the analyst manually delete 23.150.164.0/24 > from whois (and that automatically deleted the ROA and rDNS > delegation)? > > (ARIN) The split, in this case, returned 23.150.164.0/24, 23.150.165.0/24 > and 23.150.166.0/23 so the analyst selected 23.150.164.0/24 for issuance. > The next step is to delete it from the ARIN OrgID in order to have it > available to allocate to the customer. The fact that it was assigned to > OrgID GL-700 versus OrgID ARIN was missed. > > (ARIN) We have inserted a supervisor mandatory check and approval in the > process prior to the delete action being initiated. > > > ---------------------------------------------------------------------- > Jon Lewis, MCP :) | I route > Blue Stream Fiber, Sr. Neteng | therefore you are > _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ > _______________________________________________ > NANOG mailing list > > https://lists.nanog.org/archives/list/[email protected]/message/X3WPNTJZUNDJMAOOJ5F3JKI4UAT65C3L/ > _______________________________________________ > NANOG mailing list > > https://lists.nanog.org/archives/list/[email protected]/message/6AC5KQD26PVHCN4T5HAFECIZWI67NJX4/ _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/65OO7CNHXJ2MUBWRGVLRLQF5HXASJ44E/
