Thanks! (I have a few questions, which may not be answerable here, I suppose.. if they can be answered that'd be cool though)
On Tue, Mar 8, 2016 at 12:59 PM, Nate Davis <[email protected]> wrote: > > ARIN's DNS process moves DNS data from the internal database to a Secure64 > DNSSEC appliance to a hidden distribution master. From the hidden > distribution > master, zones are fetched to name server constellations from ARIN, > VeriSign, and PCH. > > About two weeks ago a script was run that reset the serial on a zone in > the database. This script was run to accommodate an inter-RIR network This script sounds like something that should/would happen periodically? (whenever there's an xfer I guess?) is that correct? > transfer, and is not executed during the normal course of operations. It > reset the serial in our database in an unexpected way, and consequently > zone transfers from the Secure64 to our distribution master did not occur. > 'unexpected way' was decreased the serial? made it a string not an integer? other? (ie: Can I dork up my zone by setting the serial in the same fashion? what should I look for?) > This script was cumbersome and error prone, and had already been > identified to be replaced in the upcoming, planned deployment this weekend. > neat, ok. > This incident exposed a gap in our monitoring that we are fixing. Our is/was the gap: "Make sure serial is monotonically increasing" or is/was it: "If you are going to backup the serial, be sure to force a reload on all masters via process X" (ie: If I make a serial change, what other things should I look for? what monitoring gap do I also have?) > current, legacy monitoring system does not adequately identify the serial > number inconsistencies between the DNS nodes, nor does it adequately > handle issues with DNSSEC signature validation. We have work underway to > replace our old monitoring system with a new system that solves these > problems. The legacy/current system should be doing the moral equivalane of: for s in $(dig +short NS zone); do dig SOA +short zone @${s} done and make sure that all servers agree that the serial/soa is the same... right? Was there other verification that was happening? (or not) is the above too naive? should we be looking for other things? For dnssec I suppose you'd be doing the above but pulling rrsig for the SOA and making sure they are all the same. > This update is being posted to both arin-ppml and arin-tech-discuss. To > avoid non-policy related discussion on PPML, we encourage follow up > discussion > on arin-tech-discuss, a public mailing list that ARIN¹s engineering team > monitors. For those not > familiar with arin-tech-discuss, please subscribe here: > http://lists.arin.net/mailman/listinfo/arin-tech-discuss > oh :) > Regards, > > Nate Davis > > > On 3/8/16, 11:05 AM, "[email protected] on behalf of Chris > Woodfield" <[email protected] on behalf of [email protected]> > wrote: > >>Agreed with Chris¹ sentiment. I¹m a firm believer in the blameless >>post-mortem particularly when paired with action items to avoid repeat >>occurrences, and I¹d hope that others can learn from the technical issues >>involved. >> >>On top of that, everyone loves a good war story :) >> >>Thanks, >> >>-C >> >>> On Mar 8, 2016, at 7:45 AM, Christopher Morrow >>><[email protected]> wrote: >>> >>> Also, i'd be super awesome if there would be a pretty detailed >>> post-mortem document published about what happened, how it happened >>> and how it was discovered/repaired. >>> >>> I believe ARIN isn't the only one having these issues, so publishing >>> so other folk can learn would be great! >>> >>> -crhis >>> >>> On Mon, Mar 7, 2016 at 10:28 PM, <[email protected]> wrote: >>>> Nate, >>>> >>>> Please let us know if ARIN monitors all their zones for DNSSEC >>>>signature >>>> expiration. >>>> >>>> Frank >>>> >>>> -----Original Message----- >>>> From: [email protected] [mailto:[email protected]] On >>>> Behalf Of Nate Davis >>>> Sent: Monday, March 07, 2016 7:59 PM >>>> To: Michael Peddemors <[email protected]>; [email protected] >>>> Subject: Re: [arin-ppml] Just so it is recorded here (DNSSEC.. ) >>>>outages >>>> today.. >>>> >>>> Michael - thanks for reporting the issue. >>>> >>>> ARIN Engineering resolved the DNSSEC failure shortly after you reported >>>> the issue. They are currently looking into the cause of the failure. >>>>All >>>> DNSSEC functions should be operating properly at this time. >>>> >>>> Regards, >>>> >>>> Nate Davis >>>> Chief Operating Officer >>>> American Registry for Internet Numbers >>>> >>>> >>>> >>>> >>>> On 3/7/16, 6:14 PM, "[email protected] on behalf of Michael >>>> Peddemors" <[email protected] on behalf of >>>> [email protected]> wrote: >>>> >>>>> We had a flurry of reports from various customers, problems with >>>>>reverse >>>>> DNS lookups.. >>>>> >>>>> Limited to the 65/8 IPv4, and from apparent reports, related to a >>>>> failure to update a DNSSEC signature.. >>>>> >>>>> Reported: Anyone with a DNSSEC enforced name server will have problems >>>>> with PTR queries for that range. >>>>> >>>>> Someone with more inside knowledge can provide more details, I am >>>>>sure.. >>>>> >>>>> >>>>> >>>>> -- >>>>> "Catch the Magic of Linux..." >>>>> >>>>>----------------------------------------------------------------------- >>>>>- >>>>> Michael Peddemors, President/CEO LinuxMagic Inc. >>>>> Visit us at http://www.linuxmagic.com @linuxmagic >>>>> >>>>>----------------------------------------------------------------------- >>>>>- >>>>> A Wizard IT Company - For More Info http://www.wizard.ca >>>>> "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices >>>>>Ltd. >>>>> >>>>>----------------------------------------------------------------------- >>>>>- >>>>> 604-682-0300 Beautiful British Columbia, Canada >>>>> >>>>> This email and any electronic data contained are confidential and >>>>>intended >>>>> solely for the use of the individual or entity to which they are >>>>> addressed. >>>>> Please note that any views or opinions presented in this email are >>>>>solely >>>>> those of the author and are not intended to represent those of the >>>>> company. >>>>> >>>>> _______________________________________________ >>>>> PPML >>>>> You are receiving this message because you are subscribed to >>>>> the ARIN Public Policy Mailing List ([email protected]). >>>>> Unsubscribe or manage your mailing list subscription at: >>>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>>> Please contact [email protected] if you experience any issues. >>>> >>>> _______________________________________________ >>>> PPML >>>> You are receiving this message because you are subscribed to >>>> the ARIN Public Policy Mailing List ([email protected]). >>>> Unsubscribe or manage your mailing list subscription at: >>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>> Please contact [email protected] if you experience any issues. >>>> >>>> >>>> _______________________________________________ >>>> PPML >>>> You are receiving this message because you are subscribed to >>>> the ARIN Public Policy Mailing List ([email protected]). >>>> Unsubscribe or manage your mailing list subscription at: >>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>> Please contact [email protected] if you experience any issues. >>> _______________________________________________ >>> PPML >>> You are receiving this message because you are subscribed to >>> the ARIN Public Policy Mailing List ([email protected]). >>> Unsubscribe or manage your mailing list subscription at: >>> http://lists.arin.net/mailman/listinfo/arin-ppml >>> Please contact [email protected] if you experience any issues. >>> >> >>_______________________________________________ >>PPML >>You are receiving this message because you are subscribed to >>the ARIN Public Policy Mailing List ([email protected]). >>Unsubscribe or manage your mailing list subscription at: >>http://lists.arin.net/mailman/listinfo/arin-ppml >>Please contact [email protected] if you experience any issues. > _______________________________________________ PPML You are receiving this message because you are subscribed to the ARIN Public Policy Mailing List ([email protected]). Unsubscribe or manage your mailing list subscription at: http://lists.arin.net/mailman/listinfo/arin-ppml Please contact [email protected] if you experience any issues.
