On 3/9/16, 11:34 AM, "Christopher Morrow" <[email protected]> wrote:
>Thanks! >(I have a few questions, which may not be answerable here, I suppose.. >if they can be answered that'd be cool though) > >On Tue, Mar 8, 2016 at 12:59 PM, Nate Davis <[email protected]> wrote: >> >> ARIN's DNS process moves DNS data from the internal database to a >>Secure64 >> DNSSEC appliance to a hidden distribution master. From the hidden >> distribution >> master, zones are fetched to name server constellations from ARIN, >> VeriSign, and PCH. >> >> About two weeks ago a script was run that reset the serial on a zone in >> the database. This script was run to accommodate an inter-RIR network > >This script sounds like something that should/would happen >periodically? (whenever there's an xfer I guess?) is that correct? > >> transfer, and is not executed during the normal course of operations. It >> reset the serial in our database in an unexpected way, and consequently >> zone transfers from the Secure64 to our distribution master did not >>occur. >> > >'unexpected way' was decreased the serial? made it a string not an >integer? other? >(ie: Can I dork up my zone by setting the serial in the same fashion? >what should I look for?) > >> This script was cumbersome and error prone, and had already been >> identified to be replaced in the upcoming, planned deployment this >>weekend. >> > >neat, ok. > >> This incident exposed a gap in our monitoring that we are fixing. Our > >is/was the gap: "Make sure serial is monotonically increasing" >or is/was it: "If you are going to backup the serial, be sure to force >a reload on all masters via process X" > >(ie: If I make a serial change, what other things should I look for? >what monitoring gap do I also have?) > >> current, legacy monitoring system does not adequately identify the >>serial >> number inconsistencies between the DNS nodes, nor does it adequately >> handle issues with DNSSEC signature validation. We have work underway to >> replace our old monitoring system with a new system that solves these >> problems. > >The legacy/current system should be doing the moral equivalane of: > for s in $(dig +short NS zone); do > dig SOA +short zone @${s} > done > >and make sure that all servers agree that the serial/soa is the same... >right? >Was there other verification that was happening? (or not) >is the above too naive? should we be looking for other things? > >For dnssec I suppose you'd be doing the above but pulling rrsig for >the SOA and making sure they are all the same. > >> This update is being posted to both arin-ppml and arin-tech-discuss. To >> avoid non-policy related discussion on PPML, we encourage follow up >> discussion >> on arin-tech-discuss, a public mailing list that ARIN¹s engineering team >> monitors. For those not >> familiar with arin-tech-discuss, please subscribe here: >> http://lists.arin.net/mailman/listinfo/arin-tech-discuss >> > >oh :) > >> Regards, >> >> Nate Davis >> >> >> On 3/8/16, 11:05 AM, "[email protected] on behalf of Chris >> Woodfield" <[email protected] on behalf of [email protected]> >> wrote: >> >>>Agreed with Chris¹ sentiment. I¹m a firm believer in the blameless >>>post-mortem particularly when paired with action items to avoid repeat >>>occurrences, and I¹d hope that others can learn from the technical >>>issues >>>involved. >>> >>>On top of that, everyone loves a good war story :) >>> >>>Thanks, >>> >>>-C >>> >>>> On Mar 8, 2016, at 7:45 AM, Christopher Morrow >>>><[email protected]> wrote: >>>> >>>> Also, i'd be super awesome if there would be a pretty detailed >>>> post-mortem document published about what happened, how it happened >>>> and how it was discovered/repaired. >>>> >>>> I believe ARIN isn't the only one having these issues, so publishing >>>> so other folk can learn would be great! >>>> >>>> -crhis >>>> >>>> On Mon, Mar 7, 2016 at 10:28 PM, <[email protected]> wrote: >>>>> Nate, >>>>> >>>>> Please let us know if ARIN monitors all their zones for DNSSEC >>>>>signature >>>>> expiration. >>>>> >>>>> Frank >>>>> >>>>> -----Original Message----- >>>>> From: [email protected] [mailto:[email protected]] >>>>>On >>>>> Behalf Of Nate Davis >>>>> Sent: Monday, March 07, 2016 7:59 PM >>>>> To: Michael Peddemors <[email protected]>; [email protected] >>>>> Subject: Re: [arin-ppml] Just so it is recorded here (DNSSEC.. ) >>>>>outages >>>>> today.. >>>>> >>>>> Michael - thanks for reporting the issue. >>>>> >>>>> ARIN Engineering resolved the DNSSEC failure shortly after you >>>>>reported >>>>> the issue. They are currently looking into the cause of the failure. >>>>>All >>>>> DNSSEC functions should be operating properly at this time. >>>>> >>>>> Regards, >>>>> >>>>> Nate Davis >>>>> Chief Operating Officer >>>>> American Registry for Internet Numbers >>>>> >>>>> >>>>> >>>>> >>>>> On 3/7/16, 6:14 PM, "[email protected] on behalf of Michael >>>>> Peddemors" <[email protected] on behalf of >>>>> [email protected]> wrote: >>>>> >>>>>> We had a flurry of reports from various customers, problems with >>>>>>reverse >>>>>> DNS lookups.. >>>>>> >>>>>> Limited to the 65/8 IPv4, and from apparent reports, related to a >>>>>> failure to update a DNSSEC signature.. >>>>>> >>>>>> Reported: Anyone with a DNSSEC enforced name server will have >>>>>>problems >>>>>> with PTR queries for that range. >>>>>> >>>>>> Someone with more inside knowledge can provide more details, I am >>>>>>sure.. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> "Catch the Magic of Linux..." >>>>>> >>>>>>--------------------------------------------------------------------- >>>>>>-- >>>>>>- >>>>>> Michael Peddemors, President/CEO LinuxMagic Inc. >>>>>> Visit us at http://www.linuxmagic.com @linuxmagic >>>>>> >>>>>>--------------------------------------------------------------------- >>>>>>-- >>>>>>- >>>>>> A Wizard IT Company - For More Info http://www.wizard.ca >>>>>> "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices >>>>>>Ltd. >>>>>> >>>>>>--------------------------------------------------------------------- >>>>>>-- >>>>>>- >>>>>> 604-682-0300 Beautiful British Columbia, Canada >>>>>> >>>>>> This email and any electronic data contained are confidential and >>>>>>intended >>>>>> solely for the use of the individual or entity to which they are >>>>>> addressed. >>>>>> Please note that any views or opinions presented in this email are >>>>>>solely >>>>>> those of the author and are not intended to represent those of the >>>>>> company. >>>>>> >>>>>> _______________________________________________ >>>>>> PPML >>>>>> You are receiving this message because you are subscribed to >>>>>> the ARIN Public Policy Mailing List ([email protected]). >>>>>> Unsubscribe or manage your mailing list subscription at: >>>>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>>>> Please contact [email protected] if you experience any issues. >>>>> >>>>> _______________________________________________ >>>>> PPML >>>>> You are receiving this message because you are subscribed to >>>>> the ARIN Public Policy Mailing List ([email protected]). >>>>> Unsubscribe or manage your mailing list subscription at: >>>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>>> Please contact [email protected] if you experience any issues. >>>>> >>>>> >>>>> _______________________________________________ >>>>> PPML >>>>> You are receiving this message because you are subscribed to >>>>> the ARIN Public Policy Mailing List ([email protected]). >>>>> Unsubscribe or manage your mailing list subscription at: >>>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>>> Please contact [email protected] if you experience any issues. >>>> _______________________________________________ >>>> PPML >>>> You are receiving this message because you are subscribed to >>>> the ARIN Public Policy Mailing List ([email protected]). >>>> Unsubscribe or manage your mailing list subscription at: >>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>> Please contact [email protected] if you experience any issues. >>>> >>> >>>_______________________________________________ >>>PPML >>>You are receiving this message because you are subscribed to >>>the ARIN Public Policy Mailing List ([email protected]). >>>Unsubscribe or manage your mailing list subscription at: >>>http://lists.arin.net/mailman/listinfo/arin-ppml >>>Please contact [email protected] if you experience any issues. >> _______________________________________________ arin-tech-discuss mailing list [email protected] http://lists.arin.net/mailman/listinfo/arin-tech-discuss
