+1, the real value of the community oversight comes in times when bad (or at least strange) things happen.
Kudos to the community and kudos to Kim and his team who appear to have handled the situation extremely well. regards Carlos (West Coast CO #4) On 11/21/14 3:40 PM, David Conrad wrote: > +1 > > Thanks very much Anne-Marie (and the other TCRs and folks involved). The > trust in DNSSEC is critically dependent on the service you all provide and I > believe that service is only going to get more important as DNSSEC becomes > more universally deployed. > > Regards, > -drc > > On Nov 21, 2014, at 9:13 AM, Richard Lamb <richard.l...@icann.org> wrote: > >> Wow. Thank you very much for the detailed description. But even more >> importantly for your participation and long travel. Without the critical >> eyes of experts like you the key ceremony would have little value. -Rick >> >> -----Original Message----- >> From: dnssec-deployment-boun...@dnssec-deployment.org >> [mailto:dnssec-deployment-boun...@dnssec-deployment.org] On Behalf Of >> Anne-Marie Eklund-Löwinder >> Sent: Friday, November 21, 2014 3:35 AM >> To: Paul Hoffman >> Cc: <dnssec-deployment@dnssec-deployment.org> >> Subject: Re: [Dnssec-deployment] A reminder - KSK event happening in < 15 >> minutes >> >> -----Ursprungligt meddelande----- >> Från: Paul Hoffman [mailto:paul.hoff...@vpnc.org] >> Skickat: den 21 november 2014 04:30 >> Till: Anne-Marie Eklund-Löwinder >> Kopia: <dnssec-deployment@dnssec-deployment.org> >> Ämne: Re: [Dnssec-deployment] A reminder - KSK event happening in < 15 >> minutes >> >> On Nov 20, 2014, at 7:26 PM, Anne-Marie Eklund-Löwinder >> <anne-marie.eklund-lowin...@iis.se> wrote: >>>> Yes, we had some exceptions during this ceremony, all of them taken care >>>> off by the ICANN team Gustavo and Kim in a very professional way. As a >>>> TCR/CO I have nothing to complain about. On the contrary, having >>>> exceptions is a part of the game, taking care of them in a good way takes >>>> practice, routines and experience. I believe all of that was to my >>>> expectations and on a very high standard. >> >>> For those of us who couldn't tune in, could you list the exceptions and >>> remediations? This sounds slightly more exciting than the normal ceremonies. >> >> Yes, slightly more exciting than usual, but nothing really bad happened. >> There will as you know be a full report published with the exceptions >> described by the IW, Gustavo Lozano from ICANN, but with my words this were >> the exceptions: >> >> E1. An adapter cable to connect the external display wasn't in place, needed >> to be provided by IKOS (minor detail). Doesn't need any remediation. The >> cable is in the ceremony room, it just wasn't put forward on the table. >> >> E2. CA missed to change the time zone on the laptop to UTC (it is EST as >> default I believe) and needed to repeat the step 7 on page 9 of the script, >> set Date&Time. (minor detail, no remediation needed, just follow the script >> - which will be made even more clear in this part). >> >> E3. The CA removed the wrong USB - instead of removing the KSR FD containing >> the SKR he removed the HSMFD, meaning that the logging was disrupted. >> Remediation: there were some suggestions during the ceremony, one is color >> coding, now all the FD's are identical except from small labels that might >> be hard to read during the process. >> >> We concluded that the only part that wasn't logged was the part when the HSM >> was disabled and deactivated. The keys were signed and the hash was correct, >> we had it printed on paper and read out from the RZM, Alejandro Bolivar >> from Verisign. >> >> Nevertheless, we altogether decided that this is not up to our usual >> standards, and that we should go back to the point in the script where we >> activate the laptop, and we restarted the ceremony from there. The reason >> behind this is that we wanted to be perfectly clear that the outcome would >> be exactly the same hash that was already made once before. Since we >> already had reached the stage where the HSM was stored in a TEB, the CA >> opened the new TEB where the HSM was stored, and started all over. >> >> E4. At some moment during this time something - unclear what - triggered one >> of the safes to think it was open, which it wasn't. If that would have been >> the case the people removing the equipment from safe #1 wouldn't have been >> able to leave the cage, which they obviously had been doing already. That >> meant that there was no one in the cage, and as you might recall, even >> though you for emergency reasons always can leave whatever part of the >> facility you are in (even though it would trigger the alarm of course) there >> is no easy way to get in, if the system thinks something is open that >> shouldn't be. You cannot open the cage if the safe is open, you cannot open >> the ceremony room if the cage door is open and so on. >> >> Anyway, this meant that no one could enter the cage to return the equipment >> to the safe. After a short discussion we decided to turn to the facility >> provider and ask them to use the physical key to the door that would let us >> in to the cage. The IKOS went off to do that, while all the others stayed in >> the ceremony room to guard the equipment and the CO:s credentials. Since >> Terremark has a formal process connected to retrieving the physical key, it >> took some time to get the allowance for that. But eventually an armed guard >> returned together with the IKOS and used the key (which btw is stored in a >> TEB with the facility). >> >>> From that point the security system needed to be rebooted, everyone signed >>> in for the ceremony room needed to get out from the ceremony room except >>> from the guard who watched everything from inside - and into the small hall >>> where we sign in. From there we could get back into the ceremony room, the >>> CA, IW and SSC could get back into the cage, open the safe and return the >>> equipment. >> >> After that the CO:s entered the cage with the IW, SSC and CA and returned >> their credentials. >> >> E5. During the process of returning the credentials to the safe, the memory >> cards on the recording cameras went full, and had to be changed. >> Remediation: Suggestion to use larger memory cards or cameras with a hard >> drive. >> >> Note: Every time the doors are opened and the security system is triggered, >> an alarm will go off. Some of you might remember that the sound from the >> alarm used to be really noisy. That sound has been replaced with an alarm on >> a very high (almost painful) tone together with a blue flashing light. For >> those watching remotely this might not be obvious. >> >> And that was it I think. :) The guard stayed until the end, to make sure >> that we didn't get locked out again. The physical key was returned into a >> TEB and the guard brought it back to where it is stored. >> >> We had a debriefing meeting after the ceremony where we discussed some >> improvements that can be made to avoid the risk of removing the wrong USB, >> color coding was one of the suggestions. The security system is about to be >> replaced next year if I am correctly informed, we have had issues with that >> before, the safes are a bit over sensitive. For instance, closing the safe >> door a little bit too hard will alert the seismic sensor and lock down >> everything. >> >> Again, everybody did a wonderful job. CA and IW were very professional, and >> everybody else showed patience and stayed calm. >> >> I hope that my description is fairly correct, there will as I mentioned be a >> more extensive report documented by the IW. >> >> All the best, >> >> Anne-Marie >> >> >> >> >
