-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I absolutely agree that the participation of everyone who were present in the room is important.
The webcast is important too (doesn't work in Debian/Ubunty btw), and the chatroom did provide some good dialogue about what was acceptable, even though that's probably not logged. For me, the choice of Adobe Connect is a mistake, as I had to struggle to find a supported device, which is a surprise, considering the (also) technical approach that ICANN approaches. As a ceremony adminstrator (CA) of signing ceremony running several times every year at PCH for signing several TLDs, I wish to confirm that there are no key ceremony without exceptions and "this is okay" as long as it's written down and documented. If exceptions are based on basic things like bring water, phones or firearms in the facility, of course, the script/rules have to be changed to allow this, but exceptions are not bad in themselves. I trust several of the people present at the ceremony (including Olaf, Anne-Marie, Kim, notsurewhoelsewasthere). Good ceremony. Good to watch. For everyone on dnssec-deployment (incl me) could someone include a reference to how to get notified of upcoming key ceremonies? Thank you and have a nice weekend, Robert ML On 2014-11-21 15:03, Mehmet Akcin wrote: > Hi > > It's can be very hard at times to concentrate the very detailed steps and minor errors can just happen which doesn't impact the overall ceremony's professionalism and result. > > All TCRs paying full attention and providing feedback like Anne-Marie had been the reason for continuous success and on-going improvements to root dnssec ceremonies. > > Great Work Kim/Gustavo. Thanks for this extremely helpful and detailed exception report Anne-Marie. > > Mehmet > >> On Nov 21, 2014, at 3:34 AM, Anne-Marie Eklund-Löwinder <anne-marie.eklund-lowin...@iis.se> wrote: >> >> -----Ursprungligt meddelande----- >> Från: Paul Hoffman [mailto:paul.hoff...@vpnc.org] >> Skickat: den 21 november 2014 04:30 >> Till: Anne-Marie Eklund-Löwinder >> Kopia: <dnssec-deployment@dnssec-deployment.org> >> Ämne: Re: [Dnssec-deployment] A reminder - KSK event happening in < 15 minutes >> >> On Nov 20, 2014, at 7:26 PM, Anne-Marie Eklund-Löwinder <anne-marie.eklund-lowin...@iis.se> wrote: >>>> Yes, we had some exceptions during this ceremony, all of them taken care off by the ICANN team Gustavo and Kim in a very professional way. As a TCR/CO I have nothing to complain about. On the contrary, having exceptions is a part of the game, taking care of them in a good way takes practice, routines and experience. I believe all of that was to my expectations and on a very high standard. >> >>> For those of us who couldn't tune in, could you list the exceptions and remediations? This sounds slightly more exciting than the normal ceremonies. >> >> Yes, slightly more exciting than usual, but nothing really bad happened. There will as you know be a full report published with the exceptions described by the IW, Gustavo Lozano from ICANN, but with my words this were the exceptions: >> >> E1. An adapter cable to connect the external display wasn't in place, needed to be provided by IKOS (minor detail). Doesn't need any remediation. The cable is in the ceremony room, it just wasn't put forward on the table. >> >> E2. CA missed to change the time zone on the laptop to UTC (it is EST as default I believe) and needed to repeat the step 7 on page 9 of the script, set Date&Time. (minor detail, no remediation needed, just follow the script - which will be made even more clear in this part). >> >> E3. The CA removed the wrong USB - instead of removing the KSR FD containing the SKR he removed the HSMFD, meaning that the logging was disrupted. Remediation: there were some suggestions during the ceremony, one is color coding, now all the FD's are identical except from small labels that might be hard to read during the process. >> >> We concluded that the only part that wasn't logged was the part when the HSM was disabled and deactivated. The keys were signed and the hash was correct, we had it printed on paper and read out from the RZM, Alejandro Bolivar from Verisign. >> >> Nevertheless, we altogether decided that this is not up to our usual standards, and that we should go back to the point in the script where we activate the laptop, and we restarted the ceremony from there. The reason behind this is that we wanted to be perfectly clear that the outcome would be exactly the same hash that was already made once before. Since we already had reached the stage where the HSM was stored in a TEB, the CA opened the new TEB where the HSM was stored, and started all over. >> >> E4. At some moment during this time something - unclear what - triggered one of the safes to think it was open, which it wasn't. If that would have been the case the people removing the equipment from safe #1 wouldn't have been able to leave the cage, which they obviously had been doing already. That meant that there was no one in the cage, and as you might recall, even though you for emergency reasons always can leave whatever part of the facility you are in (even though it would trigger the alarm of course) there is no easy way to get in, if the system thinks something is open that shouldn't be. You cannot open the cage if the safe is open, you cannot open the ceremony room if the cage door is open and so on. >> >> Anyway, this meant that no one could enter the cage to return the equipment to the safe. After a short discussion we decided to turn to the facility provider and ask them to use the physical key to the door that would let us in to the cage. The IKOS went off to do that, while all the others stayed in the ceremony room to guard the equipment and the CO:s credentials. Since Terremark has a formal process connected to retrieving the physical key, it took some time to get the allowance for that. But eventually an armed guard returned together with the IKOS and used the key (which btw is stored in a TEB with the facility). >> >> From that point the security system needed to be rebooted, everyone signed in for the ceremony room needed to get out from the ceremony room except from the guard who watched everything from inside - and into the small hall where we sign in. From there we could get back into the ceremony room, the CA, IW and SSC could get back into the cage, open the safe and return the equipment. >> >> After that the CO:s entered the cage with the IW, SSC and CA and returned their credentials. >> >> E5. During the process of returning the credentials to the safe, the memory cards on the recording cameras went full, and had to be changed. Remediation: Suggestion to use larger memory cards or cameras with a hard drive. >> >> Note: Every time the doors are opened and the security system is triggered, an alarm will go off. Some of you might remember that the sound from the alarm used to be really noisy. That sound has been replaced with an alarm on a very high (almost painful) tone together with a blue flashing light. For those watching remotely this might not be obvious. >> >> And that was it I think. :) The guard stayed until the end, to make sure that we didn't get locked out again. The physical key was returned into a TEB and the guard brought it back to where it is stored. >> >> We had a debriefing meeting after the ceremony where we discussed some improvements that can be made to avoid the risk of removing the wrong USB, color coding was one of the suggestions. The security system is about to be replaced next year if I am correctly informed, we have had issues with that before, the safes are a bit over sensitive. For instance, closing the safe door a little bit too hard will alert the seismic sensor and lock down everything. >> >> Again, everybody did a wonderful job. CA and IW were very professional, and everybody else showed patience and stayed calm. >> >> I hope that my description is fairly correct, there will as I mentioned be a more extensive report documented by the IW. >> >> All the best, >> >> Anne-Marie >> >> >> >> - -- Robert Martin-Legene Internet Infrastructure Specialist Packet Clearing House -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlRvpkEACgkQrZ2I8G3iqu3//QCeM1kH3tLW6qEnNv+o6Oa1Cvle cR8AmgNQBRLXq3KkaRXbqKfqcuCeFoPG =s0pR -----END PGP SIGNATURE-----
