RE: StartCom communication
Andrew et al, Just to inform that we have upgraded the EJBCA release to 6.9.0 in production this early morning and the CAA checking is working fine, as showed in the EJBCA´s logs. Best regards Iñigo Barreira CEO StartCom CA Limited -Original Message- From: dev-security-policy [mailto:dev-security-policy-bounces+inigo=startcomca@lists.mozilla.org] On Behalf Of Inigo Barreira via dev-security-policy Sent: lunes, 4 de septiembre de 2017 18:40 To: Andrew Ayer Cc: mozilla-dev-security-pol...@lists.mozilla.org Subject: RE: StartCom communication Hi Andrew, Thank you for your questions/suggestions. Let me answer 1.- We removed the ability to the CA administrator as a role to issue certificates, independently who´s assigned to that role. In the explanation I tried to detail exactly what happened and who did it (not to blame on them) and those two were asigned as CA administrators. So now, none can issue certificates directly from the EJBCA. 2.- This error happened when testing the CT behaviour and since then, nothing happened. Those were tests. The EJBCA system controls this and all the pre-certs generated mean the issuance of those certs. 3.- Hopefully yes. This is our intention. Primekey was delayed a little bit and we´re somehow in a rush for making the upgrade but we want to meet that deadline. Just in case, Primekey provided a manual checking of the CAA but our aim is to upgrade to the new version and use it. I´m afraid all EJBCA customers are in the same situation. Regarding the CAA test suite, we haven´t had time to test it but we´re going to test it. 4.- Yes, the use of those tools are performed after the issuance and the goal is that if this happen, a mis-issuance, be quick enough to react promptly and if possible, even before the customer can retrieve the certificate, revoke timely and start the investigation. And yes, I´ve been reading the email from Rob and I think it´s a very good idea. In fact, we´ve scheduled for next week if possible (after the upgrade to EJBCA 6.9.0) or the following one depending on how the CAA works once put in production. I´ll keep this list informed of the progress. We asked Primekey to include this kind of tools or similar before or after the issuance but there´s no such feature at the moment. Best regards Iñigo Barreira CEO StartCom CA Limited -Original Message- From: Andrew Ayer [mailto:a...@andrewayer.name] Sent: lunes, 4 de septiembre de 2017 18:06 To: Inigo Barreira Cc: mozilla-dev-security-pol...@lists.mozilla.org Subject: Re: StartCom communication On Mon, 4 Sep 2017 12:10:19 + Inigo Barreira via dev-security-policy wrote: > [...] > > a. Test certificates > > It__s been detailed in bugzilla #1369359. There__s an attachment with > a detailed explanation what happened, when, who, what was done to > remediate and to avoid it happening in the future. Those certs were > all the time under our control and lived for some minutes while > testing the CT behaviour. > > Actions: removed issuance rights for the CA administrator Could you clarify whether you removed the issuance rights of the particular administrator who issued the test certificates, or if you removed the ability for administrators in general to issue certificates bypassing technical controls? Considering that your incident report names the CA administrator and internal checker 10 times each, and states the reason for the incident as "Operators of EJBCA server didn't follow the operating protocol of StartCom", one could conclude that StartCom sees its failures as a problem with individual people, rather than a systemic problem of insufficient technical controls. What has StartCom done to correct this? > b. Pre-certificates > > It__s explained in bugzilla #1386894. Perhaps I was wrong with the > word __intent__ and then no need to revoke as they were not real > certificates, but once we had to do it, had to work with Primekey to > revoke those certs, because they didn__t exist for the EJBCA system as > they only have certificates, not pre-certificates. What has been done to fix the underlying issue so that in the future you won't create a pre-certificate unless you really intend to issue the final certificate? > [...] > > a. apply newest version of EJBCA v6.9.0 > > Primekey has just released the new version of EJBCA, v6.9.0 (end of > august). > > This new version comes with some important improvements, such as: > > - CAA automatic checking > > - Perform public key validation (RSA and ECC) > Will EJBCA 6.9.0 be deployed by September 8, when CAA checking becomes mandatory? If not, how will you be checking CAA on September 8? Does your/EJBCA's implementation of CAA pass the test suite I recently posted to the CABF list? https://caatestsuite.com/ > b. integrate cablint/x509lint in CAProxy > > These tools
RE: StartCom communication
Hi Andrew, Thank you for your questions/suggestions. Let me answer 1.- We removed the ability to the CA administrator as a role to issue certificates, independently who´s assigned to that role. In the explanation I tried to detail exactly what happened and who did it (not to blame on them) and those two were asigned as CA administrators. So now, none can issue certificates directly from the EJBCA. 2.- This error happened when testing the CT behaviour and since then, nothing happened. Those were tests. The EJBCA system controls this and all the pre-certs generated mean the issuance of those certs. 3.- Hopefully yes. This is our intention. Primekey was delayed a little bit and we´re somehow in a rush for making the upgrade but we want to meet that deadline. Just in case, Primekey provided a manual checking of the CAA but our aim is to upgrade to the new version and use it. I´m afraid all EJBCA customers are in the same situation. Regarding the CAA test suite, we haven´t had time to test it but we´re going to test it. 4.- Yes, the use of those tools are performed after the issuance and the goal is that if this happen, a mis-issuance, be quick enough to react promptly and if possible, even before the customer can retrieve the certificate, revoke timely and start the investigation. And yes, I´ve been reading the email from Rob and I think it´s a very good idea. In fact, we´ve scheduled for next week if possible (after the upgrade to EJBCA 6.9.0) or the following one depending on how the CAA works once put in production. I´ll keep this list informed of the progress. We asked Primekey to include this kind of tools or similar before or after the issuance but there´s no such feature at the moment. Best regards Iñigo Barreira CEO StartCom CA Limited -Original Message- From: Andrew Ayer [mailto:a...@andrewayer.name] Sent: lunes, 4 de septiembre de 2017 18:06 To: Inigo Barreira Cc: mozilla-dev-security-pol...@lists.mozilla.org Subject: Re: StartCom communication On Mon, 4 Sep 2017 12:10:19 + Inigo Barreira via dev-security-policy wrote: > [...] > > a. Test certificates > > It__s been detailed in bugzilla #1369359. There__s an attachment with > a detailed explanation what happened, when, who, what was done to > remediate and to avoid it happening in the future. Those certs were > all the time under our control and lived for some minutes while > testing the CT behaviour. > > Actions: removed issuance rights for the CA administrator Could you clarify whether you removed the issuance rights of the particular administrator who issued the test certificates, or if you removed the ability for administrators in general to issue certificates bypassing technical controls? Considering that your incident report names the CA administrator and internal checker 10 times each, and states the reason for the incident as "Operators of EJBCA server didn't follow the operating protocol of StartCom", one could conclude that StartCom sees its failures as a problem with individual people, rather than a systemic problem of insufficient technical controls. What has StartCom done to correct this? > b. Pre-certificates > > It__s explained in bugzilla #1386894. Perhaps I was wrong with the > word __intent__ and then no need to revoke as they were not real > certificates, but once we had to do it, had to work with Primekey to > revoke those certs, because they didn__t exist for the EJBCA system as > they only have certificates, not pre-certificates. What has been done to fix the underlying issue so that in the future you won't create a pre-certificate unless you really intend to issue the final certificate? > [...] > > a. apply newest version of EJBCA v6.9.0 > > Primekey has just released the new version of EJBCA, v6.9.0 (end of > august). > > This new version comes with some important improvements, such as: > > - CAA automatic checking > > - Perform public key validation (RSA and ECC) > Will EJBCA 6.9.0 be deployed by September 8, when CAA checking becomes mandatory? If not, how will you be checking CAA on September 8? Does your/EJBCA's implementation of CAA pass the test suite I recently posted to the CABF list? https://caatestsuite.com/ > b. integrate cablint/x509lint in CAProxy > > These tools have been integrated at the issuance process. We__ll check > all certificates issued and will send an email internally to act > accordingly, i.e, revoking the cert and to start an inmediate > investigation of the error. This is a good step, but it's too late to stop the misissuance. Instead, you should check the TBSCertificate, and not sign if there is an error. Regards, Andrew > [...] smime.p7s Description: S/MIME cryptographic signature ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
Re: StartCom communication
On Mon, 4 Sep 2017 12:10:19 + Inigo Barreira via dev-security-policy wrote: > [...] > > a. Test certificates > > It__s been detailed in bugzilla #1369359. There__s an attachment with a > detailed explanation what happened, when, who, what was done to > remediate and to avoid it happening in the future. Those certs were > all the time under our control and lived for some minutes while > testing the CT behaviour. > > Actions: removed issuance rights for the CA administrator Could you clarify whether you removed the issuance rights of the particular administrator who issued the test certificates, or if you removed the ability for administrators in general to issue certificates bypassing technical controls? Considering that your incident report names the CA administrator and internal checker 10 times each, and states the reason for the incident as "Operators of EJBCA server didn't follow the operating protocol of StartCom", one could conclude that StartCom sees its failures as a problem with individual people, rather than a systemic problem of insufficient technical controls. What has StartCom done to correct this? > b. Pre-certificates > > It__s explained in bugzilla #1386894. Perhaps I was wrong with the word > __intent__ and then no need to revoke as they were not real > certificates, but once we had to do it, had to work with Primekey to > revoke those certs, because they didn__t exist for the EJBCA system as > they only have certificates, not pre-certificates. What has been done to fix the underlying issue so that in the future you won't create a pre-certificate unless you really intend to issue the final certificate? > [...] > > a. apply newest version of EJBCA v6.9.0 > > Primekey has just released the new version of EJBCA, v6.9.0 (end of > august). > > This new version comes with some important improvements, such as: > > - CAA automatic checking > > - Perform public key validation (RSA and ECC) > Will EJBCA 6.9.0 be deployed by September 8, when CAA checking becomes mandatory? If not, how will you be checking CAA on September 8? Does your/EJBCA's implementation of CAA pass the test suite I recently posted to the CABF list? https://caatestsuite.com/ > b. integrate cablint/x509lint in CAProxy > > These tools have been integrated at the issuance process. We__ll check > all certificates issued and will send an email internally to act > accordingly, i.e, revoking the cert and to start an inmediate > investigation of the error. This is a good step, but it's too late to stop the misissuance. Instead, you should check the TBSCertificate, and not sign if there is an error. Regards, Andrew > [...] ___ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
StartCom communication
Hi all, I´ve realized that there has not been a good communication path to announce all the tasks and actions performed by StartCom during this time and this email will try to remediate it. I´d also like to ask you for some feedback, comments and/or suggestions on how to improve. I think we´ve done a great effort and developed a robust system but it´s also true that we have had some errors even though all were fixed timely (despite the discussions that are still alive regarding some of them) and put all the countermeasures needed for not happening again. Thanks for all your suggestions and comments. I will try to summarize everything. Areas 1. Remediation plan a. Legal structure and management separation b. operations separation c. system separation d. CT logging 2. Mozilla requirements (bugzilla #1311832) a. provide a list of changes b. implement changes and update CP/CPS, stating explicitly backdating is prohibited c. providing full webtrust audit d. security audit e. All certs CT logged, if possible not using own CT log 3. Issues a. test certificates b. pre-certificates for test certificates c. use of specific curves not allowed by Mozilla but BRs d. Country code e. RSA parameters f. Unicode vs punnycode 4. Improvements a. apply newest version of EJBCA v6.9.0 b. integrate cablint/x509lint in CAProxy c. check of CSRs d. integrate crt.sh in database 1. Remediation plan This is what StartCom planned to do and was indicated last year in October. It was presented to Mozilla and also in an email sent to the m.d.s.p These are the main tasks performed, which are also included in bugzilla #1311832 a. Legal structure and management separation StartCom HK, which is the main head of the group, is owned 100% by Qifei International development Co. Ltd in HK, which is also controlled 100% by 360 technology Inc. In november 2016, there were some news indicating the completed transer of 100% shares from WoSign to Qihoo 360, so StartCom officially became a wholly owned subsidiary of Qihoo 360. So, there´s no control from Wosign related to StartCom. StartCom has offices at China, UK, Israel and Spain. Besides there was a change in the management in which Xiaosheng Tan of 360 was named chairman of the board and Iñigo Barreira, CEO. Richard Wang has no relation, control nor influence in StartCom. It was removed from the StartCom directors at the UK office last year. (Note: recently had to be named again due to some issues related to the bank that have been already fixed and hence removed again) b. Operations separation All operations are performed by Startcom employees, from own premises. There´s no relation at all with any other Wosign operations or departments. All validations, support and customer service is provided by StartCom. Besides StartCom signed a contract with 360 to provide hosting services of the PKI infrastructure (CA, VA, TSA, CT, CMS, HSM, web, ), maintenance and support, as well as developing, testing, etc. c. System separation All systems used for the PKI has been updated or directly changed and all are hosted in 360 premises as stated above. Web and CMS have been totally updated, recoded under another programming language, PHP, for which 360 is more familiar with. This coding has been performed by the 360 R&D team and checked later by its security team. Cure53 was the firm hired for the security audit. OTOH, StartCom finally decided to acquire the Primekey´s PKI solution, EJBCA. We considered a significant step forward use this commercial PKI software, well known in the industry. Furthermore, the StartCom OCSP also uses the Akamai CDN as well as some other services. d. CT logging StartCom logs all SSL issued certificates in CT logs. Currently we´re using Google CT logs and also StartCom CT log. We´re also planning to use Comodo CTs Mammoth and Sabre as well as the new Venafi one. 2. Mozilla requirements (bugzilla #1311832) a. provide a list of changes The remediation plan was the whole list of changes that were presented to Mozilla and they were agreed with that plan. b. implement changes and update CP/CPS, stating explicitly backdating is prohibited All those changes have been implemented as per the remediation plan stated above. The CPS was updated accordingly c. providing full webtrust audit The full webtrust audits have been performed by PwC. StartCom did a Webtrust for CA, webtrust BR, webtrust EV and webtrust for CS. There were some findings, which are reflected in the reports, but most of them were fixed. Right now only the configuration of the TSA and the update of the Business Continuity Plan are delayed, which in any case, will be finished soon. d. Security audit For the security audit Mozilla recommended 2 firms they worked with them in the past and finally hired Cure 53. e. All certs CT logged, if possible not using own CT log StartCom logs all the SSL c
